#8 Data Science, Astronomy and the Open Source (with Jake VanderPlas)

In this episode of Data Framed, a data camp podcast, I'll be speaking with Jake Vanderplass, a data scientist, astronomer, open source beast, and renowned Pythonista. We'll speak about data science, astronomy, the open source development world, and the importance of interdisciplinary conversations to data science. I'm Hugo Baun Anderson, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBown and DataCamp. At DataCamp, you can find all our episodes and show notes@datacamp.com community podcast. Hi Jake, and welcome to Data Framed.

Hi. Thanks. It's good to be here.

So good to have you and I'm really excited to have you on the show to chat about data science, astronomy, open source, Python, all the wide array of things that you're interested in. But first, I'd like to find out about you. What are you known for in the data science community?

Yeah, so I think I'm mostly known as I'm typecast as a Python person. Appropriately, I'm probably known for scikit Learn, because that, that was my intro into the community. That's where I sort of started getting. Where I got started with open source and with contribution. And I'm also known as an astronomer. I did my PhD in astronomy, and even though I don't do as much astronomy research these days as I used to, I still feel like I'm pretty close to that community for sure.

And these are all things that we're going to chat about in this conversation, from the Python landscape to machine learning and scikit Learn to your work in astronomy. But first I'd like to know how you originally got involved in data science. What's the backstory there?

Yeah, so I started in grad school and I went to grad school for astronomy mainly because I liked physics and I liked astronomy as an application of physics. So I didn't really have data science on my mind at all when I jumped into it. But going into science these days, you have to write code, you have to analyze data no matter how, pretty much no matter what kind of science you're doing. And so from my first quarter in grad school, I started learning Python. And as I got deeper into it, I really found that I started to enjoy the data analysis and enjoy the software portion of the research that I was doing. So I published this paper in 2009 that was on using manifold learning, like local linear embedding to study galaxy spectra from the Sloan Digital sky surve. And I finished that paper and I had spent a solid few months writing this code for efficient manifold learning with a Python interface. And at the end of it, I put a tarball of the code on my website and I thought to myself, this is ridiculous. The next person who comes along who wants to do manifold learning for astronomy data, they're going to have to totally reinvent the wheel. I don't know if I could go back and download my tarball and use the code the way I had written it at the time. So I emailed the SCIPY list and asked. I think I actually asked about the ball tree code that was a component of that manifold learning code. But I asked if it was something that folks would want in the SCIPY package and they were like crickets for a while. And then I think it was Guy Lavaroku, who's been leading the SciKit learn package, emailed me back and said, hey, we could use this code in scikit Learn. And I said, sure, that sounds great. And so I got involved from there and caught the open source bug. And I've been doing this open source Python and open source what became known more widely as data science ever since then.

Yeah. And you've spoken to several really interesting issues there from the role of statistics in data science, the role of programming in data science, the role of. Well, essentially what you're speaking to with respect to putting this tarball up and rethinking that is the idea of reproducibility in science and data sciences as well. But with respect to astronomy, I. So in a previous incarnation I worked in applied math in the biological sciences and on campus. It was always when I needed to figure out how to analyze my data, how to do serious robust statistics, I always went to the astronomers because they always seemed the most adept and knew the techniques that I needed to do.

Yeah, yeah, that makes sense. That makes sense. Yeah, yeah. So the. I think that's probably due. So astronomy was kind of an early adopter of these large scale data things. As history goes. Like, I think it. I would peg it back to the Sloan Digital Sky Survey, which was this. At the time, it was a pretty groundbreaking survey. You know, previous to the Sloan Digital Sky Survey mostly what people, how people did astronomy is they decided, you know, I want to look at this particular star, this particular nebula, or this particular galaxy. And they'd write up a proposal and send it to a telescope allocation committee in the field we call them attck. So you send your proposal to attck, and you try to get allocated time to look at this particular object or class of object. And there were, you know, hundreds of astronomers vying for time on the biggest telescope. So you need a really good proposal to say, you know, this is why I should use this very expensive instrument instead of anyone else. Then they, you know, as an astronomer, you'd go up and you'd gather that data. You'd spend a few months processing it and writing about it, and then write your paper, right? And so this is how astronomy basically worked up until, say, the 80s and early 90s and starting around then, maybe a little bit earlier. But the big player in this was the Sloan Digital Sky Survey that started in the 90s. And the idea there was that instead of having individual astronomers use the telescope every night to look at specific things they wanted to, why don't we just make a telescope that scans the night sky and looks for everything, Right? So the Sloan Digital Sky Survey did this, this photometric scan of the entire sky, basically taking images of the whole sky to find out where interesting objects are, and then went back and did, over the course of about 10 years for the first survey, did a spectroscopic scan of about a million of those objects. And spectroscopic observations are like basically taking the light from an individual object and splitting it into its different wavelengths. So you get a graph of brightness versus wavelength for the object. And that can tell you huge amounts of information about what's going on in that galaxy or in that star and that quasar or whatever you're looking at. And so the real opportunity there was all of a sudden there was this huge, vast swath of data that was open to anybody who could analyze it. You know, the data was posted in this public database online, and literally anybody. You didn't have to be an astronomer, you didn't have to be affiliated with the university. You could go and download that data and try to learn things from it. So it really, in a lot of ways, changed how astronomy was done. You know, all of a sudden, you didn't have to think about specific objects and apply for telescope time. You were thinking about data mining and trying to. Trying to sift through large amounts of data to figure out what's interesting to answer the questions you want to answer. And that sort of survey astronomy idea has carried forward, and a lot of the big projects now are operating in the survey mode. There's definitely still individual telescopes that you can use to look at individual objects. In the classic sense. But surveys are a big part of modern astronomy.

So what you're speaking to, there is really a revolution in the way data is collected and then the ability for everybody to access that data.

Yeah, exactly. And we're seeing a little bit of that in other fields too, all sorts of fields. We have oceanographic surveys where you can go and download this ocean data from ships that are driving around. There's genomic data and other biological data. So lots of fields are seeing this sort of revolution. But I think in many ways it was astronomers and probably before that, particle physicists who really led the way in kind of large data intensive surveys and science.

Right. So my question is, once we have this explosion in the amount of data available, well, in any discipline. But let's speak about astronomy. Is there a need for techniques and technological infrastructures such as databases to actually catch up with this?

Yeah, absolutely. And if you look at the early Sloan Digital Sky Survey, the real, the real innovation on that front was done by Jim Gray, who's well known in the computer science database community. And you'll find quotes from him all over the place because he was so prolific. But he teamed up with a guy named Alex Saleh who was at, I think at John Hopkins at the time. He's at Johns Hopkins now. But they basically were the ones who came together and said, this is a new way of doing astronomy. We need new tools. And they said, let's develop these database tools for use in astronomy and let's start to train, let's start to train astronomers on how to do this. Incidentally, small little side note, my PhD advisor, Andy Connolly, did his postdoc with Alex Saleh working on these database things. So I was brought up in that tradition in my grad school career.

So I'm thinking as well, just the types of algorithms you'll use to do your analysis. So I'm not sure what type of algorithms you've used historically, but there are a whole bunch of machine learning algorithms, k nearest neighbors or even, you know, finding the mean, finding the average of a data set, which don't necessarily scale so well when you get huge data sets. Right, so has this, this been a challenge for the.

Yeah, yeah, absolutely. And it's, it's a challenge because the, the mode that most astronomers are, have been used to working with data and still are to, to a big extent is you download the data onto your computer and you visualize it in, you know, whatever IDL or Matlab or Python or something like that, and then you hopefully.

Python.

Yeah. Python's getting, it's more Python these days, but 10 years ago, it was definitely favored IDL and other languages. And so it's this different mode of working with data and this different way of thinking about data. So some of the work that I've done, for example, is like, one thing that you often want to do in astronomy is figure out, look for things that are varying in the sky, the stars that are brighter tonight than they were yesterday. And there's a certain class of variable stars that are periodic variables, which means that if you plot their brightness versus time, it has this regular pattern that's kind of like a sine wave, but slightly different. Not exactly a sine wave, but it's periodic. And these are important because they can be used. I mean, I'm getting really deep in here. But one reason these are important is there's this type of variable star called Cepheid variables, named after Delta Cephe, the fourth brightest star in the Cepheus constellation. And they were discovered early in the 20th century that. By a woman named Henrietta Leavitt, that if you look at the variability, the period of these Cepheid variables, you can relate that to the intrinsic brightness of these stars. And if you compare the intrinsic brightness, like the amount of energy coming out of the star to the apparent brightness, that gives you a way to determine the distance to those stars. And so that's what fundamentally what Hubble used to discover the expansion of the universe and kind of confirm this Big Bang idea and gave way to the basically birthed modern cosmology, where we're talking about expanding universe, Big Bang, dark energy, dark matter. That all comes from being able to locate and observe these variable stars with a particular periodic signal. And so if you're given an individual star, it's pretty easy to do that. You know, you just, you maybe compute a periodogram and you plot the star, fold it over a couple of the candidate periods, and you sort of look it by eye and see which one is best and decide whether you think it's, you know, really a periodic variable. And then you, you compare it to what a Cepheid light curve should look like and decide if it's a Cepheid star. Right? And this is the mode that a lot that up until the 90s, people did, they looked at each individual object. But in the, in the era of survey astronomy, we're looking at surveys like the Large Synoptic Survey Telescope that's coming online in a coup years. And this is going to have light curves for A billion stars. I hope I'm getting that right. I think it's like a billion. It'll be a billion candidate stars with 10 years of data. And you just can't scale those kind of like individual inspection methods up to that many objects. And when you start trying to do things like the Lomb Scargill periodogram and some of these period searches at scale, like tell a computer to just loop over the billion stars and do it for each of them, you end up with lots of issues of false positives, false negatives. You know, you're working with noisy data that the algorithm might not handle very well. So a real big thing in astronomy these days, I feel, is taking these tried and true methods from 10, 20, 30, 50 years ago and trying to figure out how you can apply them at scale, whether that's. Or apply them with the kind of data we have. It might, it's at scale, but it's also with heterogeneous data, with noisy data, with the various constraints of the survey, because you're no longer just observing, you're no longer using the telescope in exactly the way that you want for your particular data question. You have this, this data stream that's coming at you, and you can't, in, in many ways, you can't control exactly what, that, what sort of observations you get. So you have to modify your methods to work with this sort of noisy and heterogeneous data.

I get the impression that this is where the term data science kind of comes into play, as opposed to it being statistics or programming that when we have these huge streams of data. So as you said, data at scale, noisy data, heterogeneous data, as in a lot of other research disciplines and a lot of industries, it's when all these things come into play together, that kind of data science is formed, essentially.

Yeah, definitely. I mean, I think that's true, that it, you know, it's fundamentally for, for me in astronomy, data science is fundamentally about, about using these computational and statistical methods together to answer questions with data.

And essentially it's to do with the scale of the data as well.

Yeah, yeah. And, and data these days basically is synonymous with scale, right?

Absolutely. But you said you don't do so much astronomy these days.

Yeah, sort of transition. So I, I finished my PhD in 2012, and I did a short postdoc with the Large Synoptic Survey Telescope Group at University of Washington, and then I did a couple year NSF fellowship where I was actually in a computer science department and I was working on sort of Astronomy applications of database research for a while. And that led me into what my current position is, which is at the AT Science Institute. And our goal in the Escience Institute here at University of Washington is explicitly interdisciplinary. So we're working on connecting domain researchers like physicists and astronomers and oceanographers and geologists, connecting them with methods researchers like statisticians and computer scientists and data science, and also connecting them with each other so we can get that kind of dialogue forming. So because of that, I've sort of backed off on, like, doing 100% astronomy research. And I'm doing a lot of collaborations with people from different fields where because of my background, I end up being kind of the methodology expert. Right. So I can talk to them about what computing tools or what statistical tools or what machine learning algorithms might work well for their data. And it's been a lot of fun because, you know, as much as I love astronomy, I'm finding that I really love the software and I really love the methods, and I really love the breadth of what you can do with them.

And it sounds like you can speak to a lot of people working on different research questions in different disciplines and contribute to what they do as well.

Yeah, yeah. And that part's been really, really fun. So we have people on staff here who are data scientists and research scientists from a dozen different disciplines, and then the people I interact with on a daily basis, whether they're students or research scientists or postdocs or faculty, they're coming from around campus and all different backgrounds and yes. So, yeah, I get to contribute and have my hands in a lot of what goes on.

Now let's dive into a segment called Stack Overflow Diaries with datacamp curriculum lead Cara Wu. Hi, Kara.

Hi, Hugo. Things are going to get a little bit meta on Stack Overflow Diaries. Today we've got two complimentary questions from the R and Python sections of Stack Overflow, both on the question of how to make a good reproducible example. Links to both of these will be in the show notes.

So, Kara, tell us a bit about the importance of reproducible examples and their relevance to Stack Overflow.

If you've never asked a question on Stack Overflow before, it can be a little bit daunting. One of the most important things to do is create a reproducible example of the problem so that other people can try it out and see what's happening. Fortunately, there are some Stack Overflow questions that tell you how to ask a good Stack Overflow question.

That sounds like Inception.

That's exactly what it's like on the R side, there are a few components listed that are key to a reproducible example, a minimal data set, the minimal runnable code needed to reproduce the error, and information on the system the code is running on. You can construct sample data using functions like matrix or data frame, depending on the type of data in your problem. Another option is to use a data set that's built into R. Or you can use the DPUT function to paste part of your real data. But the key here is providing data that readers can use immediately by copying and pasting the R code that you provide.

But you don't want to include too much code, do you?

That's right, Hugo. For the code you really want it to be the minimal code necessary to reproduce the error. The more steps you include, the harder it is to identify what's relevant to solving the problem. It can sometimes be helpful to also include information on the system your code is running on, which you can do by pasting the output of the session info function. Lastly, there's an R package called Reprex, written by Jenny Bryan, which will help turn your R code into a reproducible example that you can post to Stack Overflow.

And how about on the Python side?

On the Python side, the principles are basically the same. The community recommends providing a dataset definition as runnable code, or as something that can be copied and pasted using Pandas Read Clipboard function. Don't reference data that readers don't have access to, though. Instead, provide some sample data that they can use. The data set should be as small as possible, as most questions can usually be solved with five or six rows of data, and too much data just distracts from the problem. After providing the data, you should describe your desired output and the code you've tried, as well as information on what other sources you've consulted. It's good practice to do your own research before you ask a question in case your issue has been addressed elsewhere.

That's really helpful, Kara. As you say, it can be daunting to ask questions on Stack Overflow, so it's cool that there are such resources to help.

It sure is.

Thanks Kara, once again for reading us a page from your Stack Overflow diary.

Always a pleasure, Hugo.

After that interlude, it's time to jump back into our chat with Jake and I saw and we've chatted previously about the fact you have an interdisciplinary data science seminar as well. And I looked at the recent list of talks and you have talks from social sciences, biology, astronomy, statistics, Urban planning. So it seems like a very vibrant culture you have there. I'm, I'm wondering what type of value do you see or what is the role of these discussions across disciplines in data science?

Yeah, so the main thing kind of historically, where this idea developed, the idea of escience as a cross disciplinary glue in some ways. I talked to some of the professors who've been around UW for 30 or 40 years and they often mention this thing that used to happen back when computers weren't on your desks, when computers were in the basement of the computer science building. And the way that you used a computer was you signed up for time and you went over there with your stack of punch cards or whatever and you hung out until you could run your algorithm. And one thing that apparently happened back then, I'm told, is that that brought these researchers from different places and different departments together. There was this water cooler where you'd chat with people and people would end up realizing that if you're an astronomer, astrophysicist, or you're an atmospheric scientist, you might be solving the same set of differential equations and be able to compare notes on the best numerical approaches to solving those, that sort of thing. So when this E Science Institute in its current guise was formed in 2014, that was one of the explicit goals is to kind of bring, bring back that water cooler culture. And that's, that's one of the goals of this, this interdisciplinary seminar we have. And we've, we've seen that come out. It's been, it's been really incredible. You know, I had one of my first projects in the Escience Institute. I was working with a geophysicist who was studying earthquakes under Mount St. Helens and had these, this huge array of sensors measuring these time series, like basically the, the shaking of the ground underneath Mount St. Helens and wanted to cluster them and figure out a way to cluster them in a way that was computationally feasible. And I started digging around and one of the, one of the methodologies that ended up being successful there was something that was published in an astronomy journal to cluster time series of variable stars or variable galaxies. So it's, you know, fundamentally what was going on as you're taking time series data and trying to find similar things in it. And it doesn't matter if that time series is coming from a telescope or from an earthquake monitor. Right. It's still data. And so we found those sorts of connections can be really, really fruitful.

I love that because that essentially is solving or working towards solving the problem of research occurring in silos and us constantly reinventing the wheel while our colleagues are doing or collaborators are doing similar things.

Yeah, and it's really funny. The kinds of the real barrier when you get people talking sometimes is vocabulary. One person's Gaussian process is another person's kriging and one person's principal component analysis is another person's Kern, Louvre decomposition or whatever. So as soon as you start figuring out that people are talking about the same thing, then you can really compare notes and learn from each other in a meaningful way.

Absolutely.

But you got to get people together first and start to understand that everyone really is doing some version of the same thing applied to different fields.

Yeah. And do you see these kind of aspects of silos happening between academia and industry as well? Could there be more of a conversation between academic research and data science and what happens in industrial place?

I think there definitely could. There's ways we can learn from each other, and that's another of the explicit goals of escience, is to build those academic industry connections and try to make it, instead of a one way door, make it more of a revolving door. So we have a couple people on staff, a couple of our data scientists who have come from industry roles, one who has worked in finance and worked in machine learning at Amazon and now she's here at escience advising students and working on interpretability of machine learning models. And we have another guy who has more software engineering background, but had a background in Google, Microsoft, research, places like this. And he's coming in and has been. It's really great to have around just in terms of talking about software design and how a researcher who wants to solve a problem in a way that other people can use it can approach designing their software and designing their tools. So I think absolutely we have things to learn in academia from industry and I hope it's true vice versa as well.

Something you spoke to, you discussed Escience in terms of being an interdisciplinary glue. And I know you're a huge fan of glue in general and you know you're very well known in the Python landscape and I thought maybe you could tell us about why you love Python so much and speak to it as a glue as you have before.

You know, I love it just for the reasons a lot of people say. I love its expressibility, the ease of working in it and trying things out. I've done some work in C and C and dabbled in FORTRAN a little bit and it always feels so stifling to get from Idea to implementation is a really long road, but in Python you just write the idea down, it's executable pseudocode and you have an implementation there and you know, you may need to go back and figure out how to optimize it and things like that. But in my experience, the development time, the savings of development time, are much more important than any savings and execution time that you might have if you start writing in C from the beginning. So that's one reason I like Python. But the thing that has really kept me going in Python is the community. To be honest, I love the Pycon community and the scipy community and the PI Data community, and just the fact that I can go online and read people's code and use it and contribute to it and I can post my own code and then people use it. The first time I saw someone giving a talk based on code that I had written and put out there, it was like this real high. It was like I was. It felt like I was doing something meaningful and doing something impactful. And I didn't. To be honest, I haven't always gotten that sense with my academic work. You know, you publish a paper and maybe a dozen people in your little subfield might read it and a few people might cite it the next year in their own work, but you know, you submit an algorithm to scikit Learn and all of a sudden the world is using it.

Yeah, absolutely. And I actually, I recall you tweeted once, and I'm probably going to misquote you entirely, but you tweeted something like, I'm trying to figure out what to do today. Am I going to go and do what I'm paid to do and write something that several people will get, or am I going to go and contribute to something open source and reach thousands of people in the next couple of weeks?

Yeah, I vaguely remember that. You might. You might be right.

But. So Python and all the packages you've contributed are open source and you've contributed to a huge, huge array from scikit Learn to scipy Astropy and now Altair as well. What place do you think open source software and the open source community at large have in what is now modern data science?

Well, from the academic perspective, I would say that open source plays a vital role when we're talking about academic data science. What we're really pushing for is people to use. To use software tools and to use statistical tools in a very rigorous way. And the best way to do that is to not have to reinvent the wheel every time you're doing something. If someone is going and like, where I got started in this, if someone wants to analyze astronomy data and use a manifold learning package, they shouldn't have to re implement manifold learning. You shouldn't be reading someone's paper and thinking, oh, I wonder if they got these details of this algorithm right. We should free researchers to focus on the bigger questions and those software engineering and kind of like specific algorithmic details should be taken care of for them. So in that sense, open source is really important. Like if I go and I see that someone is performing manifold learning and they're using the algorithms in Scikit Learn, I know exactly what code they're they're running, right? Whereas if someone says, you know, I wrote this C code and you can find it on my website, I'm not as confident, I'm not as immediately confident that what they're showing me in their plots is actually the algorithm as I understand it. Right. I'd have to go, I'd have to go to their website and dig through their code and see if I could find a bug and see if I'm comfortable with, you know, the way they approach things. So for, for open science, for, for science to progress in the way that we want it to, you know, people building on each other's, evaluating it and figuring out what the next step is. I think open source is vital and that's been one of the really nice. It's been one of the places where data science has really helped academic research is this open ethos as applied to science. And I gave a talk at PyCon this past summer and that was one of the big takeaways that I hope the audience got from my talk, is that the academic community has learned a lot from the Python open source community. And adopting the open source practices used by the Python community has really helped the academic community and in particular has helped the astronomy community.

And as you say, I mean, the open source software is totally open, totally transparent, reproducible and version controlled, which hopefully science will be at some point, right?

Yeah, yeah, hopefully. I mean, I think back to some of the early code that I wrote without version control and I have no idea how I managed to, to do it. Right. It's like emailing yourself a tarball at the end of the day so that when you change something the next day you can get back to it. Exactly. It's just absolutely ludicrous.

Is there a challenge in educating research scientists with respect to being computational, able to use programming languages and this type of software?

Yeah, definitely. It's a huge challenge and it's something that when I was a grad student was not really being seriously tackled by the universities. And it's still to some extent not being tackled. But we're in escience. That's another one of our goals, along with the other ones that I've chatted about, is we want to educate the campus on these data science tools. And so that means everything from offering software carpentry seminars to offering one off tutorial days where we tell people about AWS or tell people about other specific tools, things like this. One thing that I've been involved with is an interdisciplinary graduate course in software engineering. So we offer it through the computer science department, but computer science grad students are not allowed to take it. But it's designed for people from sciences to come in and learn version control and documentation and unit testing and all these things that, that are really needed for high quality, reproducible open code. But none of the departments really teach it very well. And the departments are in a difficult place. Right. Because if you think about, if you have a bunch of grad students in astronomy and you want to teach them how to do all these software engineering things, where do you fit it in? There's already a full schedule of classes and that list of subjects has been honed and fine tuned over the course of decades. And what do you, if you want to teach someone about software and machine learning? Do you drop stellar structures? Do you, do you drop, do you drop planetary science? Do you drop interstellar medium? Right. Like all those things sort of maybe, maybe sound like niche fields. But if you're in an astronomy program, you need to know about that stuff. You need to know about that to be able to talk to other people in the field world. And even more than that, there's each, each professor or each, each topic has a professor or two who's like, that's their, their area of their life's work. Right. So if you say, hey, you know, Professor X, we're gonna, we're gonna drop planetary science because we want to teach machine learning. Like you got this political problem too.

Exactly. And I think, yeah, that social and political question that every, every subject that's taught has its advocates and it's there for multiple reasons, is very difficult to navigate when trying to insert new things into a curriculum. So you mentioned version control unit testing as being two incredibly important things. What other tools and techniques would you encourage budding research scientists and data scientists to learn and play around with?

Well, I think the software engineering pieces are key. So definitely the version control and unit tests are big because that's a way to think. Make sure your code is the same as it was yesterday and make sure it works the same as it did yesterday. When things change. But just general software design, like learning about object oriented programming and whether that fits the particular problem that you're looking at. Learning how to write code that's readable, practicing that. Don't name all your variables, xyz, A, B, C all the time. And then things like code review. That's another thing we've experimented with here, is trying to get researchers to review each other's code and look at it and give feedback and especially cross discipline. You can often get some really interesting things there because you find out that what someone's done is they've done this homespun version of principal component analysis or something and you can point them to a package that does it much more efficiently at scale or something like that. So just the general software engineering practices I think are the biggest thing. You know, there's also the statistics and algorithm side of things, but in my experience people's. People have things to learn in both areas. Definitely. But the biggest weakness in academia is on the software engineering side more than the, the statistics and methods side.

Okay, and how much statistics and math do you think people really need to know to get up and running?

Yeah, that's a really good question. It's hard to say because you need all of these algorithms are somewhat mathematical. Linear algebra is way more important to me right now than I ever thought it would have been when I was in college. Yeah, it's hard to know how to best treat all that stuff because it's a, you know, it's a whole area of study in itself. So I guess I don't have a good answer for that.

Yeah, I agree. And I think linear algebra is definitely super important. A bit of calculus goes a long way. Whether people need to know multivariate calculus or not is a different question. But I always. These are things you can learn on the fly as well. Right. You don't need to have sat down and taken two years of linear algebra courses doing matrix row reduction and all of these things. Right. You can, it's kind of on a need to know basis.

Yeah, I think you can. Yeah, you can, you can learn it on the fly. But it does help to have that background. You know, I, I'm thinking about it like I did sort of learn linear algebra on the fly when I started on down the machine learning rabbit hole. But at the same time I'd had. I'd had some of it in the past because you. You touch on linear operators and quantum mechanics, and I think in an early math class, we actually talked about, you know, at least what a matrix and matrix multiplication is. Yeah.

So I think contained in that is the idea that it's really a language to become accustomed to.

Yeah, I think so. I think so, yeah. So you need some sort of background and you need some sort of vocabulary to jump into it, but I don't know what the best resources are for someone to learn that if they're coming from a background where they haven't studied that explicitly.

Let's now jump into a segment called Rich, Famous and Popular with Greg Wilson, who wrangles instructor training at datacamp. Hi, Greg.

G'. Day.

What do you have for us today, Greg?

I'd like to talk about diffing and merging spreadsheets and about how a little bit of engineering could help make data science more accessible to the 99% of humanity who aren't using version control today. Now, programmers often say mean things about spreadsheets and the people who use them, but a lot of their criticisms are misplaced. For example, you'll hear programmers say that spreadsheets are more error prone than code, but I've never seen any data showing that. Yes, lots of people make mistakes with spreadsheets, but lots of people make mistakes with Python and R as well. And so far as I know, no one has ever actually done a quantitative comparison of relative error rates. And yes, most people who use spreadsheets don't write rigorous tests and checks, but that's not a stone most programmers ought to cast. And when it comes to reproducibility, I'd argue that using a spreadsheet actually makes work more reproducible, since it guarantees that the data is actually there.

So why aren't we all using spreadsheets, then?

Well, one thing that programs do have in their favor is that they play nicely with version control. In particular, if you and I work in parallel on the same code, we can put our changes in subversion or Git or Mercurial, use diff to compare them, and then crucially, merge them in a structured way. We can't do that with spreadsheets. The closest we can get is to dump the content as a CSV file and then treat it as text. But then we lose the formulas, the formatting, the charts, and everything else.

So what are you proposing instead, Greg?

Well, Microsoft and LibreOffice both store spreadsheets and other documents as compressed XML. We know how to diff XML trees, and those tools know how to parse and render the content. All we need to do is weld the pieces together so that when version control detects conflicting changes in an Excel or Calc file, it launches a three pane spreadsheet merge tool instead of telling us that binary files differ.

That sounds like a lot of work.

I don't think so. The best estimate I've had is that three developers working for eight months could do a credible first version. And even if it takes twice that effort, it would allow literally millions of people, grant administrators, finance officers, and yes, data scientists to start using version control without throwing away the tools they're already familiar with. We'd be giving them a ramp to drive up instead of a cliff to climb. And honestly, I can't think of very many things that would help get more people started working in a more reproducible and more collaborative way.

Thank you very much, Greg. If anyone in the audience is interested in giving this a try, please get in touch. We'd love to hear from you. On top of this, if somebody out there does an analysis comparing the average number of errors in scientific papers produced by spreadsheets versus those produced by programming, send it to us. And if it checks out, we'll have you on the podcast to talk about it. Thanks, Greg, and looking forward to speaking with you again.

Thanks, Hugo.

Time to get straight back into our conversation with Jake. Something we've been revolving around is this idea of community of data scientists, community of open source developers. I actually saw you tweeted last week. I've got this in front of me, so I'm going to read it out. You wrote, okay, just want to put this out there. I'm not going to do an accent. Okay, just want to put this out there. I do my best to say yes to any request to grab coffee and chat about science careers, code life, etc. It's an explicit part of my role at U Dub, and one that I deeply enjoy. So my question around that is, what is the role of community in data science in general?

Well, I think community is important in science science more broadly. The way that you learn things is by networking with other people. This is why we go to conferences. Even though we could. We could all just watch YouTube videos and read each other's papers, but you still, at great expense, fly large swaths of people across the country to meet together. So it's super important, especially for someone who's just getting started, to be able to chat with someone who's been there before and face the specific challenges and the specific existential crises that come with being in a particular field. So I really find. I've found that to be incredibly important in my career and my life more generally. So I hope that I can be in that position to help people where I am right now. And when I say it's explicitly part of my role at uw, I mentor students and I mentor postdocs, and I'm trying to. Trying to be that for them. And we also have these open office hours at uw, which are kind of fun. So I sit at a desk in our common area for two hours a week on Tuesday mornings, and it's just wide open. It's advertised on the website, and anyone can come talk to me and ask about anything. And I. Because, you know, like I said, I'm typecast as a Python person. I generally get people coming in and asking me Python questions, which is. Which is always fun. But it's a cool part of my role, my role here. And I like to. The reason I put that tweet out there is I like to be able to do that more broadly than you do as well.

Well, yeah, And I think it's really important, particularly to bridge that gap between established working, practicing data scientists and beginners, because it cannot. Like, if you Google data science and look at the image search, you see, like, layers of Venn diagrams embedded in each other. And I think it can. Can be tough and scary. So I think efforts like this to reach out to. To. To beginners and let them know it's okay to. To reach out and encouraged is incredibly important.

Yeah.

On that note, actually, my El Salmon, who is a data scientist, she's worked in public health, she's big in the rstats community. She told me recently, I can't remember what the conference was, but there was a conference that had a buddy system where people coming to the conference for the first time would be paired up with established people and they'd, you know, hang out and walk around together and introduce people to each other and that type of stuff, which.

Yeah, that's a great idea.

It is, isn't it? Because if you go to a conference for the first time, it can be pretty intense, Right?

Yeah.

It reminds me of my first conference as a grad student. You know, I was. I started in the fall of 2016, and in January 2017 was the big American Astronomical Society meeting, and it happened to be in Seattle, so I just had to go. Go downtown. And I remember going there and Just feeling like. Like a complete outcast, you know, I didn't know anybody. And it started out with the social event at the first night, and everyone was there talking with people that they knew from all over the place. And I knew one professor that was there, so I just sort of followed her around like a lost puppy for a while until it was clear that she was kind of annoyed to have me there. But, yeah, I remember that. And it's. It's good to be reminded of that because I'm definitely in the place now where I go to the conference and I'm catching up with everybody that I've known for years. But I like that idea of an explicit buddy system to address that because it can be. Yeah, it can be anxiety inducing.

I think it can. And I think another thing.

Stand in the corner by yourself.

Exactly. Yeah. Yeah. Eating spring rolls. I think the birds of a feather tables help as well. And these, for those who don't know, these are tables where will sit and chat about machine learning or some type of modeling or something like that. And newbies are very encouraged to come and chat with experts there as well. So we've talked around a lot of ideas about data science in academic research. And I'm wondering in your mind, what are the biggest challenges facing the discipline of data science as we move forward?

Well, so we've touched on some of them. Right. There's challenges in just getting people the technical chops to do that, to use these data science skills within academia and use them effectively.

Absolutely.

But something that, as I get more broad outside of my astronomy world, it really seems like we need to spend some time thinking about data ethics as well. Right. Because there's a lot of issues when you start collecting data about everything in the world. You know, how do you keep people's data private? How do you make sure you're using it, using the data fairly? How do you train your algorithms in a way that aren't going to be biased and aren't. There's tons of stories about this in sort of the corporate, the tech world. And it's interesting coming from astronomy because we have a very specific set of patterns that are. Are built around like complete openness. Right. We publish all our data often publish all our data as soon as it's gathered. The photons are gathered from the telescope. But there's in other fields, you can't really do that. You know, when I work with people from biological sciences or particularly health sciences, like, there are issues with open data. So how do you do open science in A world where there are privacy concerns with the data. And one of the things that was really fun about my postdoc, I did the postdoc in the database group at the computer science department here. And at the time there were some folks there that were, I think they're still working on it, but they were working a lot on the idea of like differential privacy and how you, how you expose data about the world in an aggregated way that has some guarantees about the privacy of individuals. So I think some of those issues are going to be really increasingly important as these kind of data sets grow. And as data scientists, no matter what we're doing, we got to start thinking about, thinking about how the data that we're taking can be used and misused and things like that. One interesting little tidbit, I say that astronomy doesn't deal with these data privacy issues. But one thing that's interesting is that when you start doing surveys across the entire sky, a high cadence and publicizing that data, those surveys can be used to track objects that happen to be orbiting the Earth, right? And if you compare, if you find all the objects orbiting the Earth and compare it with a database of things that are publicly known to be orbiting the Earth, then all of a sudden you've found private or like secret government spy satellites like this. Right? So I know there are people in some of these surveys that have been talking to relevant government. You know, how do you, how do you screen the data so that it's scientifically useful but doesn't let foreign agents kind of sniff out military secrets? Right. So the amazing thing is even in astronomy we're getting to the point where big data is somewhat ethically challenged.

So we've discussed a lot of different applications of data science and different methodologies. What's one of your favorite data sciencey things to do? A technique or methodology or anything along that, those lines tool.

So yeah, so my, my all time favorite like machine learning method is principal component analysis. I, I just think it's like, it's like a Swiss army knife. You can do anything with it. And I'm probably in some way influenced by my PhD advisor by this. He in the mid-90s, wrote a paper that was really influential in astronomy that, that basically he, he and a grad student, I think wrote this paper that was basically applying principal component analysis to these Sloan spectra. And it was one of the first really at scale applications of principal component analysis in astronomy. So when I was a grad student, I quickly found that whenever I was going to my meeting with my thesis advisor and I had a new data set or something to look at. The first question he was going to ask me was like, well, did you do pca? So I adopted that as my first thing that I do with any data set. The amazing thing about it is you can use it for dimensionality reduction, you can use it for finding the most important contributors to any phenomena. You can look at the actual eigenvectors themselves for that. You can use it to reduce noise in data, you can use it to create a low dimensional representation of a high dimensional data set to get a, get a broad idea of the relationships. It's really this incredible tool that can do so many things.

And with scikit Learn, of course, for example, it's really easy to do pca, right?

Yeah, it's straightforward. You just plug in the data and go. Although there's some variants of PCA that are important that are not within scikit Learn like, like being able to use noisy data or weighted data or things like that.

So we've discussed moving forward what some of the biggest challenges facing data science as a discipline are. In general though, what does the future of data science look like to you?

The future, my hope is that it just keeps getting more and more open. We've seen with Python and R and Julia and some of these other tools, we've seen the power of open source software. It's really kind of snowballed, especially in the last 10 years. And the thing that gives me optimism is we're even seeing that in these academic communities around campus who are used to most more closed tools. Like I mentioned IDL earlier, when I started as a grad student in 2006, basically everybody in our department was using IDL, which is this proprietary language. You need a site license to run it, it, things like that. And at this point probably 90% or more of people in the astronomy department are using Python. So there's this gradient towards openness in the astronomy community and we're seeing that in other academic communities as well. So I, I have a real hope that this trend towards openness will, you know, eventually asymptote to 100% and everybody's code will be out there and ready to use and reproduce.

That's a great vision of the future. And in fact you mentioned your Pycon talk last year earlier and you had a great figure of the incredible growth of Python. The Python hockey stick, I think you called it. And in the show notes we'll link to that talk so our listeners can check it out.

Yeah, sounds Good.

So do you have a final call to action or a message you like to send to the data science community? Aspiring data scientists and well seasoned data scientists alike?

Yeah. You know, one thing I've been thinking about lately is. I think it's never too late to learn something new. You know, I would say if you want to learn something, just jump into it. You know, I was, I think about my own path and back when I was, when I was 20, I actually, you know, I thought my path in life was that I was going to go to a seminary and become a church pastor. That's, that's really what I thought at 20. And then at 25, I took my first astronomy course and I started, you know, learning that and learning coding. And it wasn't until I was 30 that I gave my first Python talk. And I think I'm best known for like Python talks now. But that, yeah, it wasn't until then. And now I keep learning things about software engineering and about these new machine learning techniques and deep learning and stuff like that. You don't have to be worried that you didn't study this stuff in college. Just jump in and learn it. And there are some incredible resources out there for doing that, whether they're books or videos or tutorials from conferences, or just grabbing coffee with some data science experiments expert.

That's great advice. And that leads me to wonder, is there anything new you're learning at the moment or anything in 2018 that you really want to learn?

Yeah, so one thing at the moment, I'm in kind of this fun thing right now. I just last week started 50% as a visiting researcher at Google. So one of the huge things that I'm learning right now is what it looks like to work with a large team that are all focused on the same software and in the same place. I've done a little bit of that in kind of distributed open source land, working with Scikit learn developers around the world. But I've never been in a situation where I go to the office each day and sit down with the people that I'm collaborating with on a particular software project. And I'm so excited to learn what software development looks like in that context and to see what sort of lessons I can bring back to the academic community and to the open source community.

Wow, that sounds really exciting. And that once again speaks to this interplay we discussed earlier between industry and academic data science research.

Yeah. And hopefully those connections will continue growing. Right now, if you're an academic kind of considering the future and thinking about industry. It really feels like a one way door. Right. Like once you're out, it's hard to break back in. But I'd love it if we could have more of a revolving door, more opportunities for people who have expertise on the industry side to come in and help scientists and vice versa.

Yeah, absolutely. And we'll have to have you back on the show to talk about your work at Google once you've been there for a while.

Yeah, that'll be fun. I'm almost two weeks into it now, so it's still young and I'm. I still have a lot to learn.

Sure. Jake, it was an absolute pleasure having you on the show.

Yeah, thanks so much.

Thanks for joining Our conversation with Jake Vanderplass about the role of data science in astronomy and academic research at large. We saw the challenges faced by not only the amount of data collected and streaming in, but also the different types of data that are collected these days. We discussed the importance of the open source development work world, the role it plays in scientific research, the need for community in science, and the need for a more serious conversation about data ethics. Make sure to check out our next episode, a conversation with Emily Robinson, a data analyst at Etsy. Emily and I will be talking about online experimentation at Etsy, an e commerce website focused on handmade and vintage items and supplies, and how data analysis and data science are essential to their business us. We'll also chat about much more, but you'll have to wait till next week to find out. I'm your host, Hugo Bown Anderson. You can follow me on Twitter Hugo Bowne and datacamp. At datacamp you can find all our episodes and show notes online@datacamp.com Community podcast. Sam.