#49 Data Science Tool Building

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI Transformation Journey on datacamp.com.

Wow. In this episode of Data Framed, a Data Camp podcast, I'm Speaking with Wes McKinney, creator of the Pandas project for data analysis tools in Python and author of Python for data Analysis, among many other things. Wes and I will talk about data science tool building, what it took to get Pandas off the ground, and how he approaches building human interfaces to data to make individuals more productive. On top of this, we'll talk about the future of data science tooling, including the Apache Arrow project, and how it can facilitate this future, the importance of data frames that are portable between programming languages and building tools that facilitate data analysis work in the big data limit. Pandas initially arose from Wes noticing that people were nowhere near as productive as they could be due to a lack of tooling, and the projects he's working on today, which we'll discuss, arise from the same place and present a bold.

Vision for the future.

Wes also makes clear that Pandas is not a one human show, and when people thank him for his work, he reminds them to thank Jeff Reback, Joris Vanden Boscher, Philip Cloud and Tom Augsberger, along with the other Pandas core developers that have really been driving the project forward over the last five years. I for one want to thank all.

Of them finally, and not to give too much away.

We'll also discuss the challenges of open source software development and how Wes is approaching funding and resourcing OSS with his most recent venture, Ursa Labs. Find out all this and more, including.

How much of Pandas was developed in.

A small East Village apartment that Wes may or may not have cohabitated with mice.

Well, I can't give too much away. This is just the opening monologue.

I'm Hugo Bowen Anderson, a data scientist at Data Camp, and this is Data Framed.

Welcome to Dataframe, a weekly Data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on twitterbown and datacamp at datacamp, you can find all our episodes and show notes@datacamp.com Community podcast.

Listeners, as always, check out the show notes for more material on the conversation today. I've also included a survey in the show notes and a link to a form where you can make suggestions for future episodes. I'd really appreciate it if you take the survey so I can make sure that we're producing episodes that you want to hear. Now back to our regularly scheduled programming in Python.

Hi there, Wes, and welcome to Data Framed.

Thank you. Thanks for having me.

Great pleasure to have you on the show and I'm really excited to have you here today to talk about open source software development, to talk about your work at Ursa Labs, Apache Arrow, a bunch of other things to do with tool building. But first I'd like to find out a bit about you. Perhaps you could open by telling me what you're known for in the data community.

Sure, yeah. I mean, I'm best known for being the original author of the Python Pandas project, which I understand that a lot of people use. So I started building that as a closed source library a little over 10 years ago, and I've been working on a lot of different open source projects for the Python data science world and beyond. I also wrote a book called Python for Data Analysis, which is now in its second edition, and I think that's become a pretty, pretty ubiquitous resource for people that are getting into the field of data science and are wanting to learn how to use Pandas and, you know, get their feet wet with working with data.

And congrats on the second edition. That was in the past year or so that that was released, right?

Yeah, it was, yeah. The end of, it was just about a year ago, the end of 2017.

And how did you get into data science tool building originally? Because I'm aware that your background wasn't in CS per se.

Right. I was a mathematician, so I studied pure math at mit. I did a little bit of computer science. I had had some exposure to the world of machine learning in that I was aware that it existed. But MIT didn't have a statistics program, so data analysis and statistics wasn't very familiar to me when I was entering the working world and I, I got a job at a investment manager called AQR Capital Management, which is based in Greenwich, Connecticut. And there were a number of MIT grads that had gone to work there and some of them were math majors and they kind of sold Me on the idea of getting experience with applied math and then maybe I would go back to grad school later on. And I found that in my job there, that rather than doing very much applied math, that I was really doing a lot of data munging. So I was writing SQL, I was using Excel, and really I just found that I wasn't as productive and efficient working with the data as I felt like I should have been. And part of it was like, well, okay, I'm just, just starting out my career. I'm 22 years old, you know, what do I know? But I looked around and even. Even at people who were a lot more senior to me and a lot more experienced, and it seemed like they weren't very productive either. And they were spending a lot of time, you know, obviously their skill with Excel and Excel shortcuts and so forth, keyboard shortcuts was a lot better than mine. But still, it seemed like there was just something missing to kind of working with data. And I started to learn, I started to learn R at the end of 2007, beginning of 2008, and there were at that point in time, you know, the R ecosystem was a lot less mature. And it felt like, you know, an interesting language for, you know, valuable language for doing statistics and data analysis, but we also needed to kind of build software. And so I learned a little bit of Python and thought like, wow, this is a really easy to use programming language. I had done some Java programming and thought that I just wasn't very good at Java. And so I thought, man, I'm just not cut out for building software. But I decided to have a tinker with building some data manipulation tools in Python. That was March, April 2008, and, you know, just went down the rabbit hole from there. And once I had made myself more productive working with data, I started evangelizing the tools I was building to my colleagues. And, you know, I kept pulling on one thread and ended up becoming more of a software engineer than a finance or, you know, a finance or math person.

Yeah, there are a lot of interesting touch points there. For example, your background in pure math and that you're in Connecticut. I actually, I was working in pure math and ended up doing applied maths in a biology lab in New Haven, Connecticut. Not, not in Greenwich. But at that point I actually started dealing with data a lot. And that's when I started getting into data science also. It's also interesting that Pandas, when you first developed it, was closed source. But before we get there, you've spoken a bit to, you know, why, why you chose Python. Could you explain a bit more about what was attractive about Python then? Because, of course, one of the, a lot of the attractive things for researchers and data scientists now about Python is the data science stack. You know, Pandas, scikit, Learn, Numpy, all of these things. So what, what made you really like it back in the day?

Yeah, I mean, at that point of time, 2007, 2008, in terms of doing statistical computing, Python was not. Let's think of it as a promising world that has not yet been terraformed. So think that there were kind of the nuts and bolts of a really interesting environment. I Learned about the IPython project and said, okay, here's a really nice interactive shell where you can plot things and it has tab completion and really basic interactive affordances that really help out a lot. You had the nuts and bolts of doing all the analytical computing that you need to do for data manipulation. NumPy had its 1.0 release, I think, in 2006, and had become a mature project. And the scientific Python world was defragmenting itself after the Numeric Rift, which had persisted for several years. And Travis Oliphant had sort of work to bring those communities together. But really, I think what attracted me to the language was the accessibility and the fact that it was really very suited for interactive and exploratory computing that you could. You didn't have to set up an elaborate development environment, you know, an ide, to be able to get up and running, doing some really basic things. And so having had experience with Java, I think one of the things that put me off about Java was the elaborateness of the environment, that you need to really be productive, like you really need to set up an IDE and there's all this tooling that you need to do. Whereas with Python, you could do, you know, some pretty complex things with a few lines of code and a text file and then you just, you run the script. So that kind of like interactive scripting feel of doing exploratory computing was really compelling to me at the time, but obviously it was. Python was missing a lot of tools, and so it was a bit daunting to start the process of building some of those tools from scratch.

Yeah, and you mentioned IPython and NumPy and Travis, and I suppose, you know, this is the time where John Hunter was working a lot on matplotlib and working with Fernando to incorporate it. With IPython, there was a lot of close collaboration. I suppose this speaks to the idea of community as well. And did you find the scientific Python community something that was also attractive?

Yeah, well, you know, I didn't have much interaction with the community until much later. I think the first person, there's two people that I met from the Python community who are like my first point of contact with that world, so. So one person is Eric Jones, who is a founder of Enthought, which is the original, like Python scientific computing company based in Austin, Texas.

And they also run the SciPy conference.

Yeah, so they run SciPy. And so enthought was doing a lot of. I was doing a lot of consulting work in New York City with financial firms that were getting big into Python during that era, like training and kind of custom development. And I got in touch with eric sometime during 2009 and sort of gave him like kind of the very first external demo of Pandas. And this was right around the time that we were getting ready to publish the Pandas bits on Pypi and so forth, kind of the first open source version of the project. And then the second person I met was John Hunter himself from, from Matplotlib. I met him in Chicago in January 2010. You know, at that point I was looking around for like how to engage with the Python world, having, you know, just open sourced Pandas. And because John was working, he worked for Tradelink and up until his, his death in 2012. But he was a quant there, having been a neuroscientist and kind of had been building matplotlib for many years. He, he kind of took me under his wing as kind of, I was his, he was my mentor for, you know, for a couple of years and kind of helped me enter and get involved in the community. And so I definitely feel that the, I found, found it a very warm and kind of inviting community, very collaborative and collegial. And I think I was attracted to that, you know, that feeling. You know, it didn't seem like a lot of people competing with each other. It was really just a lot of pragmatic software developers looking to kind of build tools that were useful and to help each other, help each other succeed.

Yeah. And you actually still get the sense of that when you go to scipy in Austin, Texas every July or every second year, you still get a strong sense of community and people just loving building the tools together.

Yeah, yeah, totally, totally. I mean, obviously the community has grown much bigger and I think the ratio of project developers, people working on the open source projects to the users, that ratio has certainly changed a lot and that there are a lot more users now than there are developers. The very first SciPy conference was probably the majority of people. There were people who are the developers of open source projects. But you know, still I think it's a great community and I think that's, that's helping kind of continue to bring people into the ecosystem.

Yeah, and actually I had Brian Granger on the podcast recently and we discussed, as you know, several people are discussing at the moment, that we're now kind of entering a phase transition from having individual users spread across orgs and spread across the globe of a lot of open source packages to actually having large scale institutional adoption.

Right, yeah, yeah, for sure.

And I'm wondering, in terms of Pandas starting off as a project, I'm under the impression it was started as a tool to be used in finance.

Is that the case?

Yeah, I mean it was focused. So if you look, you can go back and download Pandas 0.1, which was published to Pypi in December 2009 and see what was in the library. And compared with now, the functionality was a lot more geared towards time series data and the kinds of problems that we were dealing with back at aqr. I wouldn't say that it necessarily is finance specific. It was very general data manipulation. It was a pretty small project back then, but it was just about dealing with tabular data, dealing with messy data, data munging, kind of data alignment, essentially kind of all those like kind of really basic wrangling and data integration problems. It wasn't really until 2011, 2012 that the project got built like I built the project out and created a more comprehensive set, relational algebra facilities like it didn't have complete joins, like all the different kinds of Basic joins until 2011. So its feature set was certainly skewed by the use cases that we had in front of us back at aqr.

And how did you get the project off the ground? I know that's a relatively ill formed question, but just in terms of hours and people and resources, well, you know.

You smelt metal or you kind of, you forged weapons. You have to get the crucible really, really hot. So we open sourced the project at the end of 2009 and I think we had deliberated kind of the whether or not to open source at all for about six months or so and ultimately powers that be decided that we would open source Pandas and sort of see what would happen. I gave my very first talk about Pandas, which you can still find online at PyCon 2010 in Atlanta. And it was about using. The subject of the talk was about using Python and quantitative finance, but the project didn't really go anywhere after that. So it was hosted on Google code. This was, you know, GitHub existed, but it was kind of a Ruby thing at that time. And I left AQR to go back to grad school. I went to Duke to start a PhD in statistics, statistical science, it's called there. And I continued to do a little bit of contract work developing Pandas for aqr. And somewhere I don't. I think the catalyst for me was in early 2011, I started to get contacted by more companies that were exploring using Python for data analysis use cases. And they had seen my talk at PyCon and were interested in getting my perspective on statistical computing. And I just had this feeling that the ecosystem was facing a sort of existential crisis about whether or not it was going to become truly relevant for doing statistics. And it was clear to me that Pandas was promising, but really had not reached a level of functional completeness or usefulness to be the foundation of a statistical computing ecosystem in Python. And so I guess I felt that feeling so strongly that I, you know, I sort of had like an epiphany where it wasn't quite like, you know, shouting Eureka and jumping out of the bathtub, but I emailed my advisor and said, hey, I would like to take, take a year off from my PhD and go explore this Python programming stuff and we'll see how it goes. And I had some money saved from my first job and I, I moved back to New York into a tiny apartment in the East Village, which had mice and stuff, really not the best place I've ever lived. But I, you know, essentially was like, I'm just going to work full time on Pandas for a while and build it out and see, see what happens. And I think that's when, as soon as I started kind of socializing the functionality of Pandas and filling in feature gaps, you know, implementing joins and fixing some of the internal issues, of course I created other internal problems, but there were some, there were definitely some design problems in the early versions of Pandas that got fixed in the summer of 2011. But as soon as pandas could read CSV files pretty reliably and could do joins and a lot of the basic stuff that you need to be productive working with multiple data sets, I think that's when it started to catch people's eye toward the end of 2011 and start to take off the ground. So around the same time, I pitched the idea of a data analysis book in Python to O'Reilly and they agreed to do a book, which, thinking back on it, was a bit risky because, you know, who knows what would have become of Handis was not at all a prom, you know, not obviously going to be successful back in 2011. So they decided to take a bet. And so much so that, you know, I asked them later why they didn't put a panda on the COVID but they said, well, we're saving the panda for something really big. And so it wasn't even clear then that Python and pandas and everything was going to be a popular thing. So kind of, it's important to kind of have that perspective.

We'll jump right back into our interview with Wes after a short segment.

Now it's time for a segment called Data Science Best Practices. I'm here with Ben Skranker, an independent data science consultant. Hi Ben.

Hey Hugo. It's great to be back.

So what are we discussing this week?

Well, do you need to explain or to predict?

What do you mean?

Bremen has this must read paper called Statistical the Two Cultures. He discusses that there are two modeling approaches, the algorithmic modeling culture and the data modeling culture. By algorithmic modeling he means that the machine learning approach, which has largely been developed by computer scientists. He contrasts this with the data modeling culture, which views data as stochastic and worries about modeling the data generating process. Economists and statisticians tend to fall into the latter camp. Bremen argues that machine learning is dominant in terms of performance, accuracy and ease of use.

Is that all there is to it?

If only life were that easy for a large class of problems. He is right and traditional data modelers are rushing to adopt these methods. But the algorithmic approach fails for another large class of problems, those where you need to explain the problem, that is to understand a causal connection. Galit Shmueli has done some great research in this regard. She points out that you need to choose your method and approach based on whether you want to explain or to predict.

I see.

So use machine learning to predict and statistics or econometrics to explain.

That is pretty much the case. If your problems focus on prediction, then ML is the place to start and is incredibly powerful. Both shallow and deep learning models are producing incredible results for predictive and perceptual problems. But often we need to understand the drivers that affect a business problem. In this case, we must run an experiment or perform causal regression analysis to eliminate bias in our estimated effect sizes.

So how do you know which to use.

An experiment is the gold standard because of the magic of random assignment. But you may not be able to run an experiment. It could be too expensive, take too long, or be dangerous, such as something a human subjects committee would not allow. If you can't get experimental data, then you must use observational data and perform a causal regression analysis which captures the key features of the data to eliminate bias.

So tell me more about bias.

We really need to talk about endogeneity in depth at some point. Endogeneity is all the different ways bad stuff can be hidden in your error term, which biases results such as sample selection omitted, variable bias, simultaneity, and measurement error. For example, if you want to reduce churn, you need to build an explanatory model to determine how different levers affect churn. In this case, the data is censored because some customers remain customers past the end of our study. We never observe them churning. Consequently, we need to model the censoring process. Machine learning will have a difficult time learning how to compensate for censoring. A classical survival model has sensory baked into the hypothesis space and will almost surely work better. Finally, I should add that including endogenous features in a machine learning model can cause all kinds of problems. If you treat machine learning algorithms as black boxes, you may regret it. If your model needs to be retrained regularly, you may have this problem.

And are there other benefits to the data modeling approach?

Another huge benefit of the data modeling approach is is that you can do inference either frequent disturbation. That means you can formally state hypotheses about which levers matter and then test how likely it is that the data supports your hypothesis.

Thanks Ben for explaining why prediction isn't a silver bullet and why we also need models to explain our data.

After that interlude, it's time to jump back into our chat with Wes McKinney.

So when living in the East Village supporting yourself to build out the package, did you have any inkling that it would achieve the growth and wide scale adoption that it has?

No, not really. I mean I believed that. I mean obviously I had the belief that Python ecosystem had a lot of potential and that projects like Pandas were necessary to help the language and the community realize the potential. Like I think there was a lot of computational firepower in the numpy world and all the tooling cython and tools for interoperability with native code and so. So I just wanted to help realize that potential but I didn't really have a sense of where go there were some other significant kind of confluence of things that happened, particularly when you consider the development of stats models and scikit Learn, which brought meaningful analytical functionality to Python. Like, I think if Pandas, you know, really the big thing that made Pandas successful was the fact that it could read CSV files reliably. And so it became like a first port of entry for, for data into Python and for kind of data cleaning and data preparation. And so if you wanted to do Machine learning in SciKit Learn or you wanted to use stats models for statistics and econometrics, you needed to clean data first. And so using Pandas was the obvious choice for that. But yeah, it wasn't obvious. And you know, I recruited a couple of my former colleagues from aqr, Adam Klein and Changsh to work with me on Pandas and we explored starting a company around financial analytics in Python powered by Pandas. But we were focused on building out Pandas as an open source project kind of. While we explored kind of that startup idea. Ultimately we didn't pursue that startup, but we, it was clear that by mid-2012 that we'd sort of crossed the critical horizon of people being interested in Python as a language for data analysis.

And since then you've found certain institutions which have employed you in order to work on Pandas, right?

I wouldn't say that. Outside of my time at aqr, when I was building Pandas kind of initially, I've never been employed directly to work on Pandas. I started a company called Datapad with Changsha and so it was a venture backed company and we were building a visual analytics product that was powered by Pandas and other Python. The DataPad was acquired by Cloudera at the end of 2014. And so Chong and I landed there to work on. And my role at Cloudera was to look holistically at the big data world and figure out how to forge a better path for Python and data science tools in general in the context of the big data world. That's the Hadoop ecosystem and Spark and kind of all the technology that was largely Java based, which had been developed since, you know, 2006 or 2008. And so. But I wasn't working on Pandas in particular at that point. I sort of had taken stock of the structural and kind of infrastructural problems that Pandas had. I gave a talk at the end of 2013 at PI Data New York and the title of the talk was Practical Medium Data analytics in Python. And the subtitle of the talk was 10 things I hate about Pandas, I remember. So I had this kind of, in the background, this feeling that Pandas was built on a fantastic platform for scientific computing and numerical computing. So if you're doing particle physics or HPC work in a national lab with a supercomputer, you know, Python is really great and that's how the ecosystem developed in the late 90s, early 2000s. But for statistical computing and big data and analytics, fact that like strings and categorical data wasn't a first class citizen in that world made things a lot harder. Missing data was not a first class citizen. And so there were a lot of problems that had accumulated. And so at that point I started to look beyond Pandas as it was implemented then into kind of how we could build technology to advance the whole ecosystem and beyond the Python world as well.

So I think a through line in this is really encapsulated by a statement you made earlier, which is you want to build technologies and tools that are truly relevant for doing statistics or working with data. And I know as a tool builder you're committed to developing human interfaces to data to make individuals more productive. And I think that actually provides a really nice segue into a lot of what you're thinking about now, in particular the Apache Arrow project. So I'm wondering if you can tell me about Apache Arrow and how you feel it can facilitate data science work.

Yeah, so I got involved in, in what became the Apache Arrow project, you know, as part of my work at Cloudera. So one problem that had plagued me as a Python programmer was the fact that when you arrive at foreign data and foreign systems that you want to pull plug into, whether those are other kinds of, you know, ways of storing data or accessing data or accessing computational systems that we were in a position of having to build custom data connectors for Python, for Pandas, or kind of whatever Python library you're using. And so I felt that we were losing a lot of energy to building custom connectors into all of these different things. And this problem isn't unique to Python. So if you look at all of the number of different pairwise adapters that are available to convert between one data format and another, or serialize data from one programming language to another programming language. So sharing data was something that had caused me a lot of pain. And also sharing code and algorithms was a big problem. So the way that Pandas is implemented internally, it has its own custom way of representing data that's layered on top of numpy arrays. But we had to essentially re implement all of our own algorithms and data access layers from scratch. You know, we'd implemented our own CSV reader, our own interfaces to HDFI files, our own interfaces to JSON data. We have pretty large libraries of code in Pandas for doing in memory analytics, aggregating arrays, performing group by operations. And if you look across other parts of the big data world, you see the same kinds of things implemented in many different ways in many different programming languages. In R you have the same thing, many of the same things implemented in R. So I was kind of trying to make sense of all of that energy loss to sharing data and sharing code and thinking about how I could help enable the data world to become a lot less fragmented and people building systems who are people like me, who build tools for people, how to make people like me who are building tools a lot more productive and able to build better and more efficient data processing tools in the future. And so this was just kind of feelings that I had. And so I started to poke around Cloudera and see if other people felt the same way. And so I was working with folks on the Impala team, people like Marcel Kornacker who started the Impala project, Todd Lipkon who started the Apache Kudu project. It's now Apache Impala joined the Apache Foundation. So, you know, there were a lot of people at Cloudera that essentially agreed with me. And we sort of thought about like, what kind of technology we could build help improve interoperability. And we sort of centered on the problem of representing data frames and tabular data. And as we kind of looked outside of Cloudera, we saw that there were other groups of developers who concurrently were thinking about the exact same problem. So we bumped into folks from the Apache Drill project, which is a SQL on Hadoop system, and they were also thinking about the tabular data interoperability problem. Like how can we move around tabular data sets and reuse algorithms and code and data without so much conversion and energy loss. And so very quickly, you know, we got 20, 25 people in the room representing 12 or 13 open source projects with a general consensus that we should build some technology to proverbially, you know, tie the room together. That became Apache Arrow. But it took all of 2015 to put the project together. Now how is all this relevant to data science? Well, what the Aero project provides is a way of representing data in memory that is language agnostic and standardized and portable. So you can think of it as being like a language independent data frame. So if you create arrow based data frames in Python, you can share them with any system, whether that's written in C OR C or JavaScript or Java or Rust or Go. As long as they implement the arrow columnar format, they can interact with that data without having to convert it or serialize to some kind of intermediate representation like you usually have. So the goal of the project, in addition to providing high quality libraries for building data science tools and building databases, is also to improve the portability of code and data between languages. Outside of kind of the interoperability side of the project, there's also the goal within the walls of a particular data processing system to provide a platform of algorithms and tools for memory management and data access that can accelerate large scale data processing. So we wanted the arrow columnar format to support working with much larger quantities of data, the single node scale data that is particularly data that does not fit into memory.

And I love this idea of tying the room together, as you put it, because essentially it speaks to the idea of kind of breaking down the walls between all these silos that exist as well.

Right, yeah, yeah. No, I mean, I think if you look across just within the data science world, I mean, even though functionally we're solving many of the same problems, like there's very little collaboration that happens between the communities, whether collaborating at the software design level or at the code level. And as a result, people point fingers and accuse people of reinventing wheels or like not wanting to collaborate. But really, if your data is different in memory, there is just no basis for code sharing in most cases. And so the desire to create an open standard for data frames is just, if you want to share code, it's essential. You have to standardize the representation in RAM or on the GPU or essentially at the byte or the bit level, agreeing on what the data looks like once you load it off disk or once you parse it out of a CSV file is the basis of collaboration amongst multiple programming languages or amongst different data science languages that are ultimately based in C or C. Yeah, I remember.

Actually Fernando Perez spoke to this as well in his keynote, where you also keynoted the inaugural JupiterCon saying, We welcome so many contributions, but we need to agree on some things. Right? And these are certain things that we've all agreed upon. So if you're going to contribute, let's build on these particular things.

Right, right, yeah. No, I think the Jupyter project certainly socialized this idea of open standards by developing the kernel protocol providing a way. It's like, you know, here's the abstract notion of like A computational notebook. And here's how if you want to build a kernel, add a new language to the Jupyter ecosystem, you know, here's how you do it. And you know, that certainly has played out beautifully with, you know, I think it's like over 40 languages have kernel implementation for Jupyter. But you know, I think in general I think people are appreciating more the value of having open standards where that are community developed and that are developed on the basis of consensus and where there's just like kind of broad buy in. It's not, it's like one developer or one sort of isolated group of people building some technology and then trying to get people to adopt it. So I think Jupyter is somewhat, is unique in the sense that it started out in the Python world, but I think it's there, you know, they set out with the goal of embracing a much broader community of users and developers and that's played out in really exciting ways.

I really like the descriptions you gave and kind of the inspiration behind the Arrow project in particular, you know, the need for interoperability, the importance of these portable data frames. I don't want to go too far down the rabbit hole. I can't really help myself though. I'd like you to speak just a bit more to kind of your thoughts behind the challenge of working in the big data limit. I mean, for example, that we have computers and hard drives that can store a lot of stuff, but we don't actually have languages that can interact with unless we parallelize it.

Right, right, right. So a common thing that I've heard over the years from people who say, Wes, like I just want to write Pandas code, but I want it to work with big data. It's a complicated thing because the way that a lot of these libraries are designed, the way the Pandas is designed and a lot of libraries that are similar to Pandas, it's the implementation and kind of the evaluate like the computational model, like when computation happens, like what are the semantics of the code that you're writing? There's a lot of built in assumptions around like the idea that data fits in memory and that when you write A plus B that like A plus B is evaluated immediately and materialized in memory. And so if you want to scale out kind of scale up computing to data frame libraries, you essentially have to re architect around the idea of deferred evaluation and essentially defining kind of a rich enough algebra or kind of intermediate representation of analytical computation where you can actually use a proper query engine. Or a query planner to execute operations. And so really what is needed is to make libraries like Pandas internally more like analytic databases. And if you look at all the innovation that has happened in the analytic database world over the last 20 years with columnar databases and things that have happened in the big data world, you know, very little of that, of that innovation in scalable data processing has made its way into the hands of data scientists. So really, you know, one of my major goals with working, my involvement in the Arrow project is to provide the basis for collaboration between the database and analytic database world and the data science world, which is just not something that's happened before. Ultimately the goal is to create like an embedded analytic database that is language independent and can be used in Python, can be used in R, that can work with much larger quantities of data. But it's going to take like a different approach in terms of the user API because I think this idea of like magically retrofitting Pandas or essentially retrofitting Pandas with the ability to work with hundreds of gigabytes of data or terabytes of data, I hate to say it's a little bit of a pipe dream. I think it's going to require some breaking changes and some kind of, some different approaches to the problem. That's not to say that Pandas is going away. I mean Pandas is not going anywhere. And I think is certainly is occupying the sweet spot of being like kind of the ultimate Swiss army knife for data sets under a few gigabytes.

So does this conversation relate to the murmurings we've heard of potential Pandas 2 in the pipeline?

Yeah, so we, I think at the end of 2015 I started a discussion in the Pandas community. And so just FYI, I think people, you know, are often thanking, I see people out in the community, they're like, Wes, you know, thanks so much for Pandas. I have to remind them, like to go out of your way and thank Jeff Reback and Joris Van Den Bosch and Philip Cloud and Tom Augsberger and the, you know, the other Pandas core developers that have really been driving the project forward over the last five years. I haven't been very involved in the day to day development since sometime in 2013, but at the end of 2015 I started spending some more time with the Pandas developers. Said been building this project for it's seven years, you know, it's a little over seven years old. The code base. Are there things that we would like to fix? Like what are we going to do about the performance and memory use and scalability issues. I can't remember, I don't think at that point I don't know that Dask data frame existed. And so Dask has provided kind of an alternative route to scaling Pandas by using Pandas kind of as is, but essentially re implementing Pandas operations using a Dask computation graph. But looking at the kind of single node scale, kind of the in memory side of Pandas, we sort of looked at, you know, what we'd like to fix about the Pandas internals and that was what we described as the kind of pandas2 initiative. And around that time we were just getting ready to kick off the Apache Arrow project. And so I wouldn't say that we, you know, we reached kind of like a fully baked, you know, game plan in terms of how to create quote unquote pandas2. But I think we reached some consensus that we would like to build a an evolved data frame library that is a lot simpler in its functionality. So shedding some of the baggage of multi indexes and some of the things in Pandas that are can be a bit complex and also don't lend themselves very well to out of core, you know, on like very large, not don't fit into memory data sets, but something that's focused on dealing with the very large data sets at a single node scale. So large out of core, just big data sets on a laptop. So we are, I mean we are working on that and you know, I think the project itself is not going to be called Pandas 2 just to kind of not confuse people. I think the Pandas project is we all got together, the Pandas team, we all got together in Austin over the summer and this is one of the topics that, you know, we're going to continue to grow and kind of innovate and evolve the current Pandas project kind of not as it is right now, but my goal is to grow a parallel kind of companion project which is powered by the Apache Arrow ecosystem and provides a Pandas like user experience in terms of usability and functionality, but is really focused on kind of powering through very large on disk data sets.

We'll jump right back into our interview.

With Wes after a short segment.

Let's now dive into a segment called Studies in Interpretability.

I'm here with Patter Coyle, machine learning.

Engineer and one of the core developers of the open source statistical modeling platform PyMC3.

Great to have you on the show, Patter.

Thanks for having me, Hugo.

So we're Here to talk about interpretability in building machine learning models and in.

Data science more generally, Interpretability is telling you why a model makes certain decisions. And this is important. But it's more important in some areas than others, right? I mean, it'll be more important in insurance and healthcare, for example, than in ad tech, a space that you've worked in.

Yes, Hugo, it's fair to say that interpretability matters less than ad tech. The cost of showing a wrong ad is very different to, say, the cost of mispricing an insurance policy.

Can you speak to this a bit.

More from your perspective?

Yeah. So in ad tech, the models I worked on largely involved lots of clever feature engineering and the deployed models were really logistic regression due to the fact that they are easy to parallelize. So in adtech we care more about things like predictive accuracy because that's tied directly to the economic impact. We don't care as much about explaining the model to internal customers or regulators.

I mostly agree. However, you could imagine an algorithm that shows wealthy teenagers ads for colleges, but shows minorities ads for bail bondsmen. Having said that, in finance and insurance, being able to explain models matters a lot, right?

Right.

The cost of a mistake in credit risk models is very high. You've loaned, for example, to a customer or client who defaults. I think that as we see more applications of AI or ML in insurance, healthcare and other regulated industries, we need to be more mindful of that.

So can you comment on some work you've seen in those industries?

Well, sadly, some of the best work I've seen has been under a non disclosure agreement or NDA. One example I saw was a model for predicting credit risk for loans. The model itself was a random force with LIME on top of it. For those of you who don't know, LIME stands for Locally Interpretable Model Agnostic Explanations. And LIME is basically a toolbox that allows you to get explainable outputs from your black box models. In the credit risk model case, it was easy to build a framework for handling customer requests in regards why they were flagged up as at risk of default. And it was able to convert that information into actionable information such as pay off your credit card debt or pay off your student loans.

And how about in insurance?

Well, basically, insurance companies have to allocate reserve capital to compensate for future losses. There's a lot of historical work in the actuarial community, but it's quite mathematically basic. Mick Cooney leveraged newer techniques and using programming languages like R and Stan to compute a loss ratio that is the total amount that will be lost by an insurance company to future claims. This is a great example of where a better modeling approach can help you better understand your risk. A more complicated model is worth it in this case. Since the used case involves so much risk and so much capital, this model was interpretable. For example, one could see the naturally incorporated uncertainty in the posterior distribution. Business knowledge was also incorporated. Furthermore, there was increased confidence in the model and the explicit statement of assumptions improved interpretability. Therefore, it is clear that this modeling approach can be a useful addition to our toolbox and also can provide insights that traditional machine learning methods can provide. For users who'd love to learn more about these Bayesian techniques, then I recommend Mix resources Search for lost Kerrs case study on the STAN Case Studies website or my course on probabilistic programming is also excellent. That's called Probabilistic Programming Primer.

Thanks for speaking today, Patter.

Anything I can do to help the listeners.

Time to get straight back into our chat with Wes McKinney.

I'd like to step back a bit and think about open source software development in general. I suppose spoiler alert. Where I want this to go is to talk about your one of your latest ventures, Ursula, but I'm wondering in your mind what the biggest challenges for open source software development are at this point in time.

Well, we could have a whole podcast just about this topic and of course.

It depends on the stage of a project and all of these.

The way that I frame the problem when I talk to people is that I think open source projects face, you know, funding and sustainability problems of different kinds depending on the stage of the project. So I think in the early stages of projects, when you're building something new or you're essentially solving a known problem in a different way, it can be hard to get support from other developers or financial support to sponsor individuals to work on the project because it's hard to build consensus around something new and there might be like even competing approaches to the same problem. And so if we're talking about the kind of funding that can support full time software developers, you know, it can be a lot of money. And so committing a lot of money to support a risky venture into kind of building a new open source project which may or may not come become successful can be a tough pill to swallow for potential financial backer. Later on, as projects become, you know, wider adopted, they start becoming particularly projects that are, that are foundational and you can call them like, I think the popular term is like open source infrastructure. There was A report. So Nadia Egbal wrote the report called Roads and Bridges about kind of open source infrastructure with the Ford foundation and sort of is about this idea of like thinking about open source software as like a public good, roads and bridges and like public infrastructure that everyone uses. And you know, with public infrastructure it's great because it's supported by tax dollars, but we don't exactly have a open source tax. I mean, I could get behind one, but you know, we don't have that same kind of mentality around funding critical open source infrastructure. And I think that as projects become really successful and they become something that people can't live without, they end up facing the classic tragedy of the commons problem where people feel like, well, you know, they derive some, they derive a lot of value from the project, but because everyone used this as a project, they don't want to foot the bill of supporting and maintaining the software project. So whether you're on the early side of a project or the late, you know, in the early stage or late stage, I think there's different kinds of funding and sustainability challenges. And in all cases I think open source developers end up, particularly as projects become more successful, you end up quite overburdened and you know, a burnout risk. And I know I've experienced a burnout many times and many other open source developers have experienced periods of significant burnout.

So what can listeners who are, you know, working or aspiring data scientists or data analysts in organizations or C level people within organizations do for the open source? What would you like to see them do more for the open source?

Well, I think, I think users and other folks can help with. So as people like me, I guess I've recently been kind of, you know, working on putting myself in a situation where I am able to raise money and put to work money that is donated for direct open source development. And so, and I think so the best way a lot of people can help is by selling the idea of supporting and either through development work or through direct funding supporting the open source projects that you rely on. So I think a lot of companies and a lot of developers are passive participants in open source projects. And so finding a way to contribute, whether it's through money or time, it is difficult because many open source projects, particularly ones that are systems related to infrastructure, they don't necessarily lend themselves to casual, quote, unquote, casual contributions. So if it's your 5% project or your 20% project, it can be hard as an individual to make a meaningful contribution to a project which may have a steep learning curve or just require a lot of intense focus. And so I think for a lot of organizations the best way to help projects can be to donate money directly.

So I think something. This provides a nice segue into your work at Ursa Labs and I'd love for you to just give us a rundown of Ursa Labs and in particular how it frames, you know, the challenges of open source software development.

Yeah, so Ursa Labs is an organization. I partnered with Hadley Wickham from the R community and rstudio to found Ursa Labs earlier, earlier this year. The kind of raison d' etre of Aversa Labs was to build shared infrastructure for data science. In particular, building out the Arrow ecosystem, the Apache Aero ecosystem as it relates to data science and making sure that we have high quality, consistent support for all of that new technology in the Python and R world and beyond, and improving interoperability for data scientists that use all of those programming languages. But the particular logistical details of ursalabs is that we wanted to be able to effectively put together an industry consortium type model where we can raise money from corporations and use that money to hire full time open source developers. So at the moment, you know, so Ursa Labs is being being supported by RStudio, by Two Sigma, where I used to work right up until the founding of Ursa Labs and it's now being funded by Nvidia, the makers of graphics cards. And so we're, you know, kind of working actively on bringing in more, you know, sponsors to build a larger team of developers. And I think it's really confronting that challenge that I think for an engineer at a, at a company, as a part time contributor to an open source project, may not be as effective or nearly as effective as a full time developer. And so I want to make sure I'm able to build an organization that is full of outstanding engineers who are working full time on open source software and making sure that we are able to do that in a scalable and sustainable way and is kind of organized for the benefit of the open source data science world. So anyway, having been through the consulting path and the startup path and working for single companies, I think a consortium type model where it's being funded by multiple organizations and where we, you know, we are not building a product of some kind. It's kind of a new model for doing open source development, but one that I'm excited, excited to pursue and see things go.

Yeah, I think it's really exciting as well because it does approach a lot of the different challenges. One in particular, it's a trope, it's a common problem, right, of developers being employed by organizations and being given a certain amount of time to work on open source software development, but that time being eaten away because of different incentives within organization. Essentially, yeah.

I mean, I think there have been a ton of contributions to Pandas and to Apache Arrow from developers that work at corporations and those contributions mean a lot. So definitely still looking for companies to collaborate on the roadmap and to work together to build kind of new computational infrastructure for data science. You know, I think it's tough when, you know, a developer might show up and be spending a lot of time for a month or two and then based on their priorities within where the company, where they work, they might disappear for six months. And that's just the nature of things. You know, I think the kinds of developers that make big contributions to open source can often be more senior or like, can be very important developers in their respective organizations and so frequently gets kind of called in to kind of prioritize closed source or internal projects. That's just kind of, you know, the ebb and flow of corporate environment.

So I've got a relatively general question for you. What does the future of data science tooling look like to you?

Well, speculative of course, but you know, I think by spending my time on Arrow project, you know, my objective and what I would like to see happen in data science tooling is a defragmenting of, of data and code. So to have increased standardization and adoption of open standards like the Arrow columnar format, storage formats like Parquet and ORC protocols for messaging like grpc. So I think that in the future I believe that things will be a lot more standardized and a lot less fragmented. Kind of a slightly crazy idea. I don't know how crazy it is, but I think also in the future that programming languages are going to diminish in importance relative to data itself and common computational libraries. This is kind of a self serving opinion, but I do think that if to be able to leave data in place and to be able to choose the user interface, namely the programming languages, the programming language that best suits your needs in terms of interactivity or software development or so forth, that you know, you can use multiple programming languages to build an application or pick the programming language that you know that you prefer while utilizing common libraries of algorithms, common query engines for processing that data. And so I think we're beginning to see kind of murmurings of this defragmentation happening. And I think the arrow project is kind of kick along this process and socialize the idea of what a more defragmented and more consistent user experience for data scientists, what that might look like.

That's a very exciting future. So my last question for you, Wes, is do you have a final call to action for our listeners out there?

Yeah, I would say my call to action would be to find some meaningful way to contribute to the open source world, whether it's sharing your ideas or sharing your use cases about what parts of the open source stack are working well for you, or what parts you think could serve you better. If you are able to contribute to projects, whether through discussions on mailing lists or GitHub or commenting on the roadmap or so forth, that's all very valuable. I think a lot of people think that code is the only real way to contribute to open source projects. But actually, you know, I spend a lot of my time is not writing code, it's reviewing code and kind of steering discussions about design and roadmap and feature scope. And I think the more voices and the more people involved to help build consensus and kind of help prioritize the work that's happening in open source projects helps, you know, make healthier and more productive communities. And if you do work in an organization that has the ability to donate money to open source projects, you know, I would love to see worldwide corporations effectively tithing a portion of profits to fund open source infrastructure. I think if corporations gave, you know, a fraction of 1% of their profits to open source projects, the funding and sustainability crisis that we have now would essentially go away. And obviously I guess that might be a lot to ask, but I can always hope so. I think corporations can lead by example and certainly if you do donate money to open source projects, you should, you know, make a show of that and make sure that other corporations know that you're a good citizen and you're helping support the work of open source developers.

I couldn't agree more. Wes. It's been an absolute pleasure having you on the show.

Thanks Hugo. It's been fun.

Thanks for joining our conversation with Wes about Pandas, data analysis tooling in general, the future of data science, and the challenges of open source software development. Wes stated that he thinks in the future that programming languages are going to diminish in importance relative to data itself and common computational libraries. And his work on Apache Arrow is central to this vision. The concept of portable data structures that are accessible from a variety of programming languages and that can leverage the vast computational power we now have to work in the limit of at least hundreds of gigabytes. Many popular data science tools such as Pandas in general do not effectively leverage modern hardware. One of Ursa Lab's goals is to empower and accelerate the work of data scientists through more efficient and scalable in memory computing. We also discussed Ursa Labs which I am so excited about, and how the model of open source software development here is to effectively put together an industry consortium type model where they can raise money from corporations and use that money to hire full time open source developers. Also get ready for next week's episode, our 2018 season one finale, A Conversation with Kathy O', Neill, data scientist, investigative journalist, consultant, algorithmic auditor, and the author of the critically acclaimed book Weapons of Math Destruction. Kathy and I will discuss the ingredients that make up Weapons of Math destruction, which are algorithms and models that are important in society, secret and harmful from models that decide whether you keep your job, a credit card or insurance, or algorithms that decide how we're policed, sentenced to prison, or given parole. Kathy and I will be discussing the current lack of fairness in artificial intelligence, how societal biases are perpetuated by algorithms, and how both transparency and auditability of algorithms will be necessary for a fairer future. What does this mean in practice? Join us next week to find out. As Kathy says, fairness is a statistical concept. It's a notion that we need to understand at an aggregate level. And moreover, data science doesn't just predict the future, it causes the future. You'd best tune in for this, our final episode of Season one of Data Framed. I didn't intend for that to sound threatening, but it came across so I'm.

Your host, Hugo Bown Anderson. You can follow me on Twitter at.

Hugo Bown and Data Camp At Data Camp, you can find all our episodes and show notes@datacamp.com community podcast.