#20 Kaggle and the Future of Data Science

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Anthony Goldblum, CEO of Kaggle. If you thought that Kaggle was merely a platform for machine learning competitions, prepare to have your mind blown because these ML comps account for less than a third of the activity on Kaggle. Today, Anthony and I will discuss Kaggle kernels for reproducible data Science and the evolution of the Kaggle public data platform. To do so, we'll make tours through the genesis of Kaggle and how Anthony managed to solve the cold start problem of building a two sided marketplace. We'll discuss the exciting implications of Kaggle's recent acquisition by Google for the future of cloud based data science. And we'll hear why Python is dominating so much on Kaggle. Before jumping into the interview, I want to remind you all to subscribe to Data Framed on itunes, Google Play, Overcast, Stitcher, Spotify, we're everywhere. Do subscribe and we all win. Also write us a review on itunes. It will help us to keep on keeping on. I may even read your review on the show in the future. So I'm going to read one review now before jumping into the interview with Anthony. This review is by somebody calling themself Bring on more frogs. And the review is titled Catnip for Data Nerds. The review reads. I'm getting many more dinner party invites since I started listening. I'd have given six out of five if there was more discussion of Hugo's beard. Well, there you go folks, and thanks for the great review. Bring on more Frogs I hope to discuss more about my beard soon. I'm Hugo Bown Anderson, a data scientist at Data Camp and this is Data Frame. Welcome to Data Framed, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host Hugo Bown Anderson. You can follow me on Twitter hugobown and datacampattacamp you can find all our episodes and show notes@datacamp.com community podcast. Hi Anthony, and welcome to Data Framed.

Thanks very much for having me.

It's a pleasure to have you on the show and I'm really excited to talk about your work at Kaggle and the direction in which Kaggle is heading. But before that, I think I'd like to hear about how you got involved in data science, because a lot of people working in the data science space now approach it from a multitude of different directions and I'd like to know a bit about your journey, how you arrived at where you are.

Yeah, sure. So at university or college, to an American audience, I studied econometrics, which is basically statistics applied to kind economic problems. My first job out of college was at the Australian Treasury. I worked in a macro econometric forecasting group, building forecasting models to forecast gdp, inflation and unemployment the way I got to Kaggle. So I love statistics and I love programming. Both of them were sort of independently interesting. For me, statistics was my profession. Programming was a hobby. In I think it was 2007, I won an internship to the Economist magazine and I pitched a piece to my editor about predictive modeling. I sort of thought that was the future. And he said I could write it. And you know, it was a really nice opportunity because I could call people up, I could say, hi, it's Anthony Goldman from the Economist. I'd like to interview xyz and I got to speak to people who were doing, I guess what we'd now call data science at Tesco and data science for the Obama campaign. And that's where I really, I guess, got exposure to the application of, I guess, statistical techniques to business and, and how I went from being an econometrician or a statistician focused on economic modeling to somebody who was more interested in use cases outside of just plain econometrics.

Yeah, and I think that's quite interesting because econometrics is one of the fields which of course predates data science. But a lot of the techniques we see emerging and becoming commonplace in data science are well known, have been used for some time in econometrics in the type of forecasting you were doing.

Yeah, I would say, I mean, my favorite definition of data science is somebody who knows more programming than a statistician and more statistics than a programmer. And I would say I definitely fit that criteria. So through my econometrics I was schooled in, you know, classical statistical techniques. But as you know, I had a uni job as A programmer. I loved programming and so I really did fit those, those two buckets.

And when working as an econometrician. Is that. Is that how you say it?

Yeah, that's correct. That's a good right pronunciation.

Were you programming as well in. In that position?

The specific group I worked on was the modeling group. And so most people were using Excel. Most people in my division weren't, were using Excel. The group I was in had a model that basically knitted all the pieces of the Australian economy together. And you could not do that. That was too complex to do in Excel. We were using a. I think I forget exactly the. The language is called tsp. It was like a statistical modeling language. I don't know how widely it was used. It was a really crappy, awful programming language. But now that's what the model was written in and so that's what we were using. I think given, given my time again, it would have been much, much more pleasant to be working in Python than tsp, for sure.

So, going from Econometrics and writing for the Economist, how was Kaggle born?

Yeah, actually the inspiration for Kaggle came out of that internship at the Economist. So as I said, I felt like I was interviewing these people from Tesco and the Obama campaign, et cetera, and I felt like I was a good programmer, a good statistician. If you could give me a chance on some of those problems, I knew I'd do a good job. But I also knew that if I wasn't calling up as Anthony Goldman from the Economist, but just, you know, the random person, I wouldn't have had the opportunity. In the meantime, I'd seen every year the KDD conference had run these machine learning competitions where the winner got a got to present at the conference. And I always thought that was a very meritocratic way to decide who, who would get to speak at the KDD Conf. Similarly, Netflix ran a prize, I think it started in 2007. So they originally ran a KDD Cup, I think they ran the 2006 KDDCAP and then following up from that, they launched a $1 million prize. And so the idea of competitions as a way to get data science problems solved had been in the air for quite a while, actually the first KDD Cup, I think, around in 1997. So it had been in the air for a while. And yeah, as I said, I was, you know, trying to solve basically a problem for myself. I felt like I was a good statistician, a good programmer. If I could have the opportunity to work on interesting problems that I was interviewing people about, I would have really enjoyed it. But if I contacted the same people in another setting, wouldn't have had the opportunity and felt like competitions were a really nice way to solve that problem to give me access to companies, data sets and the ability to show what I was capable of. Of course, the irony is there are two, two ironies that come with this. One is I've never really seriously competed in a competition and unfortunately I've subsequently learned that I'm not as good a statistician or programmer as I thought I was. The quality of the stuff that comes out of Kaggle community is just outstanding and beyond. Unfortunately, I think what I was capable.

Of for sure and I think the idea of these machine learning and data science competitions is very timely as well because one thing it's doing is really crowdsourcing data science solutions and data science tech, which, you know, with the advent of Web 2.0 and online capabilities, we see everywhere this idea of using the Internet to crowdsource.

Right. I used to joke that when we were pitching VCs very early on in Kaggle's life that you could have, if you had like a buzzword, a buzzword machine that just spat out random startup ideas, that crowdsourcing data science would have to be one of the buzzwords, that one of the startup ideas that would be generated by this buzzword generating startup machine.

That's hilarious. So then what happened with Kaggle? Tell me a bit about its early days and where it's heading now.

So the very, very early competitions that Kaggle ran were mostly for researchers. So I remember the first competition of any consequence, I think it was actually the second competition we ever ran was taking genetic markers and trying to predict the progression of HIV viral load. So you can live a long life with HIV or you can deteriorate quickly. And this is partially governed by genetic markers. This is a problem that had been studied in the literature for some period of time and a certain number of markers were found you could predict with a certain level of accuracy. And it was a really amazing result. I think at the time, Kaggle had about 100 members. Even within that, 170 teams competed in that competition. And out of that 70 teams, four or five had outperformed the best of the scientific literature, which was pretty remarkable. As I said, this was not an unstudied problem. That competition got featured in Science magazine, I think in 2011. And off the back of the Science magazine article, NASA's Jet Propulsion Lab in Pasadena, California approached Kaggle with a problem around taking images of the galaxy and trying to basically measure the ellipticity of galaxies very precisely. If you could measure the ellipticity of galaxies very precisely, you could figure out what the dark matter distribution was between the observer and the galaxy. And if you could do that, you could basically create a, a dark matter map of the universe. So it's a very important problem in physics. Again, the JPL had worked on it for some period of time. The physics literature had worked on it for some period of time. By then maybe Kaggle had five or six hundred members. Quite a few outperformed the best that NASA's JPL had been able to do. And that NASA was a very, very important customer for Kaggle. They were, you know, NASA are very good brand and they were a bridge between what the academic customers that Cable had in the very early days to the Fortune 500. Our first Fortune 500 customer was Allstate, the insurance company. I think it was easier for our internal advocate at Allstate to be able to say, hey, we should put out some of our claims prediction models here. If it's good enough for NASA, you know, surely we can try it out. And Allstate was a very good result. And that got the ball rolling. Particularly we have a very good footprint in insurance today, but more generally among the Fortune 500 to get them starting to use Gaggle as a tool in their data science, science toolkit.

Yeah, that's a really interesting point because I think looking at Kaggle today, people may not realize that the first competitions weren't actually in an industrial setting. They were in basic scientific research.

And even before that, I created the very first Kaggle competition myself. It was to predict the voting of the Eurovision Song Contest. And the idea in starting any two sided marketplace, which essentially is what the competition business was, is you've got to build one side before the other. You have this cold start problem, right? No one is going to post a problem, no company is going to post a problem when there are no data scientists. No data scientists are going to join a platform where there are no problems. And so by initially starting the first, creating the first problems myself, then getting more experimental academic problems, I was able to build up the data scientists side of the marketplace. And then once I had a sufficient number of data scientists, it became much easier to go out to Fortune 500 and say, hey, this is an interesting place to put your problems.

These are the data science and machine learning competitions at Kaggle. But Kaggle is doing a variety of other initiatives now, right?

Yeah, and that's one of the things I'm really proud of now. A lot, a lot of companies will find success in an area and be able to move beyond. And one of the things I'm proudest of that Kaggle has been able to achieve is, yes, we started with machine learning competitions. It's not that well known, but actually machine learning competitions contributed less than a third of activity on Kaggle today. I'll walk you through sort of the journey and where Kaggle has gone, you know, in expanding beyond competitions. So we noticed. So one thing that's really interesting about our competitions is the degree of discussion and collaboration. We have discussion forums where people share, they share tips, they share ideas, they share approaches, they ask questions on problems they're having as they approach the problem. And one thing we noticed that people were sharing was they were sharing a lot of code. Either they'd post a GitHub repository or they would just upload their Python script into our forums. And I was always sad to see that those were the posts that really never got any engagement. And the problem always was that, you know, you take somebody else's code, they're using Python 2.7, you're using 3.6, they're using these libraries. You need to get the same version of Python, the same libraries, the same data set hooked up in order to get their code running. So it can be as much as four hours work just to get the. Just to execute their code. And so it's always sad that people were coming to Kaggle in order to be able to learn. The most valuable learning content was being able to play with other people's code. And it was happening so rarely. So we built a tool that we call Kaggle Kernels. Initially this was really just focused on making it easy for data scientists to share code with each other as they were competing in Kaggle competitions. And so the idea is you could hit new kernel, you'd get either a hosted Jupyter notebook environment or you would get kind of a more conventional ID like environment. And you can just type in your code and you hit run and the code runs on our, in Docker containers on our servers. So it's all invisible. All you do is you just stick your code in and it runs. You can either make that code public or private. And if you make it public, other people can fork it and start playing with your code. So it takes the process of taking somebody's code and playing with it and learning from it and extending it, from having to replicate their exact environment to Hitting one button, a fork button, and then their code runs immediately in an environment that you can run it. That took off in a way that perhaps not even we expected. We had high hopes for Kaggle kernels. It massively increased the amount of code sharing on Kaggle. So if you go on now, any competition that we launch has hundreds of kernels, the top kernels have hundreds of forks. So it made competition so much richer then. So think about where we were then. We had Kaggle competitions and Kaggle kernels, which launched mid to late 2015. Kaggle kernels were clearly taking off. There was a need for this. So we wanted to give people more that they could do with Kaggle kernels. And so in late 2016, we launched a public data platform where anyone on Kaggle could share data sets. So there doesn't have to be a competition attached. But you have an interest in weather data, you have an interest in NCAA basketball data, you have an interest in, you name it, somebody has uploaded a data set on that topic. And that means for any topic you're interested in or any data set you're interested in, there's a community of people sharing their kernels on top of that data set. And so you can see what other people are doing on a data set that interests you. And on the flip side, they can see what you're doing. Very powerful for reproducible research as well. Like if you're a researcher and you can post your the data you used and a reproducible version of your code through Kaggle kernels on that data set. So we went from, in the combination of Kaggle kernels and the public data platform, we went from being the place where people went to compete in machine learning competitions to actually now a much broader community where people can really do any of their side projects. And actually about a month ago we launched another very exciting feature which is the ability to upload data sets and keep them private and just have a handful of collaborators. And so now you can use Kaggle kernels as if just like Google Docs, so you can have your, you know, your work projects, your personal projects on Kaggle. You invite just the small number of collaborators that you have and you share with those, you share the work just with those collaborators and they can fork, extend, edit, comment, etc. And so that is taking us from the place where people went for their machine learning competitions, the place where people went to that for their side projects now, to the place where people can do quite a large proportion of their data science work, whether it be commercial work or fun projects or whatever.

Great. So I've spent a bunch of time on Kaggle in the machine learning competitions, in the public data sets and using the kernels, and I've had a lot of fun. But hearing about the evolution actually paints this broader picture, which now I have a stronger understanding of, of the approach behind it. And something I find very interesting in there is one thing this has done is taken Kaggle from what is a machine learning platform competition, sure, but a machine learning platform, to a broader data science platform and data science community as well. And one example of that I think, is that you can go in, put in a data set and see and people start collaborating. Maybe not on machine learning straight away, but on exploratory data analysis, for example, or data reporting or these types of things.

Yeah, totally. I look at machine learning as the tip of the data science beer, but really for any new data set, even if the ultimate intent is to do machine learning, the first thing you need to do is look at the data. It's always cool. Whenever there's a popular new data set launched on Kaggle, the first kernel that starts to get meaningful traction is almost always an exploratory kernel. It's very nice. Once that's been written, you can look and you can understand. You don't have to redo that exploratory work. You can look at the data, you can understand the character and the nature of that data without having to do the exploration yourself. So you can immediately jump to step two, which is the actual application you care about on that.

Absolutely. We'll jump right back into our interview with Anthony Goldblum after a short segment. It's now time for a segment called Tales from the Open Source. I'm here with Eric Ma, who works as a data scientist and research investigator at Novartis and has contributed his knowledge on network analysis to our Python datacamp curriculum. He has also contributed to and authored a variety of open source data science packages, including Matplotlib, PyMC3, Bokeh, NXViz, and his latest fun project, Pyjanitor, a Python port of the R package. Janitor, how are you today?

Eric hey Hugo, it's great to be here on the podcast today.

It's awesome to have you here. What are you going to be talking about today?

So today I'd like to talk a little bit about how you can get involved in open source contributions as a beginner. It's a topic that's close and dear to my heart because I would never have finished grad school without open source.

Software Open source software is definitely an essential part of data science. Before we go on to how someone can contribute to open source, though, perhaps you can tell us why you think it's important to contribute.

If you're a data scientist and you're using the open source data science stack, say Pandas or dplyr, then you're benefiting from others work given freely to you. I'd say it makes perfect sense to give back, even if in ways that are seemingly tiny and small.

That's great. Yeah. Often much of the work that goes into the stack is basically paid by someone else.

Exactly.

And it seems to me that there are some myths and barriers that are commonplace regarding open source contributions. Perhaps you could talk a bit about them.

One big myth is that open source projects will only accept code that implements new features, or that you have to have really complex ninja code skills. This most certainly isn't true. Sometimes enhancements to the examples and the docs are sorely needed as well. For example, my first open source contribution was to the matplotlib examples gallery. This was back in 2015 and I was using matplotlib heavily in my research, so I thought I'd make a contribution to get familiar with the API. Now the lead developers wanted the examples gallery to be cleaned up so that it was consistently using matplotlib's Pyplot API rather than the old pylab API, because the newer Pyplot API was going to be favored. Thus, my contribution was literally taking a bunch of import statements and replacing them with import matplotlib, pyplot as plt and import numpy as NP and then verifying that the examples worked. Thus, contributions really need not be complex at all, as long as they're addressing a legitimate need for the package. In fact, code isn't the only thing you can contribute. Open source projects often need documentation help too, and that can mean improving the clarity and wording or adding documentation where previously it was missing, or even contributing usage examples.

Cool stuff, Eric. But surely the tooling needed to make an open source contribution can be a bit complex though.

No, you're definitely right about that, Hugo. Much of open source software development uses Git as the version control system. Having some basic mastery of Git and its idioms can be very daunting at first. This is where educational materials such as software and data carpentry can help one get up to speed. In addition, education platforms like datacamp can be super helpful too. I love how you guys have a course where you teach Git for data scientists?

Exactly. The awesome curriculum is one of the true joys of working here at datacamp.

And I think an even more important thing for newcomers is also the human dimension that is to be working in a beginner friendly environment. Now that means both training and people. After all, open source software has real people behind the code. So by training, here's what I mean. Back in 2015, when I participated in the scipy sprints, there was explicitly a beginner friendly git training. We had a git tutorial organized by the sprint organizers prior to the start of the sprints, in which they showed us the basic workflow of fork, edit, pull, request and merge. In addition to that, in the Human Dimension, Thomas Caswell and Ben Root, two lead developers of matplotlib, really helped guide me along the way. I'd make mistakes and get confused with git, and during the sprint they would use the mistakes that I made as teaching moments to show me more advanced concepts that I was missing. Now it's not just, you know, tutorials and people. Other projects have different ways of also being beginner friendly. So for example, SciKit learns GitHub issue tracker has issues tagged as well beginner friendly, which you can use as a guide to get started.

Those are definitely important points. Eric, any closing words for our podcast listeners?

I think what I'd like to offer are these. Number one, don't be afraid to start with simple things and two, find people to help you along the way. Because of these, I was able to start making contributions very soon after learning Python and I've had a ton of fun doing so ever since.

Eric, thanks for taking the time to share your experience working on open source.

It's always fun chatting with you.

Hugo.

After that interlude, it's time to jump back into our chat with Anthony. So this is all very new in the CAG award and very exciting. And something that happened relatively recently as well is that you were acquired by Google. So firstly, congratulations.

Yeah, thanks very much. I think it's a very nice home for us.

And that really leads into my next question. What was your trajectory that meant that it made sense for you to be acquired by Google?

Yeah. So I guess there are two questions here. Why does it make sense for Kaggle? Why does it make sense for Google from the Kaggle perspective? One thing that was always going to be a struggle for us is Kaggle kernels. We wanted to have a meaningful free tier. So our philosophy has always been that we wanted the free tier of Kaggle kernels to be about equivalent to what you get doing machine learning on your own personal laptop. Problem is we have a community of 1.5 million people that gets very expensive for a startup. And also just having the compute resources behind you to make something like that available is a non trivial thing. The part of Google that Kaggle is part of is Google Cloud. And so there is no shortage of cloud computing capacity now. And so as a result, we've been able to increase the amount of resources that we've made available to our community. More importantly is some of the things coming down the pipeline. So we're going to make GPUs available through Kaggle kernels and eventually TPUs as well. And so if you want to train your machine learning model using the latest Google hardware, this is something that the tensor processing units, this is something that will be available relatively. The fastest way to get access will be most likely through through Kaggle. And the reason Google in particular makes sense is that no company takes machine learning as seriously as Google. And so by being inside Google and part of Google, we're able to make Google machine learning technology available to our community. On the flip side, why does it make sense for Google? Google, no secret, Amazon Web Services started earliest and have the most traction when it comes to building the dominant or the world's currently leading cloud. One of the areas where Google is going to ultimately end up being stronger. Whereas Google will be the best cloud for data analytics and machine learning. And so having the world's largest data science and machine learning community operating on Google Cloud was a strategic priority for Google. So it's one of those things that made sense in both directions.

Absolutely. And so for Google, having that community, I suppose in terms of the information it provides, also as a recruiting funnel, are these the types of things that play into it?

Yeah, I would say there's a couple of things. One is our agreement as we spoke about the acquisition and what would make it work is that we would make Google Cloud technologies available to our community, but not wouldn't force them on our community. And so to the extent that Google Cloud technologies really are the best machine learning and AI technologies, our community will get early access to them and that will help Google Cloud build a critical mass around those products. Again, assuming they are the really strong products and stronger than what's elsewhere. So that's the first thing. A lot of us went to university and we all learned Matlab. Why do we all learn Matlab? Because it was a critical mass around Matlab. A lot of companies use Matlab, which meant a Lot of universities taught it, which meant a lot of companies use that with the cloud AI and machine learning tools. There is no such critical mass. And so Kaggle is one engine for building that critical mass. The other is that as we start to integrate with Google Cloud products, Google Cloud customers will have the, they can enter through cloud.google.com and they get a blinking cursor, or they can enter and get access to Google cloud tools through Kaggle.com and if they enter through Kaggle.com, they get all these data sets they can take advantage of, they get all these kernels that they can fork and extend. And so again, if you go to aws.Amazon.com, you get a blinking cursor, whereas if you enter through Kaggle, you get this sort of very rich environment. And so it gives people a reason to be on Google Cloud rather than Amazon. Does that make sense?

Yeah, it makes perfect sense. And so I'm speaking very briefly to your Matlab point. I'm someone who has used matlab a great deal in my previous incarnation in academic research and it served a great purpose for me. But I, I am also critical of a lot of aspects of matlab. And one thing is, it's also very difficult. One of the reasons it stuck for a long time is it's really difficult for institutions to get off proprietary licenses. Right?

Yeah. I would say when Kaggle first started MATLAB, SaaS, SaaS Institute, or FAR more dominant on Kaggle, I think people have a preference. One of the benefits you have with these open source tools like Python and R is you have these really rich package ecosystems. And I think Python is on Kaggle at least is dominating at the moment. The reason being is that Python, you know, anytime there's a, the machine learning frameworks, the deep learning frameworks are much, much richer on, on Python than they are on even R and you know, the proprietary software for sure. And the reason being is that, you know, if someone publishes a paper with a new technique, it's not, doesn't take very long before there's an open source implementation. It takes quite a lot longer for it to get into those developments, to get into the proprietary software packages.

Yeah, exactly. And I think speaking to the growth of Python, and I'm sure you saw recently Dave Robinson who was at Stack Overflow, now he's chief data scientist at Data Camp, he wrote a series of articles about the growth of Python and put a lot of it down to the growth of Pandas as well, actually.

Yeah, for sure. I mean that's the one thing that R had that Python didn't have was a nice data frame.

Absolutely.

Data frame. So, yeah, I think Wes, Wes McKinney deserves a lot of credit.

Very much so, Wes and the whole, the whole team. I also, I was recently at RStudio conf and JJ Allaire of RStudio gave an incredible keynote on how they've now developed an interface to Keras for R. So he's really pushing the idea of RStudios, sorry, of R as an interface programming language, among other things. So I think we're seeing, going to see a lot of developments on that front as well. So it's pretty exciting.

Yeah. And they do, they do amazing work. RStudio is a phenomenal data science ide.

And so something you spoke to is that we have a lot of different industries and people from a lot of different industries in the Kaggle community. And I'm wondering which industries or areas of research you see platforms like Kaggle having the largest impact.

So there's a couple of ways I could slice and dice that question. So where will machine learning have the most impact in the coming decades? You know, it's. I like the saying that it's kind of hackneyed in the tech industry, but the future is already here. It's just not widely distributed. Some of the areas where deep learning in particular is proving to be very strong, medical imaging. So Kaggle has done competitions on topics like taking MRIs and trying to. Trying to diagnose heart failure, taking images the eye and diagnosing diabetes retinopathy, taking ultrasounds and diagnosis the nerve endings that are responsible for chronic pain. The results are always strong. And it's interesting because medical imaging has been something machine learning and medical imaging has been something people have wanted to do for. It's been a topic for many years. But before deep learning there were very, very few success stories. I think the main one was breast cancer imaging. You could do decently well in diagnosing breast cancer from machine learning algorithms, but as soon as you had somebody who was overweight or had calcites or, you know, some other anomaly, the algorithms would melt down. So deep, deep learning has been a real inflection point and I think it's going to make a big difference. Machine learning, obviously, to the extent that it plays a role in autonomous vehicles, that's another game changer. So these are not. Yeah, I'm a big believer that, as I said that, that we see the beginning. We're seeing a lot of the beginnings of the areas where machine learning is going to be most disruptive.

And which fields do you think there may be too much hype around? Or what fields do you see data science and machine learning not being able to have such an impact on?

I guess I think there's too much hype around the concept of artificial general intelligence. Maybe I'm not looking far enough into the future or I'm not being imaginative enough, but I feel like we periodically have step changes, right? So we used to use logistic regression and linear regression and they were the main workhorses. And then we had the random forest breakthrough and that was a big step change in what we could do with machine learning models to tackle high dimensional data sets and so forth. And we had more incremental innovations on random forest things like gradient boosting machines and particularly the XGBOOST implementation. Then around 2012 we had another step change with deep neural networks that were able to really dramatically improve what we could do with unstructured data, particularly images. And I think that step change has been misinterpreted as the beginning of discovering things like artificial general intelligence as opposed to a step change. And now we're on another plateau and hopefully eventually, you know, hopefully in the next 10 years or so we have another step change which will then lead to another plateau and then another step change and another plateau. So I think where we're making improvements on what we can do with machine learning, but I really don't think we have any line of sight into this idea of artificial general intelligence. And so we're really limited to the areas where machine learning. Machine learning is very good on high volume tasks. So things we do very repetitively. So you could look at very standard legal contracts as things that machine learning algorithm would be able to potentially do a good job on. Accountants doing very standard audits, majority of an audit can probably be done or will be able to be done by a machine learning algorithm. But as soon as some path breaking litigation or you're looking at a very creative tax scheme, this is where machine learning is not going to be very useful. So anything that's at all novel machine learning tends to fall down. I think we'll see machine learning automate repetitive tasks in a lot of different industries, but not totally upend industries.

I think as you say high volume tasks and tasks where we have a lot of data deep learning can be really good at. But one shot learning is still a very active area of research.

Yeah, and I'd actually say deep learning is very powerful and it's able to do a lot of really great things, but it's not the be all and end all. I mean, we still find in Kaggle competitions if you have structured data that is small or medium sized, you're still much better off. With feature engineering and XGBOOST and even some natural language processing problems. We find deep neural networks or recurrent neural networks do quite well on some natural some text and speech problems. We also find that there are situations where old information retrieval approaches combined with XGBOOST are still dominating.

So I'm glad you've mentioned XGBOOST a couple of times because I remember maybe it was probably two and a half years ago when I was mucking around with XG Boost and trying to figure out the ins and outs of it and the algorithms were very good. It wasn't necessarily so well documented at that point. And by far the best documentation I found on XG Boost was actually in Kaggle forums.

Yeah, and I'd say XGBoost, actually one of the areas where Kaggle has made a really positive contribution to the world is I think we've made it very clear, you know, when Kaggle first started, people used to use all sorts of wacky things, genetic algorithms, self organizing maps. I've forgotten half of it. Support vector machines would be lots and lots and lots of different techniques people used to try. Became very clear very quickly that random forest was the dominant tool in the vast majority of cases. Then XGBOOST came along and did random forest. And in 2012, this is a nice story. George Dahl and Jeff Hinton from the University of Toronto actually used deep neural networks to win a competition for Merck. Then the next competition immediately after that. Vladmin, who's now DeepMind, used deep neural networks to win that competition. And so Kaggle has been a really good place. Like if you look at the Kaggle on our blog, we have what we call winners posts which basically summarize the winners. We do an interview with the winners and the winners talk about the techniques that they used in that particular competition. I always say it's like a very good pragmatic journal on what machine learning techniques are working well and what problems. If you follow our blog, you kind of see the rise. We've had a big role to play in helping make it very clear what techniques work best on what sort of problems and helping new techniques and new frameworks like XGBOOST get attention and get the adoption that they deserve. I would argue that particularly XGBOOST would not be, or even Random Forest would be one of a whole lot of techniques people tried if not for Kaggle helping to sort out what really works for from what doesn't.

Yeah, and we'll, we'll link to the Kaggle blog in, in the show notes and and the winner's post because I think they'd be very instructive for a lot of our listeners to to check those out.

Totally.

We'll jump right back into our interview with Anthony after a short segment. Let's now jump into a segment called Language Corner with Data Camp curriculum lead Spencer Boucher. What's on tap for this segment? Spencer?

I'll answer a question with another question there. Hugo, quick, what's the most underrated language for data Science?

Scala? Julia? JavaScript? Python? No, that last one was a joke.

I'm going to spend the next couple of minutes trying to convince you that it's actually the language of the Unix command line.

Great. I love it. And for people who haven't heard Unix per se, you may have heard of it as shell bash terminal, these types of.

Exactly. So if you're a data scientist that's using a Unix system, so that's going to be like Linux or OS X, you've probably touched the command line at some point. You might still not be aware of just how powerful it can be. However, Unix is a collection of small standalone utilities that adhere to a specific philosophy of doing one thing and doing it really well. Hugo, what are some of your favorite Unix tools?

Firstly, Spencer, I love the command line. One of my favorite Unix tools is Grep, which allows me to search a bunch of text files that I'm that I may have in a directory. I also love Head, which allows me to print the first three or five or whatever. I choose lines of a file of data that I want to check out. I also love sorting stuff. So this is like order by in database queries. What I love is to be able to sort my lines in a file. And then finally, perhaps my favorite Unix tool is the pipe, which allows me to do a series of operations which output and input text and pipe them together in order to do something a bit more complex.

Yes, exactly. I'll talk more about the pipe later. A few more commands that I really like. There's find, which is kind of like your file browsers, you know, search GUI on steroids. There's sed, which edits streams of characters which can really come in handy. There's also something called unique. There was one tool that I really liked that filters for unique rows in a particular text file and then something else that really comes in handy is the cat function. So the cat command concatenates files together, just like a union operation would in something like SQL.

Those are all great. And I also actually want to mention I love wc, which is word count. So I can count the number of words, characters, lines in any given file or directory.

Yes, exactly.

So what makes these tools so great, Spencer? Why would someone ever want to use this collection of tools rather than a programming language like R or Python?

Well, for one thing, this set of tools generally has a very rich API that's optimized for the 80% use case. So that means instead of having to code anything yourself, it's more than likely that a simple command flag will handle whatever task you have at hand just right. Once you've had the chance to play around with them, these tools really become second nature, and you'll find yourself not even needing to break out a full programming language to do little one off tasks. And on top of that, they're always going to be available to you in pretty much any scientific computing environment, even if your favorite programming language might not be.

And Unix command tools, as I said, are designed to work together, right? They all take a stream of text as input, perform some operation, and return a new stream of text as output.

Yes, and that's the real killer feature here. That kind of composability just makes it a snap to chain a series of operations together. For example, it's a natural operation to say, find all the CSVs in a directory matching a particular regex, concatenate them, remove duplicate rows, order by one particular field, and then write the top five rows of that result out to a new CSV all in one chain. If you're familiar with Magrit r from R, or you're used to chaining method calls in Pandas, then this is going to be really familiar to you.

Okay, Spencer, I'm sold. But what if our listeners aren't quite convinced yet?

All right, well, if they need a few more reasons, here's two more important side benefits to the streaming nature of UNIX pipelines. First of all, you don't have to load a whole data set into memory to operate on it. Well, so think about the last time that you tried to load some giant data set into a data frame in R or Python and everything got sluggish. A properly set up UNIX pipeline won't suffer that same problem. Second, because UNIX tools operate on streams of text, it's incredibly easy to take a UNIX pipeline and parallelize it, giving you a linear speed boost up to the number of CPU cores on your computer, and it's not uncommon for that to be a factor of 16 or 32 on modern laptops. So before you fire up that Hadoop cluster on aws, do a quick check to make sure that a Unix command won't do the trick for you. Instead, you might be surprised.

Thanks, Spencer. And for our listeners out there, we're going to include the following resources in the show Notes. But if you want to check out some educational content on the command line, check out our Introduction to Shell course on Data Camp. Check out the Software Carpentry course along with the O'Reilly Unix power tools. Fantastic. Thanks, Spencer, once again.

Yep, anytime, Hugo.

Time to get straight back into our chat with Anthony Goldblum. So something we're really circling around is one of the aspects that I genuinely love about Kaggle the most, and that's this concept of community and all the people you meet there, all the ideas you get thrown around. You see ideas rise and concepts and techniques and different methodologies rise to the top when you have this critical mass of people working on these challenges. So my question for you is, I suppose it's a double pronged question. What is the role of community in data science in your mind? And how do you think about promoting community at Kaggle?

I would say it's not something I have a super well informed answer on. I think that, you know, Kaggle happened to stumble into becoming a community. We built something, people liked it. And our approach to expanding the range of things that we offer to our community in order to strengthen it is really to see how people are, what people are doing on Kaggle, and trying to support things that people want to be doing. I explained the origins of Kaggle kernels. It was being born out of watching people discuss different techniques on the forums and wanting to give people a more efficient users, a more efficient way to share code. I mentioned that the public data platform was born out of the fact that we can only launch a certain number of competitions each month and there was appetite for people to share more data sets. And so we thought, well, why don't we take ourselves out of the loop and just allow the community to themselves share. And so I'd say that as far as our approach to community management growing, the community is concerned. It's really watching what the community does, how they use Kaggle, and reacting to what we see the community doing in order to continue to hopefully build more and more useful things. I'm not sure if that directly answers your question.

No, it does, because it really suggests a very organic approach in which you're in constant dialogue with the community. And I think the development of, of kernels and public data platform, as you said, speaks to this. It's an interesting paradigm of this because they actually grew, as you say, out of the needs of the community that you were servicing while listening to them. So it does answer the question. In that sense, as you're the CEO of Kaggle, I'm going to ask you to predict something for me. So my question is, what does the future of data science look like to you and interpret that how you will in 2, 5, 10 years? Where is this, this discipline heading?

So, firstly, I've got a caveat, this prediction by saying, I said at the beginning of this episode that I won this internship to the Economist magazine. I didn't tell you how I won it. So I went up with an article that was, it was written in 2007. How the competition works is you have to submit an essay that looks like an essay that someone might, that the Economist might publish or an article that they might publish. My article was about why the uptick in subprime mortgage defaults wasn't a problem. I don't know what all the fuss is about, of course, in 2008, that went on to cause the global financial crisis. My, my, my, my track record on predictions is fairly weak. Yes, that's brilliant.

So we could actually take whatever you say and go the opposite direction.

Go the opposite, yeah. Which, which by the way, is, is just as useful if somebody is a phenomenal, reliably an excellent predictor or a horrible predictor. Each of them are both equally reliable. Right. As long as you know which one they are. So, you know, my view is, as I said, the future is already here. It's just not widely distributed. And companies like Google, I think, are a bit of a sign of what's to come. My view is that in future, data science will play a much bigger role in decision making. I think that very often decisions are made with the absence of data, not because data doesn't exist, but because going from data sitting in a data warehouse or out on some public place on the web and into a usable form that can help inform a decision in an intelligent way is currently a heck of a lot of work. And I think as the tools get better and as we train more data scientists, the friction in going from data to decision is going to get dramatically less. And as a result, the number of decisions made off the bag of data end up being, you know, much higher proportion of Decisions end up getting made off the back of data. So now what does this mean? It means that life runs more efficiently. Right? You look at some of the industries that Kaggle has worked in, fewer oil and gas wells get drilled that shouldn't be drilled because they're not going to produce. You know, fewer people are diagnosed, get going through an airport security line, are pulled aside because the algorithm that TSA is using detects a threat object that isn't there. So it means that life runs smoother, means that we waste fewer resources, whether it be in the TSA case, we waste less people's time, whether it be in the oil and gas case, less environmental, you know, less drilling. So less money wasted, less environmental impact. I think data science is going to have a huge role to play in making the world run better.

As a part of that, that future of data science, what does the future of Kaggle look like to you and how far ahead do you think about this?

Yeah, I'm really excited. I think that Kaggle has, I've said this a couple of ways. I think Kaggle has made a really good contribution to the world so far. And I think we barely scratched the surface of what we're capable of, in my view, that, you know what, Kaggle wants to be a big part, making it easier to go from raw data to decision. We, you know, we started doing that Kaggle kernels allows you to, you know, you want to do sentiment analysis on your tweets. Well, you go into our search box and type Sentiment sentiment analysis Twitter in colon kernels and you can find probably about 200 other kernels that have used sentiment, done sentiment analysis on tweets. You don't have to start with a blinking cursor. You can start with somebody else's code. You want to, you want to join weather Data to store sales data rather than crawling around the web for all the different weather data sets that might be useful through the Kaggle public data platform, particularly as it continues to get more traction, currently has over 13,000 data sets getting many, many more. Over the coming year, you'll have a single place that you can go to find all the data sets that might be useful to you to join to. So the future of Kaggle is really supporting the future of data science. The future of data science. The future of data science is making it easier for decisions to make off the back of data. And the future of Kaggle is to make data science easy to do for data scientists.

Absolutely. And I think this definitely paints a very bright, bright future for data Science and Kaggle, which I'm totally aligned with and I'm pretty much in agreement with. I wonder what type of challenges to your mind, data science will face as an industry or certain caveats we need to be aware of when moving in this direction.

Yeah, I think that Nate Silver says it really well every time he forecasts election results. He said, if I'm right, I'm going to get more credit than I deserve. And if I'm wrong, I'm going to get more blame than I deserve. Data science is going to be wrong. Just statistically, in many, many cases it's going to be wrong. And so you will find that this will, every, every time data science is wrong, people will blame it more than it deserves to be blamed. And so one challenge is that it's not totally prophetic, it's not going to be right all the time. And having decision makers understand that with data science, they'll be right more often than they would be wrong if they weren't using data science is quite a nuanced argument to make. And it's one that data scientists, as we try to become more and more influential in the companies that we work for, it's a really tricky and nuanced message for us to communicate to those who are relying on us to inform their decisions.

For sure. And I think part of the real challenge is communicating uncertainty. Right, because people want yes OR no or 60% or 70% as opposed to 60 plus or minus 30, right?

Yeah, totally uncertainty. It's a concept that I think all of us, a lot of behavioral studies on this. Human beings are not really wired necessarily to understand the nuances of uncertainty.

So Anthony, to close out, I'm wondering if you have any final call to action for our listeners out there.

Yeah, for those of you who aren't currently on Kaggle, come check us out if you're learning. We have a new area called Kaggle Learn which has some notebooks that you can fork that will help you get started. There are also the Data Camp tutorials that take you through how to do well in a Kaggle competition. If you're or you've been on Kaggle and you haven't been around in a while, come and check out Kaggle kernels and the public data platform. And if you want to be playing with some of the cutting edge and benchmarking yourself, you should get into our competitions. We have lots of competitions on just about any kind of type of data set you might be interested in. We probably have a competition for you that will interest you.

Fantastic. Anthony. Thank you so much for coming on the show.

Thanks for having me.

Thanks for joining Our conversation with Anthony about Kaggle and the future of data science we saw how Kaggle's recent products, kernels for reproducible, shareable data science workflows and the public data platform arose from serious practical needs of the Kaggle community and how now, combined with the machine learning competitions, they provide a fully fledged data science ecosystem. We saw how Kaggle's acquisition by Google bodes well for the future of cloud based data science, particularly with the upcoming GPUs and TPUs available to Kagglers. We also saw a wide variety of use cases for machine learning and discussed the particular use cases that Anthony sees being immediately powerful, which include medical imaging and the development of autonomous vehicles. Make sure to check out our next episode a conversation with Sandy Griffith, Principal Methodologist on the Quantitative Sciences team at Flatiron Health, a healthcare technology and services company focused on accelerating cancer research and improving patient care. Their mission is to improve lives by learning from the experience of every cancer patient. Join Sandy and me discussing how Flatiron is leveraging data science in the fight against cancer. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and DatacampataCamp. You can find all our episodes and show notes at@datacamp.com community podcast.