#19 Automated Machine Learning

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Randy Olson, lead data scientist at Life Epigenetics. Randy specializes in artificial intelligence, machine learning and data visualization, is an open science advocate and created Teapot, a data science assistant, and a Python automated machine learning tool that optimizes machine learning pipelines using genetic programming. We'll be talking about automated machine learning, the verticals it could impact, and how it will change a large portion of what data scientists do every day, along with how they approach the questions they ask and communication with relevant stakeholders. If you're a data scientist wondering how automated machine learning will impact your career, stick around. If you work in a vertical that uses data science to get from data to insight, there'll also be a great deal in here to interest you. Don't be shy. I'm Hugo Baun Anderson, a Data Scientist at DataCamp, and this is Data Framed. Welcome to Data Framed, a weekly data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host Hugo Bown Anderson. You can follow me on Twitter, Hugo Bownt and DatacampAttacamp. You can find all our episodes and show notes@datacamp.com community podcast. Hi Randy and welcome to Data Framed.

Hey, thanks for having me.

It's such a pleasure to have you on the show. I'm super excited to be talking about automated machine learning with you today and the type of impact we see on the ground and will in the future. But before we get to that, I like to know a bit about you. What are you known for in the data science community?

Yeah, I guess I'm known for a few things in the data science community, I guess. One having a huge Twitter account that follows way too many other accounts. But I think early on I was known for data visualization and during grad school I ran a data visualization blog and some posts from that became way more Popular than I ever expected, like the optimized road trips and everything. And I realized, like, oh, no, I'm actually not becoming known for, like the data science as the data science guy or the machine learning guy or the AI guy. I was becoming like the optimized road trip guy. And that was not quite where I was wanting to go with my career. So I started focusing my blog and other efforts more on, like, you know, pointing out all the open source tool development and the tutorials that I do, and most recently focusing on pushing this field called automated machine learning, which I'm really excited to talk with you about today.

As am I. And something about your blog that I always found really inspiring is you wouldn't only dedicate yourself to particular topics. You'd actually show how data science techniques, methodologies, and the types of ways we ask questions can be applied to nearly anything.

Right, right. Yeah, that was part of my goal, was. I mean, for me, my blog was actually a learning experience. So I was sort of learning these techniques and applying them as I went. So jumping from various topics as something interesting came up that was important to me, but also just trying different approaches, whether it was data visualization approaches, analysis approaches, machine learning approaches, that was also very important. And hopefully it turned out to be a valuable resource for the data science community as well.

Yeah, it definitely has been. And we'll include a link to it in the show notes also. So what do you spend most of your professional time doing these days?

Well, so up until About January of 2018, I was at the University of Pennsylvania. And so I was in a very fortunate position there and a very good. Where I got to spend most of my time developing and evaluating new machine learning methods and applying them to biomedical problems that we had there at the University of Pennsylvania. But actually, in January, I made the jump out of academia and into industry and joined a startup company called Life Epigenetics. And so now most of the time, I spend my time working with epigenetics data and cleaning it, analyzing it, visualizing it, and most importantly, modeling it. So to try to predict, you know, various outcomes that are interesting and useful for the life insurance industry.

Well, firstly, congratulations on the new position.

Thanks. It's very exciting. Move.

Absolutely. And secondly, could you just give us the elevator pitch on what Epigenetics is?

Yeah. Okay. So the chief science officer hopefully doesn't listen to this and get mad at me because I'll do my best. I'll do my best. I remember I'm a computer science guy. Yeah. So we have this basic idea of genetics, right? A genetic code that we all have that is translated in some way to express and basically become what we are. Right. Epigenetics is a layer on top of that. You can think of it sort of like an interpreter of your genetics. And so there are various things that can affect your genetics and how they interpret your DNA. And so we're looking at how certain lifestyle habits like smoking and drinking affect your epigenetics and whether we can basically look at your epigenetic profile and see, are you a smoker, are you a drinker, so on and so forth. That's the elevator pitch, I'd say.

Fantastic. So we're here today to talk about automated machine learning. But first I'd like to explore a bit about what machine learning is and the places where you find it really, really interesting. Can you give me a brief rundown of what machine learning is to you?

Sure, yeah, very brief, Very high level at least, because otherwise we can spend the whole hour just talking about that. Yeah, so I think if you look in most machine learning books or the Wikipedia or anywhere else, most places will say that at a high level, machine learning aims to give computers the ability to learn without being explicitly programmed. So that's a quote that we'll see around a lot nowadays. I think most machine learning focuses on what's called supervised learning, where basically this idea is that we can provide data to the computer that we sometimes call features that are sampled from some system. The most common example is like the iris flower classification example, where you take measurements of the flowers and then you want to provide corresponding labels for that data. So maybe you'll take some measurements from a flower and say, okay, this is this kind of species. Once you provide enough of those samples, maybe dozens, maybe hundreds, maybe thousands of those, you feed that information to a machine learning algorithm and it's basically going to try to learn some mapping between those features and those labels. And hopefully a good machine learning algorithm will learn a good mapping that will also work well on unseen samples. So if you go and measure new flowers out in the wild and you give those features to the machine learning algorithm, it can accurately classify them in that way.

Great. So in that sense it generalizes to other data in the world.

Yeah, exactly right. It's trying to learn general patterns about that system that you're measuring the features from.

That was a really cool explanation of machine learning. And I think the iris dataset is great for pedagogical values, although I feel like I've seen it too many times in my lifetime. But I know there are a lot of applications that you're really interested in of machine learning powers, so maybe you could tell us a bit about those.

Yeah, sure. Well, so, one, I'll say, yeah, the IRIS dataset is definitely way overused. I've been trying to branch out myself, and I actually encourage anyone in machine learning who's making tutorials to do the same. But, yeah, in terms of interesting applications that I've seen now, I would say two that are really interesting to me are. Well, one is personalized health care. I think it's really interesting that, at least in the hospital systems, we're working on collecting information about people's health histories and their treatments and so on and so forth and getting that into a useful format. And so now we can start answering questions, or at least asking questions like, based on my personal health history, what am I at risk for? Or more importantly, what interventions can I make to prevent some disease or some adverse outcome that may happen to me in the future? And of course, it's important to point out that we have a long way to go. But there have also, if you watch the tech news, at least there have been a lot of small wins along the way. Another really, really interesting application to me is taking that idea and bringing it into insurance pricing. So now can we make predictions about people's risk factors, how likely they are to develop cancer or some other disease, and factor that into insurance pricing? I never thought that I would be going into an insurance pricing industry, but it turned out to be very much the same as my last job working in these biomedical applications. So that was a very surprising discovery for me.

And are these the type of questions you think about and work on at Life Epigenetics?

Well, so right now, we're mostly focused on modeling the epigenetics data. But as we expand as a team and as a company, we're definitely going to be expanding our horizons in terms of, you know, looking at full medical records and how are those predictive of people's longevity and various other things that are of interest to the life insurance industry. So these are definitely things we're keeping in mind, even if we're not necessarily focusing.

But you do use machine learning in your work at Life Epigenetics?

Oh, absolutely. I mean, just before we hopped on to do this interview, I was playing around with Keras. Well, I shouldn't say playing around. I was doing serious work with Keras and building some deep neural network models of the data that we're working with now.

Fantastic. Are There any other interesting applications that you like at the moment?

Yeah, so aside from what I would say, the obvious ones to me, which are the ones that have actually drawn me, to draw me in my career, another really, really interesting one that I've seen coming out of the deep learning research is being able to impersonate celebrities in video. And this is, I remember when I saw the first one they were doing, they did some videos with, with Donald Trump. So basically taking a video of him during an interview and then they're having him say all kinds of silly and weird things. And it was, it was really funny. You know, especially you're in the right group of friends. But if you think about it, there's a lot of potentially malicious applications of that kind of technology. Right. You can imagine if the people who had made that video of Donald Trump had more malicious political intentions and they wanted to force him to say something that looked bad on the US or anything else, which, you know, Donald Trump doesn't need much help doing that. You can imagine that. You know, they can have him say certain things that may alienate even more people or anything else, or any other politicians.

Right.

You can do, you can imagine they can do the same thing for, for any other public figures trying to discredit them by saying something that goes against who they really are. Right. And I think one of the most disturbing applications that I've seen, and it seems to be getting shut down, but not completely because it's. The Internet is like non consensual pornography. So it's now possible to, you know, make it look like some actor or actress is involved in a pornography film when they had no idea about it. Right. They did not give their consent. They did not want to be involved in that whatsoever. But because we have this technology now, it's entirely possible. So there's all kinds of malicious applications of this technology. It's really interesting, but it's also something that I think we really need to sit down and think about in terms of should we really, what should we do about this new machine learning technology that we've developed.

Yeah, I think it's very concerning and a conversation we need to keep having. I'm wondering if machine learning could be used to detect these types of malicious applications as well.

I would say yes. I hope so. Probably. I would say anything is possible when you have good enough data and good enough labels.

Yeah. Okay, great. So we've seen what machine learning is. We've seen a bunch of interesting applications. Let's talk about machine learning, automation, what.

Does that mean, yeah, sure. So basically, if we go back to that quote that I gave when I was describing machine learning, where the idea of machine learning is to say that we want to give the computer, the computers, the ability to learn without being explicitly programmed. If anyone ever said that to me, I would call them out and say that they're just telling me BS right now. Because I have used machine learning a lot in my career at this point and I know that there is a ton of manual programming when it comes to machine learning. If you look at just a standard machine learning workflow, it's not just like you take a machine learning algorithm, give it data, and magic happens. No, you actually have to pre process and clean the data first. You have to reshape the data so it's in a good format to actually apply machine learning algorithms to. You have to choose the right machine learning algorithm for the problem. And that's not as simple as just choosing your favorite one. There is no such thing as the best machine learning algorithm for all problems. And then you also have to tune not only the machine learning algorithm, but you have to tune all the pre processing, pre processing steps that came before that. So tune their parameters, so on and so forth. And so usually this is a highly iterative process where you start with some basic analysis and you maybe try to fix a machine learning algorithm on it doesn't work well. So you go back to the drawing board, you try a different algorithm, you try different parameters, what have you, and you easily spend hours, days, maybe even weeks just stuck in this loop, depending on how knowledgeable you are about that process and how large the data set is that you're working with.

Absolutely. And I, I occasionally joke that it's. There's as much an art to it as there is a science, right?

Oh, absolutely, yeah. I mean, there's, you can, there can be a huge time savings by just having, you know, experience, you know, personalized experience with the domain that you're applying the machine learning algorithms to and also the algorithms that you're using. Maybe a beginner, it would basically be a random search, whereas an expert knows where to jump depending on the problem. I should also point out, by the way, that it's the same issue in deep learning where when you start with a, let's say, classification problem that you want to use deep learning for, you have to decide on an architecture, you have to decide on the activation functions, you have to decide on so many different things about that neural network that it's also a huge iterative process that's really pretty much a black art.

So these are things that we could potentially automate. Yeah.

So when I looked at this whole iterative process here, I said to myself, there's no way we can't automate the majority of this process. And it turns out when I started my so this was a conclusion I came to at the start of my post hoc back in 2014. And it turns out I wasn't the only person who had that idea. So, so there's this whole field called automated machine learning where the idea is that we want to automate this whole process of not only tuning the parameters of machine learning algorithm, but choosing the best machine learning algorithm for the problem, or even best combination of machine learning algorithms, choosing the best pre processing steps, so on and so forth.

And so in terms of what we can automate at the moment, what's the lowest hanging fruit for automation?

Well, right now I'd say that the lowest hanging fruit for automated machine learning is basically anything where you can make a change or tune the machine learning pipeline. Where a machine learning pipeline is basically the steps from reading in and pre processing the data all the way to actually fitting the machine learning algorithm where you can make any change to that pipeline and then measure some sort of goodness of that solution. That can be some loss function or anything else, anything that you can compute and measure. Right now most of the focus in automated machine learning is on supervised learning. Of course, because you can fit a machine learning algorithm, calculate it's let's say five fold or tenfold cross validation accuracy and use that as a measuring stick of oh, did this change I make, was that a good change or a bad change? That's definitely one of the major focuses right now. Another one that's come to fruition in the past year or so has been deep neural network architecture search. So I briefly mentioned before that the whole deep learning artificial neural network architecture design process is very similar to tuning a machine learning pipeline. So you can apply the same sort of optimization process here to help you figure out the architecture of your artificial neural network.

That's really interesting because there are. We do know that certain architectures are better at certain domain challenges. Right? Like you'd use something different for image classification than you would for text classification.

Exactly.

These are the things that are the lowest hanging fruit. What do you see being the most difficult to automate with respect to machine learning?

Yeah. So basically automated machine learning will be able to automate the easier aspects of machine learning. Right. You know, choosing the pre processing algorithms, choosing machine learning algorithms, tuning the parameters. And that's what's great about that, is that it's going to allow humans to focus on the more difficult aspects of machine learning and data science. So there's two aspects that come to mind for me right now, one being technical. So in a sense of working with messy, really heterogeneous data, maybe data that comes from the web, for example, that can be very, very difficult to work with in an automated fashion, that usually requires someone to sit down and parse out this unstructured data and figure out how to structure it properly on the other end. There's also a creative challenge that is difficult to automate, and that's really more thinking like a data scientist in the sense of looking at a business problem or even an academic problem and asking, how do I translate this problem from a business problem into a data problem? What data should I collect to tackle this problem? How should I collect that data? How should I label that data, so on and so forth. Those are things that are still very well within the human domain rather than the automated machine learning domain.

Yeah, absolutely. And I think this also speaks to the fact that automated machine learning doesn't mean automated data science, right?

Oh, absolutely. You know, I think that we should be looking at automated machine learning and automated machine learning tools as more like data science assistance rather than replacements for data scientists. For sure.

Right. And your project teapot is actually called a data science assistant, right?

Exactly. So there's no coincidence there. That's definitely been part of the branding of that project.

We'll jump right back into our interview with Randy Olson after a short segment. Let's now jump into a segment called Data Science Best Practices with Spencer Boucher. What are we talking about today, Spence?

Today I thought it would be a good idea to talk a bit about data sets and where they come from. When you're playing around with a new data science technique or teaching one to somebody else, you're always going to need some data. So today's data science best practice is to broaden the collection of data sets and data sources that you're familiar with. So I for one, am a little bit sick of classifying IRIS flowers every single time I learn a new classification algorithm. The IRIS dataset is just so pervasive because it has this great strength of being included in a base installation of R. But we can, and we definitely should be spicing things up as much as we can. What built in dataset are you personally sick of, Hugo?

Well, let me think. I've definitely analyzed and visualized the mtcars dataset enough times now that I would definitely not say no to having something new.

Yes, that is another data horse that I think it's safe to say we've beaten to death a little bit as a community. Other data sets like the MNIST data set for classifying handwritten digits have also become very popular in recent years. And while MNIST is a much larger example, that isn't just a toy data set. I think we can still do better.

So where can we go to find cool data to play with?

Spence well, this is the good news. Open data has just completely exploded in the past 10 years, so there's no shortage these days. One of my favorites is the UCI machine Learning repository, which has been around for just ages, at least in machine learning years, and it's currently up to, I think, 427 data sets curated to be especially interesting for machine learning problems. Each dataset is even tagged with the number of features, the number of observations, types of features, and even which type of machine learning problem that data set is ideally suited for. So if, for example, you want to play around with a new classification algorithm you just learned that only works for categorical data, you can quickly find a data set that fits that set of conditions just right.

Great. And also, 538, for example, publishes the data behind most of their analyses right on their website. This is a great and reliable source, especially when you're interested in hot button political topics or sports data.

Yes, and if it's government data you're after, you can go straight to the source@data.gov I personally recommend poking around the Federal Reserve economic data that's got historical data for everything from unemployment rates to interest rates to GDP and much more. Plus, just a couple of weeks ago, Rafael Irizarry, a professor of applied statistics at Harvard, released a new R package called dslabs, which contains data sets that he found pedagogically useful in data science courses that he's taught over the years. One of my favorites in that package is actually a data set from the popular blog Spurious Correlations. So the whole point of the Spurious Correlations blog is to demonstrate how easy it is for things to be correlated even when there's clearly no causal relationship, by scanning a huge amount of historical data and cherry picking relationships. So that one is really great and fun for teaching about correlation and causation.

Awesome. So also Reddit actually has a data set subreddit that features a lot of bizarre, up to the moment topical stuff. All of Trump's tweets are there for example, new repositories of data like the others we've mentioned will also bubble to the top in this subreddit as they get launched.

Yes, and here's another really good one. Kaggle now lets users submit data sets for other people to upvote as well, all the way up to 20 gigabytes. Some really cool, wacky off the wall stuff winds up there. Everything from Kickstarter projects data to the brainwave data of confused students from EEG is on the front page even as.

We speak right now.

Thanks Spence. We really only scratched the surface here. Next time you find yourself loading the mtcars data set, spice up your life a little instead and go find something new. Spencer, I'm also glad that you mentioned Kaggle's open data platform. Next week I'll be speaking with Anthony Goldblum, the CEO of Kaggle, about their open data platform, Kaggle kernels, the future of data science, and much more. Kaggle is now more than a platform for machine learning competitions, so make sure to tune in after that interlude. It's time to jump back into our chat with Randy. So maybe you can tell us about the main tools that exist in the automated machine learning space.

Yeah, sure. So I'll focus more on the tools that do more than just hyper parameter tuning. There's a lot of tools out there. Even really popular packages like SciKit Learn have the ability to do hyperparameter tuning through grid search and random search and other means. So I would say that the top projects in terms of the free and open source space are going to be Auto SKlearn and auto weka. So auto sklearn was of course built on top of scikit Learn and Python. Auto weka was built on top of WEKA and Java. There's Teapot, as you mentioned, which is a project that I was working at while I was at University of Pennsylvania, which is also built on top of scikit Learn in Python. And then I actually recently discovered, well recently in the past six months or so, there's a very nice package built by a company called H2O and they just simply call it H2O AutoML. And they have a very nice platform that works in a whole bunch of languages, has a nice web interface and everything else, and also free and open source. And then on the more commercial side, there have been some very nice commercial automated machine learning tools out there, primarily Datarobot, which is a really, really interesting tool. It's worthwhile to go and look up a demo. And then there's also H2O driverless AI, which is H2O may sound familiar. Basically they took their automated machine learning platform and then added some secret sauce and made it a commercial product as well.

And if you were suggesting someone who wanted to get started with AutoML, where would you suggest they go and why?

Oh, I would strongly suggest going to one of the, I would say one of the open source tools right now. And I mean, I guess it depends on their, on their proficiency. If you're familiar with programming and data science techniques and especially if, let's say you're familiar with scikit Learn, I would strongly recommend going to, you know, Autosk Learn Teapot one of those tools and just apply it to either one of the example data sets that either of those projects supply. Or even better, apply it to one of the projects in your own work or your own hobbies. We've designed these tools to be very, very easy to basically take a data set that you're working with and take a workflow. Let's say you're working in a scikit Learn workflow. It's extremely easy to take Autoscalearn or Teapot and plop it right in there and boom. Now you're using automated machine learning.

Right. And my understanding is that those two actors kind of wrappers around scikit Learn Pipelines, right?

Exactly, yes.

And for those listeners out there who may not know too much about scikit Learn Pipelines, these are ways of combining a bunch of pre processing steps and then your machine learning algorithm in a really intuitive fashion. The API is very nice there. And so Teapot, now I'm just looking at it on GitHub, you have 32 contributors, nearly 4,000 stars, nearly 600 forks. This is pretty exciting, right?

Wow, Is it almost up to 4,000 SARS? It has been very exciting to see the community response to this project. I think that's been part of the driving force behind it, especially lately. Now that I'm not getting paid to develop Depot, it's great to see that, getting feedback from the community in terms of how it's been useful. What are the challenges of the tool? The response for this project has been great. I think it shows that there really is a strong demand for automated machine learning tools out there. Not just research into it, but actual tools that people can use and integrate into their day to day workflows.

That's right. And I suppose my next question is, you have a lot of contributors. Do you get much insight into the industries it's used in or the questions people are trying to answer when they use Teapot for example.

So unfortunately no, people are very, most of the time when people. So I have seen some blogs where people write about, hey, I'm using it for this problem on this open data set. But I think actually a lot of the users of Teapot have been at companies on closed source projects. And so usually when I have people reaching out to me about that, they're very close lipped about it and they talk to me in a very general sense of what kind of problem they're working on and maybe even provide generated data. But I really wish I could get a better sense of the exact problems that people are applying it to.

Yeah, I think so. Do you have any thoughts about which industries you see these types of automated machine learning packages being capable of having the most impact?

So yeah, I think industries where there isn't a strong data science presence already will have, will gain tremendous value from these out of machine learning tools. Because the basic idea of these tools are to, you know, allow someone with a minimal amount of experience to take a data set and run a machine learning analysis on it at the level of someone who is fairly proficient at machine learning.

Right.

So the idea, the whole idea behind machine learning is we want to take the brain power of, let's say all those people on Kaggle and all the smartest data scientists out there and put it into a single algorithm. So if someone who's just coming into the field of data science, they want to bring machine learning into their field, there's tremendous value there. And being able to just basically pip, install a tool and the next thing you know, you're running a really advanced and really accurate machine learning analysis on your data.

I like that a great deal. In your mind, is there any danger in it then being misused? I mean, this is a general question actually. As data science tools and their associated APIs become more and more user friendly, more and more intuitive, is there a challenge in terms of them being misused by people who don't understand the underlying math, for example, and misinterpreting the results?

I mean, yeah, I would say that that's always been a concern, even before automated machine learning. Right. You know, I think that the authors of Scikit Learn, for example, heard this a lot. When people are like, oh, what are you doing? You know, let's keep these machine learning algorithms difficult to use so only the people who really know them and implement them from scratch can actually use them. And so I think this is just a common thing that you hear every time we make something easier and easier to use so, yes, there is technically a danger there. Right. And that someone can build a machine learning system that gets really, really high accuracy, but for all the wrong reasons. That's why I think it's important to think of tools like this that make machine learning and similar powerful tools more accessible is you still want to keep an expert in the loop whenever possible to help you interpret what you're getting. So you can get a really good solution from AutoML, but you still need an expert there to help you with the actual application and interpretation of what you've got. And so there's still value there, of course, because you didn't have to do, you didn't have to hire a whole team of data scientists or machine learning engineers to implement all of that software and do the whole machine learning analysis for you, but there's still definitely a need to have those experts.

Yeah, I like that a lot because that also really highlights the importance of communication and education as well, as opposed to suggesting that or stating that these are skills that are reserved for the few who've been educated at a certain level, actually opening it up and saying, hey, you use this, but let's, let's keep the communication going both ways as well and see what we can both learn in that space.

Yeah. And I think that's where the science and industry, especially science is going, is much more collaborative space. Right. You know, when I was at the University of Pennsylvania, it was quite common for us to sit down in a meeting where we had a genetics research professor there, we had physicians from the hospital there. You know, we had people with statistics background, people with a pure computer science machine learning background, like myself. You know, it's. Yeah, I think it's, it's in line with that whole idea of, you know, having a broader collaborative approach.

Yeah, cool. And something you've spoken to there is the importance of interdisciplinary research in data science. And I think it's really interesting that you can have a bunch of different people in the same room working on different problems, but they're essentially using the same types of techniques. And I think that automated machine learning is that we're going to see more and more people from diverse fields using it.

I certainly hope so.

Yes.

So I want to talk about when you mentioned what the main tools are in the current landscape. You mentioned both open source and commercial tools. One thing you mentioned though, is that you were funded in some sense to work on Teapot when you're at UPenn.

Yeah, right. Well, I wouldn't say I was necessarily funded, funded directly, but I was very generously funded by my PI there, Jason Moore, who is funded by the nih, to develop machine learning methods and applications that have a biomedical focus. So even though TPOT is a general machine learning tool, we found it tremendously useful in the particular biomedical applications that we were working great.

And I know there are lots of questions around viable models for funding open source software development these days. Is this something that you think is a good way to fund such projects?

Oh, I mean, I think that's how a large portion of open source projects have been funded, at least, you know, large scale ones is either through basically public tax funding.

Right.

You know, I was funded by the nih, but there's plenty of other projects that have come out of NSF funding and numerous other agencies in the U.S. and across the world. It's not just the U.S. but also, you know, I think that even at commercial companies there is a huge opportunity there to also fund open source development. Because when it comes down to it, a lot of us are working on very, very similar problems. If you take a sort of very high level look at the field of data science, when it comes down to choosing the right machine learning algorithm, tuning hyperparameters, so on and so forth, we're almost always facing the same kind of problem there. So there's no reason that we can't all work together on that and provide a common, hopefully better and higher quality solution there in the open source space that will add value not only to the company that's funding it, but also to the data science community in general. That's actually been a major focus of, shall I say, my preaching in the Life Epigenetics team is pushing, saying, hey, we're not necessarily a machine learning company. We're not going to be selling machine learning as a service, we're applying it. How about any algorithms that we develop as a part of our work here, let's make sure to publish them as open source because it's actually value both ways. We provide value to the open source community, but the open source community provides value back by providing feedback, more code refinements, bug reports and fixes, so on and so forth. That was definitely something we saw.

Yeah. And I'm sure as you say, more and more, well, issues, but also probably pull requests. Even better. Rolling on in. That's a really exciting experience.

Yeah, it's a great way to start the day. When I see a good pull request coming on one of my repositories, that's awesome.

So we spoke to the fact that there are Both open source and commercial tools currently for automated machine learning. What does the future hold with respect to open source tools and commercial nature of such tooling?

Well, I think so, yeah. So like right now most of what we're seeing is basically the open source automated machine learning tools for the most part are research prototypes that are coming out of research labs that are pioneering some particular way of doing automated machine learning. Whereas the commercial AutoML tools for the most part are relatively simpler than the research prototypes. You know, they're obviously still high quality and scalable and whatnot, but they tend to focus more on the scalability and especially the user experience, making it easy for a non expert to use the tool and less so on the advanced optimization techniques or whatever else that you'll typically see in the open source research prototypes. But I think going forward we're going to see more companies like Google and H2O that I think are run by really smart, really good people who invest in both. Where you can have a automated machine learning tool out there that is open source and it's something that you develop, backed by your company and also supported by the open source community. But then you can also have, let's say a specialized version of it where you put in, I'll call it the secret sauce that makes it work even better, maybe on a particular problem domain or maybe just in general. Right. You know, a better optimization algorithm or whatever else. And that can be your commercial version. Right. So you can support and have most of the tool out there as open source, but still support a business model. It gets companies to buy in and I think that works really well because when something is free and open source, it's an easy buy in.

Right.

You just send one of your engineers to try it out, they tinker with it for a day, maybe a week, whatever, and if it works out really well, they become interested maybe. And maybe they become interested in your commercial product, you know. So I think that that's going to be a great way to get people to buy into AutoML tools. But of course, thus far I've been a career academic, not a businessman, so who knows, maybe I'm just talking on my phone.

So you think though that open source tooling and commercial tooling will exist kind of in parallel or one on top of the other in in future?

Yeah, I mean that's at least the way I see it. Yeah, well, I mean that we'll see. I, I generally try to avoid predicting the future when it comes to tech because most of the time we're wrong.

Yeah, Absolutely. Because I wonder just with, with thinking about data science tools in general, Python and R a lot stronger than a lot of their, I suppose, proprietary counterparts these days. Right. They've been in the open source software, has been really embraced by the data science community.

Absolutely. Yeah, that's true.

So we hinted at this before when we discussed what won't be automated. I'm going to ask a provocative question which I don't necessarily agree with, but my question is once we automate everything away, what will we be doing?

Well, actually, I don't know if we'll. Well, hold on, I guess I should back up and I should ask, when you say automate everything away, do you mean all the data science work or do you mean everything, like everything we work on today as human beings?

Yeah, I mean the data science work.

The data science work.

Okay.

Well, you know, I'll be honest, I don't think, I think I've thought down that path quite far enough because I don't actually see us automating all of data science away. And I sort of alluded to this earlier that, you know, automated machine learning is more of a data science assistant and I see it being that way for quite some time. You can think of an automated machine learning tool as acting as like a junior data scientist level. So let's say, yeah, there's still going to be a need for that high level of guidance of these automated machine learning systems. Right. It's going to be extremely difficult to automate something like how do we translate this business problem into a data science problem? And so we're still going to need these sort of higher level managers for thought of in machine learning systems. Now I should also point out, by the way, that I've been very excited to see that several companies and groups have started basically throwing their automated machine learning platforms at Kaggle. And they've been fairly consistently scoring and let's say the top 1% to top 10% or so. But they never quite get in the top two or three of these competitions. And I've been talking to basically everyone I can find that's in automl and especially in automated machine learning in Kaggle. And the issue that always comes up is that there is always some really smart feature engineering technique that sort of brings you to the next level in terms of solving that problem. And usually it requires some sort of domain expertise or some knowledge of heuristics or whatever else. And sometimes it can even be a simple feature engineering technique technique. But that's what you needed to take the machine learning algorithm to the next level. And that is extremely difficult to automate. We could sit down and think about thousands of problems and possible custom feature engineering techniques for them, but whenever we enter a new domain, who knows if those will be useful or not. So I think there's several aspects of data science that will squarely remain in the domain of humans for quite some time, at least until we start getting closer to really higher level artificial general intelligence.

Could you give any example of such feature engineering?

So, okay, so here's one example that I ran into where there was a kaggle problem where, oh gosh, it was a couple of years ago, so I'm not remembering the exact details, but it was basically, the features are certain hands in poker, right? So you get basically each card that you have in your hand and you want to classify what sort of hand this is, is this a straight, is this a pair or whatever else. And the data set gave it just very basic information, basically, what card is this? What color is it? I think that was it. And then you had to classify it. And so I threw teapot at it and it actually did pretty poorly. I was kind of disappointed. But then some people were scoring way up there, like nearly 100%, if I remember right. And the reason behind that was because they had expert knowledge about playing poker, they were able to construct features that weren't necessarily the answer, but they constructed more useful features that the machine learning algorithm could then use to properly classify the hands of the cards. That I think is a nice, really basic example of a custom feature engineering technique that requires some domain knowledge and can be tremendously valuable for a classification problem. But it's something that would be very, very difficult to automate.

Yeah, absolutely. And that I suppose that type of feature engineering would capture interactions between cards in some sense.

Exactly, yeah. And capturing feature interactions is a huge, huge challenge in machine learning.

We'll jump right back into our interview with Randy after a short segment. We're back with another principles of Data Science education segment with Spencer Datacamp's very own curriculum lead.

Absolutely right. So today I want to highlight some strategies for learning data science. Data science is as broad or as narrow as you want it to be. And that means that whether you're diving deep into like one specific area, or if you're taking a breadth first strategy and trying to become a generalist, there's always going to be more to learn. If there's one thing you can be sure of in your career as a data scientist, we can at least guarantee you that you'll Never gonna feel like you've learned it all.

If there's so much to keep learning, Spencer, how can someone stay on top of it all?

Oh, there's so many different ways. One of the best is just to find an RSS service that you like and start following the work of whatever your favorite data science bloggers are. Everybody's got a blog these days, and once you find that subset of data scientists that are doing work you admire, you can let them show you what needs to be on your radar.

Any other things to keep in mind?

Yes. So there's a whole body of cognitive psychology literature behind the science of learning, and honestly, it doesn't get talked about enough. We actually know a lot about which study patterns are objectively more effective than others. For example, one big mistake that people make is to chunk their learning into these big one time events and then move on to something else when they're done. For example, if you're trying to learn Python for data analysis, spending a whole afternoon on a book or an online course in Pandas is a great idea. What's an even better idea though? Make sure you return to that same material on a regular basis over an extended period of time. This strategy of spaced repetition is going to give you a much better bang for your buck in terms of the amount of time that you spend.

So Spencer, how can you make sure you stay engaged with what you're learning?

Well, one of the best ways is to dive into something practical. So write a blog post, participate in a Kaggle competition, track down a data set that you find interesting, and flex your muscle on it. At datacamp, we have an ever growing list of projects that can get you working on the skills that you need to learn in either R or Python on real data. This is also one of the big reasons that we're developing datacamp for mobile. If you're strapped for time, you can leverage downtime like your hours on the train to internalize your data science and programming knowledge over the long run. If you're still learning the basics of R, Python or SQL for data science, you can practice that material on the mobile app on a regular basis, which is going to improve retention of the content dramatically. If you're especially interested in the science of learning and you want to make your process for learning data science as data driven as all of the great things that you can do with that knowledge, I recommend checking out the Learning Scientists blog and podcast@learningscientists.org we'll put that link in the show notes for you. Go forth and make your data science education just as data driven as the work that you do with it.

Fantastic, Spencer, look forward to speaking next time. Time to get straight back into our chat with Randy. So one other thing you mentioned was you see AutoML as a data science assistant that may be able to do things that a junior data scientist could do. I'm wondering, there's a challenge currently in the data science landscape which revolves around training junior data scientists and data science career paths. I'm wondering how, you see, how would that be affected by having AutoML to do these types of tasks.

Well, okay, so the way I see it is a really good automated machine learning system is going to free not only junior data scientists, but all data scientists from this sort of monotonous circular process that we often go through in terms of pre processing, standardizing and modeling the data. So I think it's going to allow people to really focus more on general problem solving. That's something I've been hitting on again and again here, which is, I think.

That.

More important to me than the technical skills of being able to program and knowing about machine learning algorithms is really thinking more, more about what is the problem that we're working on and how do we solve that as a data problem. Right. That is such an important skill that I always stress with all of the students and all the people I work with that unfortunately sometimes goes overlooked in some of the machine learning courses out there. And so I think that it's going to allow people to focus more on that rather than the nitty gritty of, you know, how does a random forest work or how does an SDM work or whatever else. And I think that's actually a really good thing.

Yeah, I like that a lot because it really speaks to highlights two things. One is being able to think and actually ask the correct questions and also give the correct answers. But also it speaks to a translation challenge in terms of we've got a question about a business or about the world in academic research or about biology, and we need to translate that into our data science question and our technical question and then translate it back to the real world. And I think those are the types of things where our time would be much, much more well spent.

Absolutely. Yes.

We've spoken a lot about machine learning and automated machine learning. I'm wondering what one of your favorite data sciencey techniques or methodologies is.

Yeah, so I think anyone that looks into the research papers that I publish won't be surprised by this. But I would say one of my favorite techniques that I Always try to plug, wherever I go is evolutionary computation. And it's actually been a very exciting year for evolutionary computation because it's sort of like becoming popular again. It was popular for a while, but now that people are finding applications for deep learning and everything, people are becoming interested in it again.

Yeah.

And so the thing that I really love about evolutionary computation is that it's very easy to understand conceptually. The basic idea of evolutionary computation is that we want to take this process called evolution that we observe in nature that produced basically all known life on Earth and perhaps even elsewhere, and we want to bottle that in the computer and use it as an optimization technique. And so it's really easy to understand conceptually. You can go through the, let's say, four or five steps of the algorithm and understand at a high level and know what's going on without having to pore over dozens of pages of math, which is especially important for me because I'm not a super mathy person. And I think it's also great because it has a very flexible solution representation. So basically whenever you're using evolutionary algorithm, you say, hey, this is what I want my solution to look like. And evolution algorithms can work with basically any kind of solution. You're not stuck with artificial neural networks, you're not stuck with a machine learning model or a graph or whatever else. You can use basically whatever you want. And that means that you can apply it to almost any kind of problem out there, which is what I've tried to show somewhat on my blog. And then also the third great part about it is that it has this concept of a fitness function which basically allows you to tell the algorithm exactly what you want out of a solution. So, you know, in machine learning, typically we have to use some sort of fixed loss function or, or accuracy or whatever else. With the evolution algorithm, you can say, I want my solution to only cost me $5 or whatever else. You can specify that kind of criteria directly in the optimization process itself. And also not to gush too much about evolutionary algorithms, but I think another great. Yeah, I think another great part about it is that because evolutionary algorithms consider multiple solutions to a problem simultaneously, you can actually use this, this idea called Pareto optimization to basically look at a trade off between multiple things that you want from it. Right. So to give some very explicit examples, let's say in machine learning, typically we're making some trade offs between, let's say, the complexity of our algorithm or the interpretability of our algorithm and the predictive performance of that algorithm. Well, in A Pareto front sense. You can think of that as basically a curve in solution space where you can look at this trade off between how complex is my model and how accurate is my model. And that's especially nice because it lets you decide on what trade off you want as an end user. If you want simply the most predictive algorithm out there and you don't care about complexity, then choose that one. If you're working in a space where you need a really interpretable model, well, then maybe you're willing to sacrifice a little bit of accuracy for that. And I think that's a really, really great bonus that you get. Just because of how evolutionary algorithms are designed.

Absolutely. And that actually that also once again highlights the type of things that data scientists spend their best time doing. Right. Thinking about these types of questions and while we're automating away the other stuff.

Exactly.

Great. So if somebody wanted to get started with evolutionary computation in Python, for example, where could they do that?

Well, there's a number of libraries out there. The one that I usually use myself and recommend to people is called Deep. It's D E, a P. Usually if you type like D E A Python into Google, it'll pull up that package and they have, they've basically written up all kinds of different evolutionary algorithms that are out there and have some really easy to use examples and interfaces that make it really easy to jump in and start using evolutionary computation in just 10 minutes or so.

Cool.

I think that's especially great. I always try to encourage people, especially when they're coming into the field, to use a pre existing package because like I said, the greatest part about evolutionary computation is it's really easy to understand and it's actually really easy to implement because of that. There are probably thousands of implementations of evolutionary algorithms out there because everyone says, oh, I can do this better, I'm going to implement my own. But I think the Deep developers have done a very, very good job of covering most of the bases that you'll need when you're applying evolutionary computation to a problem.

Good.

And we'll link to Deep in the show. Notes also. So my final question is, do you have a final call to action for our listeners out there?

Yeah. So can I have three?

Yes, of course you can.

All right. I feel, I feel like, you know, I rubbed a bottle and a genie came out and I said, can I have more wishes? Yeah, so you know, these are all related. So, so at least there's, there's that. Okay, so my first one, I'm going To say is in machine learning, stop using only your favorite machine learning algorithm. It doesn't matter what that is, whether it's a deep neural network, whether it's a random forest, whether it's XG Boost, whatever else, stop only using that algorithm when you approach a machine learning problem. We all know that there is no such thing as a one best machine learning algorithm for all problems. So why do we ignore that, especially when we become experts at this, why do we ignore that advice and focus in on just our favorite algorithm algorithms? And I should point out, by the way, that this is the most common source of value gained from automated machine learning platforms is that data scientists come in and they have, or even machine learning engineers come in and they have their handful, maybe two or three favorite machine learning algorithms and they get vastly outperformed by an automated machine learning system. Who is considering perhaps dozens of machine learning algorithms that I've actually seen many companies write and talk about this, where they say that was an immediate source of value from automated machine learning. It broadened our horizons in terms of what we're considering in the machine learning space. The second call to action that they'll have is to stop using default parameters for machine learning algorithms. This one sounds almost obvious to anyone who's used machine learning before, but it's provable that in Scikit Learn, for example, that the default parameters are almost always bad for most problems. We actually did a research paper on this just last year and improved that. So there, there is always a good way to tune the parameters of your algorithm and gain 5 to 10% accuracy or, you know, predictive, predictive performance, whatever metric you're using.

That's really interesting. And actually, I think I recall you tweeting out one of the key figures from, from that paper last year.

Yeah, exactly. Actually, that might be a good thing to link to as well. We have several visualizations in there showing the results of that really huge experiment. Took way too much computation time to do, but basically showing that there's always value from doing model selection and there's always value from tuning the default parameters. And related there stop using grid search for tuning parameters. That's such an antiquated thing. Now we have random search, we have better optimization, parameter optimization methods based on Bayesian optimization and other techniques. There's no reason to use grid search anymore. Basically there's always better methods for that. And then my third call to action, which might not be unexpected here, which is to start using automated machine learning tools, whether it's at work or in your personal Hobby projects or in Kaggle. Automated machine learning will definitely give you a competitive advantage. Now, while it's new, a handful of people are using it and it's incredibly easy to integrate into your workflow and it's going to make you look like a much more productive person. Trust me. I know some people who have used Teapot, for example, to solve a problem and Teapot actually has the ability to export the corresponding Scikit Learn code. And they basically said, yeah, that was me. Give me a raise now.

Thanks.

If you do start using this automated machine learning tool, tools especially Peapot. But you know, let me know. I'm curious to learn about where people are applying it, how well it's doing, what are the challenges that it's facing. You know, I think having that kind of information and sharing it with the community is going to be tremendously valuable. So also, you know, definitely drop my email in there so people can easily get in touch.

Yeah, I will, I'll do so. And also let us know at Data Camp at Data Framed. And it's a running theme on this show, on this podcast to encourage listeners to blog and put jupyter notebooks or markdown files up there. As you know, Randy is a great testament to. So if you start using these tools, write a short blog either on your own blog or you can write on our Data Camp community about it. Because that would be a lot of fun, I think.

Absolutely. That's actually another one of the top things that I always recommend to students working with me is to start a blog, start putting this code out there, start getting feedback from the community and learn how to take that feedback. Right. You know, sometimes people are not so nice, but maybe there's an ounce of truth in what they're saying. So yeah, I think it's such a great way to interact with the open source community and the data science community.

Yeah, absolutely. Randy, it's been an absolute pleasure having you on the show.

Thanks again for having me. I had a blast.

Thanks for joining our conversation with Randy about automated machine learning. We saw how AutoML, as part of automated data science assistance work, will increasingly take much of the drudgery out of data scientists work. From pre processing to model selection and hyper parameter tuning, freeing up data scientists to work more on using their skills to pose and answer the interesting and important questions in their research, discipline or line of work, whether it be insurance, pricing, medicine or astronomy. One of my favorite takeaways of this episode were Randy's three calls to action. One, branch out in terms of the machine learning models you use, two do not use the antiquated technique of grid search anymore and three Start playing around with automated ML technologies such as Teapot. I know that I will make sure to check out our next episode A conversation with Anthony Goldblum, CEO of Kaggle if you thought that Kaggle was merely a platform for machine learning competitions, prepare to have your mind blown as mlcomms account for less than a third of the activity on Kaggle. Today, Anthony and I will discuss Kaggle kernels for reproducible data science and the evolution of the Kaggle public data platform. To do so, we'll make detours through the genesis of Kaggle and how Anthony managed to solve the cold start problem of building a two sided marketplace. We'll discuss the exciting implications of Kaggle recent acquisition by Google for the future of cloud based data science. And we'll hear why Python is dominating on Kaggle. I'm your host Hugo Bown Anderson. You can follow me on Twitter hugobown and datacampatacamp. You can find all our episodes and show notes@datacamp.com community podcast.