#17 Biology and Deep Learning

Hi, I'm Maggie, VP of Curriculum at.

Datacamp, and I know firsthand how overwhelming it can be to keep up with.

The rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated.

The best learning content to help you.

And your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Sebastian Rajka, a machine learning aficionado, data analyst, author, Python programmer, open source contributor, computational biologist, and occasional blogger. Sebastian has also recently accepted a position as Assistant professor of statistics at the University of Wisconsin, Madison State, starting this summer. Congrats, Sebastian. And congrats to the University of Wisconsin, Madison. Today, Sebastian and I will be talking about the role of data science in modern biology and the power of deep learning in today's rapidly evolving data science landscape. How is Sebastian using deep learning to build facial recognition software that also prevents racial and gender profiling? Stick around to find out. I'm Hugo Bown Anderson, a data scientist at Data Camp, and this is Dataframe. Welcome to Dataframe, a weekly Data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter hugobown and datacampattacamp. You can find all our episodes and show notes@datacamp.com community podcast. Hi Sebastian, and welcome to Data Framed.

Yeah, hi Hugo. Thank you for the invitation. I'm wondering what questions you have for me, and I'm happy to talk about everything you like.

Fantastic. And I've got so many, so many questions for you and I'm really excited to have a general conversation about data science, about data science in biology, machine learning, deep learning, the Python ecosystem, all of these things. But before all of that, I want to find out a bit about you. What are you known for in the data science community?

Oh, yeah. So, yeah, before we dive into the many topics you just mentioned, I think we can cover maybe, or we can record an audiobook almost, if we want to cover everything. So yeah, what I am known for. So I started blogging. I think it was the beginning of grad school. It was usually in the Evening hours after eight hours of research, during my Ph.D. and, and I Wrote the articles on R and Principal Component Analysis and Python and that's maybe how it started. And then at some point I signed up at Twitter and I started tweeting and people started following me and we had interesting back and forth on Twitter and yeah, somehow I got hooked into this social media thing and that's maybe how people know me these days when I go to conferences, people say, hey, I know you from Twitter and stuff like that, but all of your work.

Here actually led to you writing more and writing a book in fact. Right?

Yes, the publisher contacted me. I think it was probably based on my blogs. And they said, hey, you know about machine learning and Python and would you like to write a book? Somehow that was kind of appealing to me because writing blogs is nice. But then I thought, okay, I have so many connected topics and if I could put them into one kind of storyline, that would make sense.

Yeah.

But honestly, I saw that they already had so many books out there on machine learning and Python. That was the publisher pact. And I declined the offer. But then I talked to some friends and my friend said, hey, that would be a nice thing to do. I like your stuff and your tutorials, you should definitely write a book and blah, blah. And so, yeah, then I kind of reconsidered, emailed the publisher, hey, you know, I said I wasn't interested, but somehow I changed my mind and would you still be willing to have me as an author on the book? And yeah, that's how I wrote the book.

And it's called Python Machine Learning and it's in its second edition now, right?

Yeah. So last year they contacted me, hey, we would like to update the book. People like it and it's so useful to people and we wanted to have some additional chapters on TensorFlow and things like that. And we were developing these methods for biometrics, also based on TensorFlow. And then we thought, okay, we can maybe have some chapters on TensorFlow. And then we partnered up and with Waheed.

Fantastic. And in parallel with all of this, you've also been highly involved in the open source community, right?

Yeah, that is one thing. So I write a lot during my work and I code a lot during my work, but somehow when I get home and sometimes I feel like I need to do more coding. I mean, so I'm pretty active on GitHub, so I have my own, like open source packages. So for example, I think one, I don't know when I started it, it might be 2014, it's this ML extend package. And for that one, I Was I'm also contributing to scikit Learn sometimes. And sometimes though I have to implement algorithms related to machine learning or data science in general that, that I can't find anywhere else. For example, ensemble methods. The voting classifier, which is also ported now to scikit Learn though, but the stacking classifiers and there's a stacking classifier for. With cross validation, which is very popular on Kaggle, I think because I get many emails with questions and tips if I have tips for them because they. A lot of people use that in Kaggle competitions. And yeah, that is about machine learning. But yeah, I also do a lot of biology stuff. So one other package maybe would be biopandas, that's a very simple package that reads protein structure files and let's say just files of molecules into Pandas data frame. So the coordinates and atom specifiers. And here the goal is. So in biology we have all these different tools and every tool has an API to parse these files. And it's pretty confusing. You have to kind of relearn an API like almost like learning a programming language to deal with these files. And I thought, why don't we just use Pandas? Because everyone knows Pandas and it has a very powerful selection syntax and you can do a lot of things in Pandas that you usually also do, so you don't have to relearn all the commands and stuff. I'm always happy to contribute to the community to make stack of tools that we have even better if I can.

I like the idea and the philosophy behind biopandas. The idea of actually taking, I mean, a lot of work has been put into Pandas and there's a lot of machinery there and leveraging that for other, other disciplines, which it wasn't necessarily created for, and opening, you know, the world of biologists and people thinking about our protein structure and molecules. In that sense, to be able to leverage Pandas is. Is incredibly cool.

Yeah, it really, I mean this is very efficient. And also on the other hand, if you don't know Pandas, that may be encouraging you to learn Pandas and then you can use Pandas for all kinds of things or the other way around. There's a high chance that you already know Pandas, so you save a lot of time and you know that the things you're doing are working. Because Pandas, I mean, has been developed for, I don't know, five years, seven years. It's pretty robust. There are not many bugs compared to a quick implementation reimplementation for certain projects and things like that, yeah.

And what it means is people aren't reinventing the wheel and that software development isn't happening in. In discipline silos as well, which is incredibly important. How did you get into data science originally? What's the backstory here?

So, yeah, I mentioned my blog. So I was blogging and I think. But before I started blogging, I was a biologist and at some point I was becoming a purely computational biologist and I needed to analyze a lot of data and I got into R because R was popular back then. It might have been 2010, 2011. And so I did a lot of R. And there's also small, a very small book I wrote. It was called Heat Maps in R. But the problem in computational biology is you often have to collect your data yourself from certain sources. And R is not really great at that. I mean, R is a great tool for analysis, but if you have to do some custom stuff, if you have to write some code that grabs some data here, cleans it up, merges it with something else or all the processing stuff, yeah, that is usually not that convenient. R. So I picked up Python. I was using Perl before, but somehow I needed to use a package that was in Python. I saw the language and it clicked with me. So I was doing Python and R and doing some very basic data analysis. But yeah, that was usually not enough. So I, I was pretty good at coding, but I was a little bit lacking the, I would say the methods for doing predictions and statistical modeling and things like that. So there was a great class at MSU taught by Aaron Ross. He was back then a new professor. He just joined. It was the first semester he taught The Class A CSE 802 statistical pattern recognition, if I remember correctly, was, I think was six, seven years ago. So what I liked about this was it was mostly based on base statistics, pattern recognition using base classifiers. That is maybe how I got into this data science, machine learning stuff. So I was so excited about this class that I did a lot of reading, a lot of studying on my own and started applying these techniques to my projects and took other classes. And I had my own little side projects. There was one, I think it's also maybe already four or five years ago, the Music Mute, a moot project where I tried to predict the mood of music music of songs based on the song texts. And I did a lot of like hobby projects in data science that got me really excited. And at the same time, I learned something that I could use for my research projects. And yeah, maybe that's. That's how I got into data science.

That's, that's fantastic because a lot of budding data scientists at Data Camp and elsewhere ask me how to get into data science. Should they learn Python or should they learn R or SQL or whatever it may be? And I, my response always is don't worry about the technology yet. Find some questions you're interested in and figure out what tools you'll need to answer those. And that, I think your, your backstory really speaks to that, that every tool you picked up in that version of your story in any way you picked up in order to solve either biological problems or hobby projects you're interested in. Very much motivated from a, from a practical point of view. And I think that's, you know, one of the best ways to learn. Right.

Yeah, I mean, tools, honestly, they are useful also, but they come and go. You know, it's like a tool is a tool. I mean, ultimately it's about the problems you want to address or things that you are really excited about, projects that you want to tackle or problems you want to solve. And tools help with that. But it's, I think it's not very motivating to start a class and learn about a tool because then you, you don't have any, maybe motivation to use it or you don't know why you want to use it. Getting into data science by having a passion for solving problems, that is maybe the most important thing.

Yeah, I couldn't agree more. And so that's why we're here, to talk about data science in biology. Right. So before we dive in to this intersection of data science and biology, I'd like to step back a bit and just talk about biology in general in terms of what some major questions in biology are and what are your research interests? What do you really like to think about? What keeps you ticking?

So, yeah, so biology. I mean, if you ask me about biology, there are so many, many, many different project problems in biology that need to be solved in so many different fields. It's, it's not so simple so that we have a lot of things going on in biology. And just to give you maybe a quick overview too, before I tell you more about my stuff. So in biology we have, like, I mean, Titus Brown, for example, was on my committee. He, he was working mainly, or he's working mainly on genome analysis, assembling genomes. I'm not sure if you heard of. You probably are familiar with DNA. That is the code that in our body that is responsible for generating everything like the cells and the proteins and things like that. And a lot of people in, I would call that field maybe bioinformatics, a lot of people ask questions and address problems based on DNA. They analyze genes. Genes are like the functional units that encode something. Looking at them, for example, detecting early diseases or biomarkers for certain diseases and even, I mean, like sequencing a genome, they sequence that in little chunks and somehow you have to assemble these chunks. That would be one very challenge, I mean, a challenging and interesting problem area because that is becoming more and more affordable. And if people, let's say, can get their hands on these data sets, I mean, you can, I think, sequence your genome for 200 bucks or something now. I mean, there are a lot of interesting questions to address there based on the genome level. Or you have the whole microbiology community, like people who work with bacteria, like taking soil samples, looking at certain properties of the environment and things like that. Or looking at metabolites and little markers in the environment for certain things. Or the. The zoologists and plant biologists, for example. I had a friend from soccer here, that's a kind of funny thing. They call it the hyena lab. So they study the behavior of hyenas and grad students get sent to Africa for like a year or something, which I think is very exciting. And so they get to be outside more than other grad students, I think a little bit of sunshine and. Yeah, so their stuff is also really connected to data science in terms of. They use a lot of statistics, like basically mainly Bayesian statistics. And so it's not only. I mean, if you say, okay, the person is mainly observing animals behavior, it doesn't mean it stops with the observation. You have to analyze your data, you have to draw conclusions.

It's.

I think it's all data science in, in a sense. And so. But what I'm mostly doing is things related to protein structure modeling and drug discovery. So maybe to explain what a protein is. Not sure how many listeners heard of proteins. I mean, most of the people who are listening to this podcast right now probably know proteins from the nutrition facts exactly.

Not necessarily in the biological or molecular sense.

Protein. To summarize what a protein is, it's a blob in your cell that has a certain function and it's made up of. It's called amino acids. So we have 20 different amino acids in the human body. And each amino acid is basically a small molecule that is composed of course of atoms. So I would say, top of my head, amino acids. I mean there are 20 different ones they are composed of 10 to 30 atoms. And if you connect these amino acids, you get a string of amino acids. And if you fold the string up, you get a protein. But if it's, it's only functional if it's correctly folded. And this is a big challenge in biology. If you have the sequence of amino acids, how they are connected, how to predict how it would fold, because then you can study the function of proteins and the analysis of diseases. You can understand what's wrong with a certain protein and why, what does it do in the body. So protein folding is maybe the most challenging field in biophysics, I would say.

Yeah. And, and, and proteins are responsible for so many things, right? Like transporting molecules, response replicating DNA, all of this type of stuff.

Yeah, exactly. So DNA is basically just the code that gets translated into proteins, the which are the functional units. So you have a code like a letter code. And this letter code there is a machinery which itself is protein machinery. So it's kind of a chicken egg problem. What was there first? Right, but this machinery translates the code into other proteins that have a certain function in your body. It could be receiving signals like our eyes. They receive a signal, light signal, it's all everywhere proteins are involved, or cells, transporter proteins who transport, let's say, sugar into your cells or break down sugars. And I mean it's, it's basically the functional, the functional units of our body. I mean maybe a good example would be think of DNA as if you are a coder. Think of it as C code. You're writing some code and then you compile it into a program and the compiled program that would be maybe your protein. And if you have a lot of these compiled programs, each has a different function. In Linux you have the GREP function, you have the find function, you have, I don't know, you have so many different functions. And if you put them all together, all the different functions and little programs, you get maybe your cell or your human body and things like that. So the functional units like this little Linux or Unix programs, the command line tools and things like that, these would be your little functional units, the proteins. My job was mainly working on problems that involve proteins. But proteins, they have so many different functions. And many of the proteins bind to small molecules. The small molecules kind of regulate these proteins. You can think of it, you have a lock and a key. And proteins are usually very specific. They only bind a certain small molecule. And if this small molecule binds, it has a certain function. And for example, could be as a, I think it's cyclooxygenase molecule involved, I think, in inflammation and things like that. And there's a certain signaling molecule in your body that does something with it. But we can also develop drugs that can bind to these proteins to either over activate it or block it. And for example, aspirin would be a common one where you have a small molecule and a small molecule, if it blocks to this protein, it inhibits the function, then the headaches go away, for example. And as you can imagine, I mean, there's, there are a lot of use cases for, let's say, drug discovery. And so what we were mainly working on was, or what I was mainly working on was related to the understanding how proteins and small molecules bind, how we can predict how they bind, because that is your basis for finding drugs, for example.

We'll jump right back into our interview with Sebastian after a short segment. Now it's time for a segment called Data Science Buzzwords with Data Camp curriculum lead Spencer Boucher. What have you got for us today.

Spencer Today we're going deep on deep learning, one of everybody's favorite data science buzzwords. Everybody's doing it, or at least they claim to be doing it.

What exactly is deep learning?

Spencer well, in a certain sense, Hugo, the answer to this one's pretty easy. Deep learning is neural networks.

That begs the question, though, what's a neural network?

So a neural network is, is just one type of machine learning model that maps a set of inputs to a set of outputs just like any other. They get their cool name because they're loosely inspired by the way that human brains work as a network of neurons arranged into connected layers. A particular input from one layer will either activate or deactivate the neurons in the next layer. If you're familiar with logistic regression, you can imagine a neural network as just a bunch of logistic regression stacked on top of one another.

So where's the deep part come in? Ah.

So one of the most well known papers referencing deep learning is Geoffrey Hinton's Learning Multiple Layers of Representation, which was published in 2007. Hinton was literally just referring to the geometric deepness of a particular class of neural network. A lot of the cool advancements in machine learning since then, like convolutional neural networks and recurrent neural networks, they've essentially boiled down to strategies for setting up the geometry of the neurons in a neural network. And because they all tend to involve many, many layers, the phrase deep learning has stuck these etymological waters. They've been muddied a little bit in recent years, though, because popular implementations of neural networks like Word2Vec, for example, are often referred to as deep learning, even though they actually only consist of two layers of neurons. It's honestly up to you whether you want to be a stickler for the term or play it a little more loosey goosey.

So why are people so excited?

Mostly just because you can do so many impressively cool things with them. One of the first big waves that deep learning made in popular culture came as far back as 2012, when a classifier was trained to recognize faces and cats from YouTube videos. Even without having any label data on either faces or cats right now, neural networks, they're still at the bleeding edge of our coolest technological breakthroughs. They can generate descriptive captions for images from scratch. They can understand human languages, both written and spoken, and even translate them back and forth. And these days, they can even drive your car. Actually, some modern versions of deep learning are Turing complete, which means that they're really starting to look more like a programming language than any simple mathematical model.

So it's all sunshine and roses then?

Well, even a tool as powerful as deep learning comes with a set of downsides, of course. For one thing, they generally require a lot of data, which isn't always available or very cheap to collect, as you know, Hugo. Plus, the incredible complexity that powers a deep neural network also makes it really difficult to understand. It can be hard to determine exactly why a neural network is working at all, or the reasons that it may have made a particular decision, although progress is being made in the interpretation department. Slowly but surely. For applications that require transparency and understanding, though, you're often still better off with a more interpretable algorithm, like a tree based method.

Thanks for helping to demystify deep learning for us Spencer listeners. If you want to find out more about deep learning, make sure to check out next week's episode with Michelle Gill, a deep learning expert at Nvidia, an artificial intelligence company that builds GPUs, the processes that everybody uses for deep learning. After that interlude, it's time to jump back into our chat with Sebastian Rajka. It'd be nice to know what type of role data science actually plays in in these challenges for you.

Yeah, so yeah, a good project maybe would be, but it was a large scale statistical analysis of different properties of proteins and their small molecules that are interacting with them. And what was kind of interesting, I mean, people are studying these protein small molecule interactions for, I don't know, 50 years, 100 years, or at least like the whole Last century. And we were taking a data set of non homologous proteins. That means they were all kind of different, they looked different. They were not just duplicates or very similar proteins and a whole set of different molecules, small molecules that were bound to them. So they knew. We had the data set from a database where they crystallized them so people know the 3D coordinates of how they are bound of both of the structures. And then we analyzed how they interact with each other. And for that we brought a tool to assign, it's called hydrogen bonds, a bond between an atom on the small molecule and the protein. And in order to do that, we had to implement a lot of rules, like geometric constraints, distance constraints. So was a lot of coding. That's where the coding part came in. But then we assigned all these hydrogen bonds to our data and then we wanted to analyze them to find trends. And what we found, I mean, it sounds very simple, but it's actually a big thing. So that ligands, there are two types of atoms, like acceptor atoms and donor atoms. For hydrogen bonds, we found that ligands on average have twice as many acceptor atoms than proteins. So twice as many acceptors than donors. So there's a high bias in this kind of interaction. And that has, I think, very important implications for drug discovery. If you are designing, for example, ligands, you may want to design ligands in a way, or small molecules in a way that they have more acceptance because it's maybe more favorable. And also when we looked at the statistics and the geometry, acceptor atoms, they have a wider range where they can accept hydrogen bonds, and donor atoms have a very, I would say, narrow range. And if one thinks about this, I mean, this is where maybe the data science, the problem solving, the thinking, the questions come in. So if you have a very narrow range, that makes you more specific to something. And if a protein has a very, very narrow range of accepting hydrogen bonds, so proteins have mostly donors, and if the donors are more geometric constrained, then one might argue maybe that has to do something with this to make proteins more specific for small molecules. Because we have a lot of small molecules in an oil environment or in our cells. We just don't want our proteins to bind to all kinds of things. So to make them more specific, that nothing gets out of order in our cells, that it only binds to things that it is supposed to bind to. I mean, a very, one might say a very simple analysis. But we sliced and diced the data sets by atom types and things like that. But yeah, I would say there Was a lot of data science application behind it or the statistics?

Yeah, absolutely. And I think, you know, seeing that the impact on the ground involves, you know, drug discovery is incredibly important and seeing that statistics plays a role in that. When you're discussing protein and small molecule interactions, graph theory, as you said, plays. Plays a role. There a lot of computational algorithms needed to be implemented. Was there any machine learning that, that you used for this type of stuff?

Yeah, so machine learning, it was one project where it was particularly useful. It was not about drug discovery, but very related to that. So drug discovery is usually more like a later stage where you find, let's say, small molecule that inhibits or activates a protein. And then if it goes to clinical trials, let's say, involving humans, then you, you speak maybe of more like the real drug discovery part. But in early stages of drug discovery and inhibitor discovery, what you do is you want to find candidates that are to be tested in lab or experimental assays. And in this project, it was not about a drug, but a small molecule that influences the mating behavior of certain fish in the Great Lakes. That sounds maybe funny, but there's species in the Great Lakes and it is very harmful for the environment because these fish are predators of other fish. And the native fish. And the native fish, basically the populations, they decrease. This fish population increases. It's bad for the fishery. And then the fishery or people try to get rid of this problem by having small toxic molecules in the lake that kills the larvae of that fish. But then also it affects other fishes again. And so we thought about an environmentally friendly approach. What's kind of interesting about this fish is the sea lamprey. What it, what it does, it's the female fish are swimming, or the male fishes, sorry, are swimming to a small river or two rivers attached to the Great Lakes, and they build nests. And once they are done with the nest building, they secrete pheromones which attract the female fish from the lake to the nesting sites. And then they have, like, little babies, little sea lampreys. And we thought if we can prevent the females from smelling this pheromone in the water, then they might not swim to the nesting areas. So we wanted to find a molecule. And of course, like always, there's a protein involved, a protein that binds to this pheromone and that causes in the brain the signaling pathway. It is then our goal was basically to find a molecule that also binds to this protein, but instead of activating it, blocking it, so occupying this binding site so that the real pheromone can't bind. So the question now was, okay, where do we start? So, yeah, what I did is, or where I started was there's a database out there. They have 12 million commercially available molecules. The three, three structures of it. Okay, that is maybe a starting point. But our collaborators, they are experimentalists in zoology and they do all the experiments, very hard work. And although they work almost, I would say, 20 hours a day or something, based on what I've seen, based on their results, they must be very busy, but they can't test 12 million molecules. So my part here was to narrow it down to, let's say, I think in the end we had 299 molecules that we ordered and tested for one iteration of this project. And, and you want to find molecules that work at first, you just screen them. You find similarities. You've tried to, you have a model, a computational model of your protein. You want to see if it is potentially fitting into the binding pocket of this computational model. You want to compare it to the real pheromone to have something that is similar. Because then you think, okay, if it's similar, it might have the same effect, but it doesn't tell you if it's activating or inhibiting it. And once we got some results though, from our collaborators, once they tested a bunch of molecules, we could use machine learning to see or to predict, to first to deduce what's important, what makes the molecules active, what do all the active molecules have in common?

And what were the features or what, what type of features were you using to predict?

So good, good question here. What features I focused on was just the atoms themselves. So I over, I had the real pheromone and I overlaid my molecules with a real pheromone in the way that they would are predicted to bind to the protein. And then I could see, okay, what do they have in common? What kind of atom types do they have in common at certain positions? So my features were just the atom types, the, the position of the atoms and what kind of atom it is. If it is, let's say, an oxygen, a nitrogen atom, because for example, an OH group and hydroxy group, it's an oxygen with a hydrogen that can accept or donate a hydrogen bond. It can make two different kinds of interactions. Or we had a negatively charged sulfur atom at one end, which turned out to be the most important part for binding or for triggering the signaling pathway.

In the end, were you able to predict it quite well?

So, yeah, we found a molecule that blocks in the experimental assays where they put electrodes into the brain of the fish and measure the electric signal. One molecule was for instance, able to block it by 92%. So almost completely the signaling by the pheromone, I have to honestly say what they do right now, it works. So it kills the larvae. It has some bad environmental effects. It's not that bad if you use the molecule in a right concentration. It's not too toxic. But it's always good to have a say, environmentally friendly approach and also a kind of alternative approach because usually like, you know, from ensemble machine learning stuff, it's usually good to combine different things to get even better results.

Right.

Cool.

And I'm really excited to see, see what happens next with this. So we've discussed, Sebastian, applications of machine learning in biology. I noticed on the back of your Python machine learning book, it actually says machine learning is eating the software world. And now deep learning is extending machine learning. And I know how passionate you are about deep learning. And in fact that you've started writing a new book on artificial neural networks. I'm wondering if you can talk about the impact that deep learning has on biology or on some of your work.

Yeah, so with a deep learning topic, I think, I mean of course we have all the hype, but it is really useful. And I think also right now what is really interesting to see is recently people started using deep learning for solving certain problems in biology. For example, synthesizing new molecules with autoencoders and predicting the toxicity of molecules. And the results looked amazing. I mean these are of course academic results. How much of that is used in, let's say, outside of academia? And honestly, I also have to admit that my deep learning work is more in biometrics, not biology. So biometrics, that's. There are two parts of biometrics. There are two interpretations of that. On the government website, for example, they talk about biometrics as something related to biology. But I mean here biometrics related to biometric features, let's say human face recognition and things like that. But before I go to that, maybe you can put that in the show notes because I recently bumped into Alexander Kalinin. He's in Arbor and he wrote an awesome manuscript. I think it's about 100 pages. It's on Bio archive and it's a review of deep learning in biology. And if someone is interested, I think that is maybe the starting point or to see what deep learning, what impact it has on biology. Yeah. So my biometrics projects. So I was recently working on a new Neural network architecture. We called it semi adversarial neural networks. The challenge is in biometrics, you have all these great algorithms that can deduce all kinds of features from a face, for example, but the problem is they can also get misused. For example, if you have an image database, you just want to, let's say, detect, okay, you have a database of criminals and you want to just cross reference. Let's say you have a camera at the airport and you want to see, okay, is this person in my database? So you collect data, you have images, or you have different image databases. So, but I mean, it's not intended, but what might happen is people can misuse the data sets, they can do a profiling agenda, profiling, for example, or by age or things like that. And we were trying to kind of find a method to prevent that. So I have this idea of using an autoencoder, our semi adversarial neural network. Our goal here was to change a face image. If you have a face image in a way that a classifier can't predict the gender anymore, so that you hide the gender from the face image. But at the same time, the image should still be useful to a face matchup. Let's say you want to identify a person, you have, let's say, a database, and you want to see if this person is a real person in your database, that should be still possible. But at the same time, you shouldn't be able to use this image for any kind of profiling, let's say, deducing the gender, having statistics about what. Let's say if you have a online, not an online store or a physical store, and you have pictures of your customers, you, you shouldn't do profiling on that or stuff like that because it's like a privacy issue, right?

Yeah, well, that's incredibly, I mean, profiling, particularly with respect to deep learning and machine learning algorithms, is going to become increasingly a greater issue. So this has, this creates real value immediately.

And this work is really exciting because it's something that is, that can be also used for other things. The semi adversarial neural network structure. So what it basically does, it's a dual objective that we want to retain one objective, for example, the face matching, while confounding another, for example, extracts the gender information. And the challenge here was because deep learning, as you may know, I mean, that's a huge hype, or was a huge hype this year, or not hype, but a big topic last year, the adversarial attacks. So Neural networks are super easy to exploit, to fool, to make just very small changes that throw them off completely. But yeah, so for our method I mentioned our method should hide the gender information, but the problem then is people can just use a different classifier. They can use traditional computer vision systems. So our method also aims to target all methods to not only apply to a certain model that comes with our architecture, but if someone has a commercial classifier or some other software that does gender classification, it should also not be able to work on this image anymore.

We'll jump right back into our interview with Sebastian after a short segment. Now it's time for a segment called Statistical Pitfalls. I'm here with Michael Bettencourt, applied statistician and one of the core developers of the open source statistical modeling platform Stan. What's up, Mike?

Hugo, Today I'd like to talk about when your best just isn't good enough. Well, at least when your best fit isn't good enough.

What do you mean by a best fit?

Any statistical analysis uses data to learn about a system and then inform decisions about how we interact with that system. For example, we might observe user traffic to learn how users behave, and then use that knowledge to determine the optimal user experience that we can present to them in the future. An important step in that process is how we characterize what we've learned. A common approach is the best fit, where we consider only a single configuration of our system that is in some sense most consistent with our data. For example, we might consider only the maximum likelihood or maximum a postiori estimators in a statistical analysis.

A best fit seems like a reasonable way to characterize what we've learned about the system, no?

Well, it does have a certain intuitive appeal, and that appeal is definitely promoted by the myriad of statistical packages that prioritize delivery of a best fit result. Unfortunately, finite observations will never give us a complete picture of our system, and utilizing only the best fit will lead us to more fragile decision making.

Can you go into that a bit more?

Absolutely. The ultimate problem is that the information encoded in our observations will always be limited. For example, the inherent variability of observations obfuscates the system being studied. Moreover, observations are often partial, limited to just a subset of the population of interest, or just a short window of a time dependent phenomena. Reporting a best fit ignores these limitations and assumes that the observations completely characterize future interactions with our system. Ignoring what we didn't observe leads to poor generalization and ultimately poor decisions. For example, consider the problem of designing an optimal user experience. Optimizing that user experience to only those behaviors seen in a small study disregards the experience of unobserved users, or even observe users whose behaviors may vary in time. These limitations are especially pertinent when we try to scale our user base to new audiences. Additionally, the limitations of a best fit aren't just limited to business applications. Many of us have made the mistake of jumping to conclusions and social circumstances where we assume someone's motivations from limited information instead of giving them the benefit of the doubt by considering all of their possible motivations.

So how do we avoid the limitations of a best fit?

In order to facilitate robust decision making, we have to consider something with which many of us are inherently uncomfortable, and that's uncertainty. Cognizant of the limitations of a finite observation, we have to be careful not to focus on just the simplest or the easiest explanation for what we've observed. We instead have to consider all of the explanations consistent with our information and devise a response as compatible with as many of those explanations as possible. We may not be able to exactly infer the behavior of an entire population given just a few observations, or the full time dependence of a system given data in just a short window, but by reporting our uncertainty of the behavior in those circumstances, we give ourselves the best opportunity to succeed.

So what's the takeaway here, Mike?

When making decisions or communicating the results of an analysis with stakeholders who will make a decision, we have to prioritize uncertainty. Instead of focusing on the best fit, we have to consider confidence intervals, posterior distributions, or simply narratives about the range of possible behaviors in our system. It's an uncertain world out there, Hugo, and ignoring that uncertainty will punish you sooner or later.

Mike, thanks for that introduction to the statistical pitfall of the best fit.

As always, it was my pleasure and I can say that with certainty.

Time to get straight back into our chat with Sebastian Rajka. So with all your work in deep learning, for your research, also there's the software development stuff you do, the books you're writing, you have a lot of insight into the inner workings and the societal impact of deep learning. Where do you see the future of deep learning going?

Yeah, so that is. Wow. Interesting questions. So interesting questions. So top of my head, I would say the adversarial text that I just mentioned, I think that is an issue. If we want for certain applications it's not so important. But let's say in biometrics for example, it's super important that these methods are robust and secure. And I think that is maybe a one focus area where People try to make these methods more robust and it will of course require more work in this area. I think Ian Goodfellow has a website on that and all about adversarial attacks and things like that. That is very important. But also I saw a lot of people are now finally ready or I mean, to get more into the. How, how do deep learning. How does deep learning work? How can we interpret our methods? There was, I think Jan Lecun had a discussion on Facebook, he started that, that people tried now to further understand what deep learning really does. I mean like beyond the black box thing. I mean, we understand how the algorithms work, but really what our models deduce, what kind of information they did use and how.

And this speaks to interpretability and representations as well, right?

Exactly, exactly. And also, I mean, as I mentioned earlier, deep learning is becoming more popular in biology and that will hopefully also become more mature that the ideas get really implemented into real applications that are used more widely. And for example, on Monday I went to a seminar was also interesting where people use deep learning for basically doing the opposite. Not extracting features from data, but using features to generate the data. And also, I mean, last year, Hinton's capsule networks, that's something I think will be a hot topic this year especially. And also there was like this. How did it go? The quote by Hinton where he said we should throw it all away and.

Start again with respect to back prop.

Yeah, yeah, yeah. And so that is also interesting. Interested to see how that goes. I mean, I think there's some, I mean, of course a big kernel of truth here. These are all kind of classic methods. But here's also, I think the distinction between deep learning research or the development of new methods versus applying these methods to solve problems that were currently challenging. I mean, we don't have to. For that. We don't have to necessarily mimic the human brain. It is maybe, I mean, if a logistic regression classifier was enough and the certain good predictions on certain data sets, I mean, the goal here is to make good predictions and not necessarily, let's say, mimicking the brain or having different methods out there that make different kinds of predictions. It's more about. There are two different goals, I think.

And of course regressions and linear models have the added bonus that they're interpretable in some sense.

But yeah, there are also other interesting methods now. I think what you just said is very interesting that. So, yeah, for example, yeah, model interpretability is becoming more and more popular now. I mean, I think it was about two Years ago they, there was the LIME method which stands for local interpretable Model Agnostic explanations. So it's an interesting idea where they locally approximate the predictions to have a locally faithful model. And from that, like in a logistic regression model or linear regression model, you can look at the coefficients and see how each variable contributes to the overall prediction. And there was also recently a new method, I think it was called SHAP which stands for Shapley Additive Explanations. There was ODS C seminar and that is I think the next step which is particularly trying to address the accuracy interpretability trade off issue. And it, I think it combines six different methods. I think when I looked at it it was, they mentioned six different methods, LIME is one of them that they kind of unify to have an even better method to make approximations to interpret very difficult models or black box models which are not so easy to interpret in terms of what makes them work and what features do they use to make the prediction.

Exactly. And I think the ability to take black box models and make them somewhat interpretable with a trade off for accuracy in, in some sense allows us, it gives us so much power in terms of asking the right questions and giving the right answers to a wide array of people who may not know, who may not be technical.

Right. And here this is also a very important point. I think in any kind of scenario there's always something you have to think about. It's do I want to just. Do I only care about accuracy or predictive performance or is interpretability important? And there's usually a, I would say a trade off like a kind of no free lunch Ethereum. Yeah, there are different kind of objectives. And what is more important to you? That's a very important question to ask. And yeah, with these methods you basically bring the benefit of having something interpretable to the models that are maybe better performing but not so interpretable. So it's helping with this kind of trade off thing.

Absolutely. So we've been discussing all types of data science techniques and methodologies. I'm wondering what's your favorite data science technique or methodology or something you love to do or love to use when doing data science?

I like the bootstrap a lot, you know, like that you can do more like the simulation to get confidence intervals instead of memorizing all the formulas and making assumptions about normal distributions so that you just simulate your simulation I think is, I like simulation a lot. Like bootstraps and stuff or the permutation test. If you do a hypothesis testing that you don't have to remember, okay, do I have to do a T test? Do I have all the assumptions here correctly and does my data look okay for this kind of test and blah, blah, blah, so I can just simulate it, use a permutation test. And yeah, I would say these are, that's cool.

And the great thing about these, these tests or these approaches to tests are you, as you say, you don't have to make many assumptions or have any underlying models or.

Yeah, they are non parametric and don't require any. I think it's something I think from the data science toolbox that is only now possible because we also have the computational power and I like that a lot. It's very elegant, I would say.

Yeah, me too. And I think for anybody out there who hasn't had much experience with the bootstrap or permutation tests, I can definitely recommend a lot of Alan Downey's books and I'll include one of those in the show notes. And also we've got a couple of three courses on data camp called Statistical thinking in Python courses which do all the statistical stuff through resampling techniques, which was a lot of fun to create with my collaborator Justin, Justin Boyce. So we've talked a lot about modern data science in biology, what the deep learning landscape looks like currently. What does the future of data science look like to you, Sebastian?

Oh yeah. So I would say the future of data science is, I think a lot about that is about the community. I think that a lot of people realize that they can benefit from a lot of tools. And yesterday there was a tweet that I retweeted a talk by Titus Brown and he said something along the lines of that we should maybe also teach data science from a domain specific perspective instead of just presenting tools. And here's the tool, learn the tool and maybe you can do something with it in your project. I think it's interesting to also introduce people from a domain specific perspective so that let's say I have a biology problem and then I throw in some data science techniques to solve a particular problem. And I think by that people more and more people will discover that data science techniques, computer, a little bit of programming or the use of statistical packages and modeling or a little bit of machine learning will be useful in their work and that more important, more people will be incorporating the tools that we have in their, let's say, everyday life, making it a standard toolbox for every maybe researcher, but also programmers and other people. I think where this is going, I mean we have the core Right now we have Numpy, we have Scipy, we have Pandas and scikit Learn. Very mature tools, very powerful tools. And I think the baseline for everything that got developed in the recent years, not the baseline, but the core, I would say, and everything else, I think is becoming more and more interesting. But it's built on these things and we're getting, we are getting really interesting tools like Dask, which parallelizes things like Pandas in parallel, basically, and out of core learning or out of core computations. And Pandas is also currently being redesigned, Pandas 2, which is making the underlying data structure or data representation more efficient. And there's a lot of improvements in terms of, hey, these are the tools that we use to solve different problems. But hey, let's scale that up, make that run faster, more efficiently and let's improve what we have. But then on the other hand, we have also new methods, new methods that may take a little bit of a different approach, combine different things. For example, I always mention it's a really cool tool, Randy Olson's teapot, which is a really cool idea. So he's taking one of the core packages, scikit Learn, and he uses genetic algorithms to evolve optimal data science pipeline, machine learning pipeline, which involves not only the hyper parameter tuning, but also feature extraction methods.

So, and this is, this is what we're really talking about is automated machine learning in some sense.

Exactly. That is what I was trying to get at.

In fact, I'm going to have Randy on the show at some point to talk about automated machine learning. So that's going to be very exciting.

Oh, that is really cool. Now that I don't want to take away too much, but yeah, that is what I wanted to get at that Machine learning, automating, machining, not machine learning itself, but the whole data science maybe pipeline, that it is becoming more accessible that it is, I would say we always will have experts that really focus on that because at some point, I mean, there's so much a machine can do. You can automate, I think everything. But sometimes, I mean, it is useful to have experts, but not everyone has to be an expert.

Yeah. And I think it's important to actually, you know, people are scared. There is a general fear that will data science be automated. And I don't necessarily subscribe to that. I think we can automate a lot of the stuff such as hyper parameter tuning and a lot of the machine learning techniques to free us up to do more enjoyable, creative work as data scientists. Right.

Yeah. I mean, maybe it's not the best analogy. But back then when cars were just being invented, everyone who wanted to, let's say, test or drive a car had to build their own car, but just to get from A to B or something like that. But now, I mean, don't everyone has to know how to build a car in order to drive, right? So I think with data science you automate these tools to make them available and there will be always, I think there is no reason to be afraid of something because I mean, it's never about the tools and things like that. It is about the questions you ask and the problems you want to solve. And a machine you can't automate the asking of questions because it is about what you are interested in, what you want to solve, what problem you want to solve. And that is something that involves the human. And also on the other hand, you have the two kind of fields, you have more the applications and you have the research. And the research will never go away. You will always want to improve things and develop new methods. And for the application, why not automating things that are boring, like you said, hyperbolic tuning. Because then the research, the application is mostly focused on addressing the question, if you can address the questions more easily, why should you make your life hard and implement low level code if you can automate the tedious stuff, right?

And you've mentioned so many interesting things in here. One thing. So to paraphrase what you've said, one part of the future of data science looks like all the tools that have already been developed actually spreading through society, spreading through a lot of different disciplines to help people answer questions that have yet remained unanswered. The next is to build on top of these tools to build kind of the next layer of data science packages and libraries in the open source. And then there's the making everything more convenient to automate things so that we can concentrate really on the really fun stuff. Sebastian, we've come to the end of this, this great interview. I'm wondering if you have any final call to action for aspiring and well, well seasoned data scientists out there. What would you like to see the community do to charge forward?

Just making tools more available in data science, if you develop something that you think is useful to others to share that. And also I think we need to get better a little bit at rewarding open source efforts. And I think it would be nice to have like a reward system in academia that it is not only about how much papers you get out, but also how you contribute to society, like how, how useful your Tools are for helping others because it's not only about your results, it's sometimes or it's very, very, very often the tool that allows others to do their research. And the tool is often also research itself.

I agree completely. And I think the incentives of course need to become far more aligned so that open source software development and actually developing tools for working research scientists is rewarded in an academic setting in a way that it isn't quite yet, but hopefully we'll see that in the coming years.

Yeah, and I talked so much about academia now, so maybe from an industry perspective, what industry can do is I think a lot of the tools, of course there are sponsors and things like that, but a lot of these tools that we use, we use for free. And also a lot of industry folks are using software, open source software for free. And it would be just nice if it's possible for the company to kind of sponsor those projects to allow people to kind of reward people who are working. It's mostly a free time effort, but there are maybe in future some more full time employees that help developing open source software because I mean, there's only so much you can do in your free time. And the thing is, it would be really nice if more companies for example, would, let's say, sponsor or fund things like that, open source software that they use themselves, that they find useful. And for example, like PYPI was for example, industry funded and it's right now, I think funded by a new grant from the Mozilla Open source Software support. And this is super nice and I think if industry could do a little bit more in supporting open source software, that would be another great call to make.

Yeah, it would be great to see it be a very strong established part of the culture. Sebastian, it's been such a pleasure having you on the show.

I enjoyed it a lot.

Fantastic. As did I. Thanks for joining our conversation with Sebastian about the role of data science in modern biology and the power of deep learning. In today's rapidly evolving data science landscape, we saw how integral data science and open source software development are to modern biology, from genomics to ecology and biophysics to researching protein structures and the implications in drug discovery. Sebastian also showed us the importance of deep learning and his semi adversarial neural networks in building algorithms that can identify faces from databases and simultaneously obfuscate forms of profiling such as racial and gender profiling. Make sure to check out our next episode, a conversation with Michelle Gill, a deep learning expert at Nvidia, an artificial intelligence company that builds GPUs the processes that everybody uses for deep learning. We'll talk specifically about the modern superpower of deep learning and where it has the largest impact, past, present and future, filtered through the lens of Michelle's work. At Nvidia, I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBaun and DataCamp. At DataCamp, you can find all our episodes and show notes at datacamp. Com Community Podcast.