#98 Interpretable Machine Learning

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or or you're a data leader looking to scale data driven decisions in your organization. Join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in. Hello everyone, this is Adele, Data science educator and evangelist at datacamp. A major challenge for both practitioners and organizations creating value with machine learning is model interpretability and explainability. We've seen a lot of examples over the past few years of machine learning disasters, whether it's recruiting systems, rejecting candidates based on race or gender, credit scoring systems, penalizing disadvantaged groups, and much more. This is why I'm excited to have Serge Macys on today's podcast. Serge is the author of Interpretable Machine Learning with Python and for the last decade he has been at the confluence of the Internet, application development, analytics. Serge is a true polymath cross. Currently he's a climate and agronomic data scientist at Syngenta, a leading agribusiness company with a mission to improve global food security. Before that role, he co founded a search engine startup incubated by Harvard Innovation Labs and he was the proud owner of a bubble tea shop and much more. Throughout our chat we spoke about the different machine learning interpretability challenges data scientists face, the different techniques at their disposal to tackle machine learning, interpretability, how to think about bias in data, and and much, much more. If you enjoyed today's conversation, make sure to subscribe and rate the show, but only if you liked it. Now onto today's episode. Serge, it's great to have you on the show.

It's nice to be here.

Awesome. I'm very excited to speak with you about Interpreter Machine Learning, your book on it, Bias in data, and all of that fun stuff. But before, can you give us a bit of a background about yourself?

Well, at the moment I'm a data scientist in agriculture. Yeah, there is such a thing. A lot of people are like what they figured it only like high tech like companies, data scientists work. I have a background in entrepreneurship. I created a few startups and for the longest time the way I defined myself was like a web app person. Someone that did all kinds of web related things whether on websites or on mobile devices. Yeah, that's what I did. I went from being a builder also to be like have more like a more managerial role as well. Yeah, and basically that sums it up. But I've done a bunch of other stuff. I was also very interested back when I was deciding what to study. I was interested in computers for sure. But the application of it where like I had interest in graphics, graphic design, 3D modeling, anything in that space which now I get to work with the other side of it which is computer vision, which is also very interesting. Computer graphics and computer vision are like two sides of the same coin. I also once owned a bubble tea shop. Well, something to note about all the roles I ever worked with, even the bubble tea shop, is that data was always there. Data was throughout my journey. I just did not know it was my true love. It's like this story where your true love is hidden in plain sight and you don't realize was always in the background. And I've only like in the last seven years brought it into the foreground what I do.

That's awesome. So I would love to talk about that bubble tea experience but let's talk about interpretable machine learning given this is the topic of today's episode. Well, I think there's some understanding today within the field on why machine learning models need to be understood and interpreted. I'd love to set the stage for today's conversation by first understanding the why behind interpretable machine learning. So I'd love to understand in your own words, if you could set the motivations behind why you wrote the book and why this topic is so important.

Fundamentally machine learning comes to solve incomplete problems so we shouldn't be surprised when the solutions are also incomplete. We tend to think, okay, we, we achieved a high level of predictive performance and therefore the model is ready to be out in the world and it's just going to work like a charm and it does it. So that's just the nature of things. That's the nature of machine learning. And if it were that simple in which it were just like any other bit of software, we wouldn't solve it with machine Learning, we have procedural program, if statements and so on. That would be our solution. And the other reason why it's important is for ethical reasons. And unlike other technologies before it, this is a kind of technology that replaces that something that has become very human in a sense. It's not like other animals don't make decisions, but the kind of decisions we make have a level of, for lack of a word, intelligence that is aiming at foreseeing well beyond, like, our immediate needs. So we're not just thinking of the next meal and everything. We're thinking, we're thinking of grander things, organization or society, supply chain, all sorts of things. And so there's a lot of reasons why we would use machine learning. And in a way, they aim to replace our cognition. So the thing is how to trust a model the same way we trust a human? I would dare to say there's a lot of reasons not to trust a human in the first place. But ultimately we want to trust something that we can understand. And if it's a black box, why are we taking orders from a black box? That's the grand scheme of why to use interpretability. The reason that brought me to the book is I became aware of the topic when I had a startup back in 2017, 16 and so on. I became frustrated that I couldn't debug my own models. As someone that had programmed for so long, it was just so strange that there was this thing blocking me. If I wanted to figure out, why isn't this working? It would always, not always, but a lot of times point to the model and it was like, okay, why is the model doing this? At the time, there was very little resource on it, at least for practitioners. Everything was like in academic circles and there wasn't a clear understanding for me coming from outside, what does this mean? Right. So as someone that's obsessed with decision making, I found this intriguing and concerning. So I felt like, weird to be promoting a technology I couldn't understand. That got me into a rabbit hole. And then I started to learn all the terminology, read all the papers. Then eventually some books came out. My book was the third book for practitioners on the subject to come out. I thought, how late in the game does it have to be for it to be like the third book on the subject? So that's it.

That's really great. I'm really excited to talk about the book and underline the challenges into the solutions that you discuss to a lot of interpretability challenges that data scientists face. But before I want to kind of harp on some of the terms and the lingo that is used in the industry to talk about this problem. Last year we had Miria Luciana Accente, head of responsible AI at Pilbladisiuc PwC, come on the show. One of the topics we discussed was the distinction between the different terms such as responsible AI, explainable AI, interpretable AI, ethical AI. I think a small reasons why we have these new terms is probably driven by the comms department of many organizations working on the problem. But I'd love to understand from you how you view the differences and the overlaps between these different terms.

I find there's a lot outside of like industry, there's mostly. There's a lot of debate and confusion around the terms. First, let's separate like ethical AI is like more a mission to inject like ethical principles and human values into AI. So it's an ideal and yeah, it has a lot of parts. I'm not going to diminish like the contribution. But it's not necessarily driven by really clear objectives because everything is not only vague because there's no like perfect recipe for it, but it's also like there is a disconnect between people working on the ethical side and the people that are actively practicing in the field. And the rest of the terms that don't have ethics in it or fair in it are related to the imperfect application of the vision because there's many reasons to interpret machine learning models that don't have to do with ethics. Also for other reasons, it's just good business practice. So you have then explainable AI, which is another term that's used a lot, or XAI people like to use the X instead. And then interpretable machine learning or interpretable AI, and they're used interchangeably in the industry, in academia, they tend to use interpretable to refer to the white box models and explainable to refer to the black box models or vice versa. I'm of the opposite camp. I tend to think explainable is more like a confident term, is if you can explain something, the whole thing backwards and forwards. So I think it's best to actually start using it for the highly interpretable, intrinsically interpretable models, linear regression and so on, because you know how it was made, you know how everything was made, you can extract all the coefficients and so on. Whereas interpretation is something that it's perfectly okay to use with something that you can't understand completely. We do it all the time. The whole field of statistics is based on it. How many things are just shown through a Chart. Nobody claims to know every single thing about say the economy, but they can in average explain a trend. And so it's an interpretation, it's not an explanation in the sense that they're like 100% certain. Responsible AI is like a newer term. I think it means the same thing than explainable and interpretable AI mean, except with all the baggage. I think there's a lot of baggage with explainable AI and interpretable machine learning from where they came from and the debate as far as what they mean. So responsible kind of takes that away because you expect something to be responsible to also have some ethics in it. But it's not necessarily entirely ethic. Yeah, it doesn't have that semantic confusion that the terms explainable and interpretable because if I ask you to interpret something, you are explaining it. So that kind of creates this whole mess. I actually prefer responsible, but I don't know how much it's going to catch on. So that's what I think about those terms.

That's great. I appreciate that. Holistic definition. So let's start talking about the book. Let's first talk about the challenges in interpretable machine learning. One of the chapters of the book outlines these challenges. Do you mind walking us through these challenges and how exactly do they affect model interpretability?

The three concepts I think you're referring to are fairness, accountability and transparency. Fairness is what connects to things like justice, equity, inclusion, in other words, that models don't necessarily have or are adding an indiscernible bias or discrimination. Accountability, on the other hand, is what connects to things like robustness, consistency, prominence, traceability, privacy. In other words, making sure that models can be relied on over time and that can make someone or something responsible should it fail. And then transparency is what connects explainability, interpretability. It's just like the baseline properties. In other words, understand how decisions were made and how the model is connecting the inputs to the outputs. In other words, I tend to see it as a pyramid. Transparency is at a base because you can't have the other two if you don't have transparency. In the book, naturally I focus more on this level. However, a few chapters focus on fairness and accountability. I always speak of these being like where the focus should be. Mostly transparency is not by any means a solved problem, but the other two are more complex. So we're like tackling the low hanging fruit. And that's sometimes very convenient, but it's not necessarily always the best choice because it doesn't matter if you can explain the model if it's not Robust.

And if it's not fair, so okay, that's awesome. And so let's talk about that explainability and that interpretability and machine learning. Do you mind walking us through the main challenges practitioners face and how exactly do they affect model interpretability? When it comes to interpretable machine learning.

Model interpretability is impacted by three things that are present both in the data and the model. There are non linearity, non monotonicity and interaction effects. These elements add complexity and as I said, they're present everywhere. The most effective models match the nature of the data. So the nature of the data and the models are very highly connected. So if the data is linear, it makes sense that you use a linear model. That's why linear regression has all these assumptions baked in. And if the shoe fits also with neural networks, why not use a neural network? That's why I think neural networks forte is unstructured data. Because unstructured has the properties that you would expect. It has the complexity, the non linearity and so on. One of the trickiest is the interaction effects because feature independence is an unrealistic assumption. In other words, more likely not multicollinearity is present in the data. It's really hard to interpret models where many features are acting simultaneously to yield an outcome in a counterintuitive and contradictory ways. It's not as simple as satirist paribus, I.e. all things remain equal, especially when you have large amounts of data and many features. So those are the challenges in machine learning. It's often discussed in statistics, but statistics, like when a lot of things these things were discussed. There wasn't this idea that data would explode the way it has in volume and velocity and all the different ways that we come to call big data now.

Okay, that is awesome. And I'm very excited to discuss and outline with you the solution to these challenges. But first let's talk about another big challenge that is really important within the realm of interpretable machine learning and that is bias. Data is probably one of the most important assets in generating robust, interpretable and responsible machine learning systems. And bias is a big problem when it comes to data. So can you outline where does bias in data comes from?

Bias can come from two sources and they trickle down. They can either come from the data generation process or from the tooth, which is where the data generation process connects to the data itself. The many bias that infect the data generation process are sampling bias, convergence bias, participation bias, measurement bias. For instance, an old school example is if you do a survey by the phone to See what percentage of the population approves a president. The problem is that your data is not likely to be representative of the population because not everybody is reachable by phone and not everybody would answer a known phone number. Not to mention those that respond and are willing to accept the survey are certain kinds of people that may not reflect the general viewpoint of the general population. So that's going to be a problem. That's just an example of the kind of bias that affects the data generation process. There are also data entry errors that you may have. As I said, they're random if you're lucky, and those are presence all over the place. As for the truth, the data generation process may be unbiased, but it captures what I call ugly truth. So instances of real discriminatory behavior. There are also fake truths, which is what? Someone does something deceitful to trick our IT systems. And this happens all the time. But a lot of times it goes under the radar. Maybe it's not done to sabotage a system or you know, it's just done to get some kind of benefit that the system provides. And then there are the changing truths. And this is which is how our data generation process captures data at one point in time, but even in a short time afterwards, reality might be different. Our data generation process might be biased in that sense.

Okay, that's really great. And on that last point around ugly truths, given that these ugly truths are unfortunately rooted in bias we humans produce in the data generating process, will we be able to solve for this type of bias and machine learning through technical solutions?

Machine learning is all about predicting the future. But there's this quote by the founder of Atari and it's very clever when it says the best way to predict the future is to change it. And there's absolutely no shame in that. If you actually bias your models to counteract the bias present in the data, you'll eventually improve the future. So that's something that's part of the loop. People don't realize that they can do. A lot of models have that property, have that ability. Say you have a pricing model for, you know, real estate. And you could realize, what if I make that pricing model so that it actually counteracts the bias people have already with prices so they don't go out on a frenzy and buy all the available homes. And at the worst possible time, you can do all kinds of things to do that. Another example is if you have a biased data set that is about, say, criminal recidivism. And it's about detecting who could possibly Go back into jail. You could go commit a crime again after being in jail. You could argue we shouldn't have models do that, we should have human judges do that. But then you have to see, okay, well how good are human judges in that we can do a model that actually improves the false positive rate over human. Why not test it? Why not see how we can actually counteract the biases maybe in ways in which we benefit society. Say we see, okay, well women are more likely not to recidivate and they have children. And so by having them in jail we're actually going to perpetuate some kind of cycle. So why don't we bias the model in such a way that it's beneficial rather than perpetuating the bias to begin with? So these are questions that I don't intend to solve. I think it's something that sociologists should get together with economists and the people working on these problems from a technical point of view. But the mindset shouldn't be let's predict the future. The mindset should be let's change the future for the better.

I really like that perspective, especially since you're taking the data generating process and using it against itself, the bias data generating process and this form of ugly truth and you're able to, as you said, engineer and change the future for the better. Harping on the book as well. You know, one thing I love about the book is how grand it is and many examples of interpretability in real life machine learning use cases. Circling back to the earlier part of our conversation where we discussed the why behind interpretable machine learning, I think one thing that's also super important to take into account is the degree of risk associated with a model is highly dependent on the use case industry affected population by the model. For example, a credit risk modeling that is determining loan outcomes has a much larger playground for potential harm than a customer churn model which is driving retention strategies for a SaaS organization, how do you evaluate the risk of a machine learning use case? How would you describe the spectrum of risk here for any given machine learning application?

Well, I'd say that risk is determined by several factors. First of all, you want to know to what extent your algorithm decision impacts stakeholders. And what are the stakeholders you want to quote, unquote protect their interest from? They're all weighted differently. You know, like one thing is like the bank trying to protect its officials or protect the employees or protect the customers. To what degree are you willing to have risk on every level because the goals aren't always aligned. So you might think, okay, well, the short term risk is with the back, which is profits. But the long term risk, it will alienate the customers and those customers will go elsewhere. So the idea is not to be shortsighted in the assessment of these risks. The best way to understand that is understand it as a system where stakeholders are not in isolation. There is not one magical metric. You might think, okay, well, I want profits to be my magical metric, profits for the next quarter, for instance. But a lot of these things measured on different timescales and with different actors all interacting with each other. So once you start to see them as a system, like, you could develop like, for instance, a causal model and try to understand how pushing one lever in one way will impact the other. But this takes an idea of experimentation and not just inference or prediction. So it's like taking it to a next level. My book actually has an example in that sense of how you use cate, or conditional average treatment effects, to measure what is the least risky option for the bank and for the customer in a circumstance like that. Not all problems are like that, though. There are AI systems whose risks are elsewhere, whose risks are in misclassifications of a specific kind. And so they might have instances of bias present in their model that they're not even aware about, or they're aware about, but they brush off and they think, oh, it's not a big deal, but the level of reputation damage that comes from something like that can sink the system down completely and nobody will want to use it ever. So shouldn't you de risk that? And so that's something you should realize when you output this, at the very least to put out disclaimers, it has a weakness with this kind of input or was only meant to be used on these circumstances. And people should heed these kind of disclaimers because maybe an official from the company will say, oh, why don't we take this and repurpose this for that? And then you get the sort of disaster that you got with Zillow. It was never meant to be used in that way. Then there's also risks that you get for how robust it is towards adversarial inputs. That's something that a lot of as AI is being adopted in a lot of different ways, we fail to realize how it can be gained. I connected with my early experience with the Internet, how naive we were with the Internet. Early on, we didn't realize how our data could be exploited or how our credit card information could be stolen. And so little by little, like everything became more and more robust, like first with ssl and then until, I think it was like eight years ago, nine years ago that Google came along and said, okay, well, no more websites with SSL will be featured or will be ranked by our algorithm. So now SSL is a must. But in that journey there was a whole level of awareness that became towards, okay, this is a technology and that can be used for dangerous means. So we have to protect it to be useful for the rest of us that are not using it like that. And I think people will become aware of it. I just hope the outcome isn't painful for anybody in the sense that, okay, all of a sudden, if everybody, not just celebrities, are getting deep fakes generated to game all kinds of systems, who's going to use AI systems from then on? So I think a way to improve this is to gauge monetary value in other ways. Stop the single metric mentality that we have not only in machine learning, but in business, which is, okay, we have this metric and this is why we have to chase. And all metrics are tied to this one. I think it's something that would solve a lot of things. If we started to factor environmental problems, social problems, other stakeholders of all kinds into our general equation of the well being of a company or a society, we'd be better off. And so, yeah, it's going to take a very careful discussion of how to weigh everything, what is more important. And it's a very uncomfortable discussion, but I think it has to be done sooner or later.

I love the holistic perspective here. And before we talk about solutions, do you want to give maybe an explanation of the Zillow example? Because I think it's highly illustrative of the problems that you're discussing. For those who may not be aware.

Of it, Zillow had come up with a pricing model. Hopefully I remember this correctly because it happened many months ago. It had this pricing model that was just for the benefit of users. So users would come and would say, okay, estimated value of this home is this much. This, of course had an effect on the users because maybe it was overpriced, maybe it was underpriced, but they seem to think it was accurate. But this had no ramifications other than inflating or deflating a market. But then they figured, why don't we get into the game of buying and selling homes? We have this metric, we can use this metric, this number we can use to actually get a benefit, because we know what it's estimated at and we can negotiate a price based on that and so on. So they sooner or later realized that their estimates were inflated and they probably it was beneficial for them to be inflated because who doesn't want to go and see their home and see, oh, it's actually, I don't know, $50,000 more than I thought it was. And it's good to their stakeholders, their customers, the people that come and see that. But to them it was terrible because they ended up losing a lot of money that way. So they couldn't make back. They couldn't flip the homes for what they thought they could flip them because they in a way had contributed to inflating the market. And that was very painful. They had to lay off a lot of people if they used the model in the wrong way. They had never designed it to do that. And quants do this in finance all along. They don't necessarily use machine learning model for that. For doing pricing optimization. They use all kinds of optimization methods that are purely math based. In the best case, the closest thing to machine learning you'll see there is reinforcement learning. And yeah, I think it was just a bad idea.

Yeah, it's a very fascinating example. Given that, let's discuss the solutions to many of the challenges that we discussed. The book does an awesome job at breaking down a lot of different techniques available for practitioners looking to drive better interpretability. Can you walk us through these techniques and the crux of these different techniques as well?

When people talk about interpreting models like everybody's interpreted a model at a very, very, very basic level and that's evaluating performance. So especially once you break down the performance by cohorts, segments, everything, you're already engaging in error analysis. And error analysis is an interpretation technique. But beyond those traditional interpretation methods, there's also feature importance. And that's probably the first one people will learn and say, oh, I didn't know this existed. And it allows you to quantify and rank how much each feature impacts the model. Depending on what you're dealing with, it may vary. It might be for a tabular data set, it's every column, even every cell. You can break it down. But for NLP model, it depends how you tokenized it. It might be every word, it might be every character. It really depends how you tokenized it. And then for an image, it's every pixel. That's how it's usually used. Then feature summary, on the other hand, examines individual features and their relationship with outcome. They include methods such as partial dependence, plot, accumulative local effects, and ideas to figure out well A feature might be important to the model, but how is it important? What is driving things? You start to see the relationship of that feature with the model. You might realize for cardiovascular disease, as age increases, the risk increases. You're already seeing in those terms in which it's not just that the feature is important at certain values, it has more impact or it has a negative impact or a positive impact, or it actually goes up and down. And that's when we start to see, well, it's non monotonic because it's not going in one direction. Then feature interactions methods quantify and visualize how the combination of two features impact the outcome. Usually we go bivariate, but you could even go into three levels if you wanted to, or four, but that was just crazy. But yeah, you see how they work together. Sometimes you'll realize that a feature by itself has no impact on the model. But I tend to talk about it if it were like a basketball game. There's players that they're never shooting the ball into the nest, but they're assisting another player to do that. And that's how it is in a machine learning model. Often, well, it depends on the kind of model, but they tend to work together. Well. There's also another kind I didn't mention. Those kinds I mentioned are all like global interpretation methods. But actually the vast majority of interpretation methods that exist, especially thanks to deep learning are local interpretation methods. These are ones in which you're trying to understand a single prediction. You're not trying to understand the model as a whole, but you're trying to understand a single prediction. Sometimes you can even take interpretation methods and apply them to single predictions and put them into a group and to say, okay, well all these predictions that were misclassifications tended to be misclassifications because of this. So you can also do things with a bunch of single predictions on that level as well, which can be very useful.

That's really awesome. And I love also the holistic nature of how you approach these different techniques. One of the techniques that we've heard a lot about over the past couple of years in the data science space, and that's covered quite a lot in the book as well, is Shap values, which are short for Shapley additive explanations. Can you dive more into more detail about what Shap values are and how and why they're useful?

To understand Shap you have to understand first Shapley values. Shapley was a mathematician and Shapley values is a method derived from coalitional game theory and generally I explain this as a basketball analogy. So I tell people, imagine you're blindfolded at a basketball game and a loudspeaker just announces when a player exits and or enters the game by their number. They don't say their name, just by their number. And you don't know if that player is any good, say, like Michael Jordan or bad, you know, like, say, I don't know. I imagine Mr. Bean is a bad player. Good players and bad players just hop in and hop out at any time. And the only way to tell if they're any good, if their presence made a difference to the score. So with this, you can get an idea what players are contributing the most, both positively and negatively. For instance, you notice when 23 is playing, score increases a lot, no matter who else is playing. So you get this idea, oh, it must be really good. But imagine you're blindfolding and you're taking notes as well. So you start to, okay, quantify that. And so the difference is core becomes a marginal contribution. And this starts to make a lot of sense. Once you quantified all the different marginal contributions and once you run through all the possible permutations of players in the court, imagine it was like an endless game. You can calculate all the average marginal contributions of each player. And this is called the Shapley value for model. The features are the players. Different subsets of features are called coalitions. And you can calculate all the average marginal contributions for each player. Differences in predictive error are actually the marginal contributions. And you're blindfolded, of course, because it's a black box model. That's why you're blindfolded, of course. Problem is, computing Shapley values is very time consuming, as you can imagine, all the different permutations of all the different features, unless you're talking about two or three features, it's just like enormous. Shap combines other methods in many cases to approximate Shapley values. If not, it just takes a sampling. It doesn't do the entire permutations. So it still adheres to a certain degree to all the mathematical principles which exist within Shapley. And often the thing is that SHAP is the closest we have to a principled way of calculating feature importance. And this is important because it has advantages because of all the mathematical properties, you know, it has symmetry. There's a whole bunch of dummy principle. It has a whole bunch of principles. And since there are Shap values for each feature and observation, another advantage is it can become a local interpretation method as well, which is rare. Because usually you're just one or another.

I love that explanation. Using the basketball analogy, I think it's very useful to use it as a mental model for how to use this technique. Now, given the proliferation of these interpretability technique, what is the mindset that a data scientist needs to adopt when trying to understand the results coming from interpretability techniques on machine learning models? This is whether using SHAP or other techniques, as in what level of certainty do these techniques provide? And do data scientists need to inoculate themselves from a false sense of certainty when interpreting the results of interpretability techniques?

Well, certainty with machine learning can never be too high. I think that's a fool's errand in any case. And you have to start seeing models the same way you see another fellow human. If you did something and I asked you why do you do that? And you told me, because of this and this, I can't be 100% certain. I just saw inputs and outputs and then you gave me an explanation. But I don't know if to believe you or not. That's called post hoc interpretability and it can't be 100% certain. But what you can do is you can use different methods and then get a good idea. If several people saw you do something and then they asked you, your mother asked you, your sister asked you, I asked you, everybody around asks you, you tell them all different stories, but they have some commonality. You can take that commonality with higher certainty. And that's what I expect people to do with these models. Don't rely on a single model and call it the truth. Single interpretation method and call it the truth. Rely on several. That's always a good practice. And another thing is a lot of them are stochastic in nature, so they might give you slightly different results every time you run them. That's certainly the case for lime, which is another very popular method. So why not average them out, average back the values or take the median? You'll get something closer to the truth in that sense. So it'll be a lot easier to interpret the more you run it. There's also other cases in which they have number of steps. Integrated gradient has number of steps. So if you increase the number of steps, of course it's going to take longer, but it's going to be a better explanation. Our number of kernel shap, which is one of the truly model agnostic ways of doing chap, has a parameter called number of samples. And if you increase that number of samples, you'll definitely get a better something. You can trust more. There are ways of approximating better certainty within each method, but also something I recommend is not relying on single method. As I said, it's an interpretation. It doesn't have to be perfect, but it's better than nothing. It's better than blindly trusting the model and just never examining what it's doing.

Definitely. And given the results of some of these interpretation models, we've discussed how to interpret them, we've discussed the mindset that we need to be able to adopt here. What are some of the common, common diagnoses you can make about a machine learning model that has suspect interpretability results? And how can practitioners remedy these diagnoses?

You mentioned bias, and that's a big one. Fortunately, there are many bias mitigation methods that remedy this on three levels. So you can remedy bias in the data. You can do it with the model or with the predictions themselves. Another one is complexity. Models can get too complex and that can lead to poor generalization. So regularization is what I prescribe on the model side, but also pre processing steps like feature selection engineering can address it on the data side. And if you're dealing with something like for instance, images, there's all kinds of preprocessing steps you can do with images. Unless there's a reason, for instance, to have a background. You can remove the background of an image, you can also sharpen it. You can do whatever needs to be done. But realize that whatever is done on the pre processing side during training has to be done on the pre processing side during inference. So we also in NLP do a lot of pre processing. Sometimes you realize a form of feature engineering selection is actually stemming or lemmatization. You know, like taking away pieces of the word to make it simpler for the model. You can certainly do that. That will make the model more generalizable because there might be words that seem like another word and they pretty much mean very similar things. And it's just by taking only the stem, you're making it more clear and less complex. So there's also robustness. And for that we can augment the training set data or address the model itself with robust training methods or even in the predictions. There's many methods that can be used for that and for model consistency. As long as you monitor data drift and retrain frequently, you can tackle it. But it's also good practice to train. Using a time based cross validation with time is an important element for the typical cats and dogs scenario. It's not like we expect cats and dogs to evolve. That much during their lifetime. There might be slightly new breeds, but it might not make a difference if the images are like two years old. But for a lot of cases, like I find in my my own work with agriculture, like every season is different, you know, like different weather, different everything. It is good to update the models a lot and use cross validation to make sure that doesn't matter what year you use, the model is going to be robust and consistent.

That's really awesome. And especially on that last point, I think that best example of consistency has been through Covid, right. FMCG models that are highly time series based when it comes to for example, predicting stocks of items like for example, toilet paper. These have a completely different dimension pre pandemic than post pandemic.

Yeah, for sure.

Awesome. So as the industry evolves and machine learning is further adopted and the need for interpretability goes higher, what do you think the future of interpretability will look like?

I think there'll be definitely a change in mindset. I think it will come naturally, organically, if you will. But the problem right now is that model complexity is seen the culprit of all ills of machine learning. But it isn't always. I think after all, the things we try to solve with machine learning are complex or they should be. I mean, I don't think maybe there's some novices out there taking like a very simple tabular data set and throwing deep learning on it, but I don't think it's like a big issue. But I do wonder that we're taking like the brute force approach too far. There's been an arms race with model complexity. It's not my area, but I have to wonder if we need to leverage trillion parameter model language models for natural language processing tasks. There has to be a simpler, less brute force way of achieving the same goals. After all, humans only have 86 billion neurons and we only use a fraction of them for language at any given time. I have to wonder what is going to come of this arms race and if it's going in the right direction. That of course is going to change interpretability. Whatever direction it goes at. If it becomes more brute force in everything, it might hit a limit in which interpreting it is going to become impossible, at least through traditional means. Or we'll have to use more approximation based things that will give us a less reliable interpretation. There's the other big issue we've discussed throughout the session is bias. Generally the idea is to train machine learning models that that really reflect the reality in the ground. So the Idea is, okay, let's go a bit back to basics, think about what it's like to have a data centric approach. Forget about the model, let's go back to the data. Understand how to not only improve the quality in order to achieve better predictive performance, but as I said, actually improve outcomes, change kind of the flow of things. Because a lot of the things is we don't realize what the technology is coming out of it. It's like when social media came out, everybody, oh, this is wonderful. We can finally communicate in these very rich ways, no matter where you are in the world. But we didn't realize what can happen from that and how once the genie is out of the box, how do we put it back in or how do we improve things? The same things can happen with AI and I think a data centric approach forces us to do that. And then there's of course things that can improve. With model interpretability, there's more and more methods coming out, methods that I find super promising and will take a few years for them to reach the pipeline that are in production. But I think something that will make a transformative change in the field is the nuts and bolts of machine learning are right now data cleaning, data engineering training pipelines, the drudgery of writing all the code to orchestrate training and inference. In coming years, new and better no code and low code machine learning solutions will displace these quote unquote artisanal machine learning approaches. And I believe the best ones will make interpretability prominent. Because once creating a sophisticated machine learning pipeline is less than one day's work in a drag and drop interface, we can devote the rest of our time to actually interpret the models and improve them. So there'll be more into more iterations in that sense. Right now the iterations are like, oh, let's achieve predictive performance, but that's a better predictive performance. But once you have no code system maximize predictive performance for you automatically, then what will a data scientist do? And I think the best thing we can actually do is interpret models and we'll learn how to extract the best value out of the models and improve them in other ways. We're not improving them right now through that. So how to achieve better fairness, how to make them more consistent, how to make them more robust, how to package them in such a way in which they have all these properties that outline their properties, their weaknesses, their strengths, data provenance, a whole bunch of things.

That's really awesome. And going beyond the future of technical interpretability, you Mentioned in the book that there are legal, social and technical standards and procedures that need to emerge to really realize the full potential of interpretable machine learning. What do you think needs to change in terms of regulation and how we organize ourselves as a species and society to realize progress on that front?

Yeah, I think there's a lot of things I would propose. I think there's a lot of people thinking about these solutions. I think for one, certification is a must. I think before deployment. You can certify models for things like adversarial robustness. There's only very few methods and standards for that right now, but they'll evolve for fairness. That's another one. And you can also. Right now there's so many different metrics for fairness, but you could at least specify which one you use. Then even a level of uncertainty. There's a method called sensitivity analysis, which I discussed in the book, which can tell you what level of certainty you can have with the outcome. Not so much the interpretation method. It's about the outcome. Then the model card. That's another good one that wouldn't already exist. I mean, a lot of people criticized it because think, okay, well that's so simple. But it's an important step. So if you deploy with. Along with the model, you deploy a card which tells stakeholder important properties about the model, such where the data came from, potential weaknesses and what intended uses you prescribe for it, et cetera. Like whenever we have a product out there, it just makes sense to have like in the label, all the ingredients and everything. It's. I see it like that. Also, I advocate for abstention. If a high risk model, you have a lot of low confidence predictions. Like for instance, you might have if you're telling a bank customer, okay, your loan has been denied because you're deemed 51% risky. No way. That's stupid. Just leave it up to a human. I think that's ridiculous. There should be a band in which you say, okay, well, this is too close to call. Have a human do this and then you can improve the model. The people can look through this and say, okay, this is why the model thinks it's like that. Because it has this, you know, this customer has slightly less collateral than we would expect. Or because, but I would give it to this person because they actually have this. But that's not in our data. So maybe there's ways we can improve the model through actually catching this and sending to a human. Then we have monitoring. I think it should include more than predictive performance. We could in addition for checking for data drift and continuum, model robustness, feature importance or fairness metrics. There's just so many things we could model, we could monitor with the. For the model. Another thing I advocate is the manifest. Have something that the model has that you save, that can let auditors trace model decisions much like a black box does in a plane. I think that's super important and that nobody can tamper it. If there's one use case for blockchain, it's definitely that. No, there's a ton of use cases for blockchain, but I think that's definitely one more and then exploration. I think one of the things that should be in the manifest, if the model could auto destruct, I definitely think it should. Models should have a strict shelf life, like milk gone bad. It should be toasted as soon as it meets the date, no questions asked. And then every training procedure, if it's in place, you'd aim to replace the model just before that date. I think that's very important. That's what I would suggest.

Yeah. A small list. Definitely a small list. But I really appreciate this really holistic perspective. I think really it's one of the few interviews I've had where someone lays out such a holistic perspective on the future of machine learning. So finally, Serge, as we close out, do you have any final words before we wrap up today's episode?

Nothing in particular. I think people should be excited about where we are in this field at this time. I think it's a good time to be joining or even as a user seeing what's coming out. As I say, I equate it to the mid level maturity that the Internet had. Say in the early aughts, like 2001, 2002, things were starting to work well all of a sudden the Internet wasn't the crap that it was before that people were excited about it, but they were like, these websites are so ugly and this one never works and this browser shows me this and not this other thing. And so I think that's where we at. And still. But on the development side, we're still going through growing pains. People are still writing their code by hand, as I said, in every single line. And they think, okay, well that fit all the stuff we do. It just seems like it's going to seem very antiquated very soon. And I'm excited for that as well. But one of the good things of being in this space at this time is that you get things at the ground floor, you learn how to see things without all the abstraction. You'll see them in a few years, because once it's all drag and drop, it's going to open the floodgates for less technical people, and that's a good thing. But at the same time, it's not going to allow people to see the guts of things the way we see them right now.

That's awesome. Thank you so much, Serge, for coming on Data Frame and sharing your insights. Thank you. You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment, and share episodes you love. That helps us keep delivering insights into all things Data. Thanks for listening. Until next time.