#99 Post-Deployment Data Science

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization, join us for in depth discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Hello everyone, this is Adele, Data science educator and evangelist at datacamp. Last week on the show we had Serge Macis on the podcast to talk about interpretable machine learning. Throughout that episode we talked about the risks that may plague machine learning models in production, and as these risks grow, what are the tools at the disposal of practitioners to iterate and understand model performance post deployment? This is why I'm so excited to have Hakim Al Ahras on today's episode. Hakeem is the co founder and CEO of Naniml, an open source Python library that allows data scientists to estimate post deployment model performance, detect data drift, and link data drift alerts back to model performance changes. Throughout the episode we spoke about the challenges in post deployment data science, how models can fail in production, some cautionary tales to avoid, why NANIML is open source, the future of AI, and much, much more. If you enjoyed this episode, make sure to rate and comment, but only if you enjoyed it. Now onto today's episode.

Hakeem, it's great to have you on the show.

Yeah, thanks a lot for having me.

I really appreciate it.

Adel, I'm excited to speak with you about post deployment data science, your work, leading Nanny ML, and so much more. But before, can you give us a bit of a background about yourself and what got you here?

Yeah, sure.

So I'm originally American, born and raised in New Jersey. My educational background is in biology, so actually quite far from data science. Originally I was mostly working on evolutionary biology and population genetics and at the last year of my bachelor's I did A course in bioinformatics. And that was kind of an R.

And using like a bunch of machine.

Learning techniques for population, genetic purposes and also genetics, just genetics in general and stuff like that. And that, like, really hooked me to.

The concept of what you can do.

Using programming and machine learning. And originally I actually wanted to be a doctor. And my idea was like, using personalized medicine.

That was what I was like, super passionate about.

So I was like, oh, man. Using machine learning and genetics, you can.

Get to a point where you can really, on a personal level, prescribe certain treatments and like, really help people, like at scale, but personalized.

And that was something I was super passionate about. And so then I was like, okay, before I go to medical school, I want to go do a master's of bioinformatics. And I ended up moving to Belgium to do the master's at the KU Leuven, which is. It's funny because I know Data Camp.

Is headquartered in Leuven, so it was. It's nice, it's a nice city.

And basically my motivation behind that was pretty funny. I wanted the highest, cheapest master's program.

That existed for bioinformatics, and you end up with the K Leuven.

It's like ranked 29 in the world and it was €600 a year. But then I really got hooked onto the data science machine learning side and I abandoned my dreams of becoming a.

Doctor and decided to just go full.

Force into kind of machine learning. And then I worked a bit as a data engineer, a data scientist. I started a machine learning consultancy also with my co founders from naniml. And then eventually we just saw that every time we put models into production, we always got this question about, okay, what happens next? How can we trust the models? How do we know that they're performing well? And at the time, there was a lot of smart teams working on what we call the MLOPS part of it. So the infrastructure behind it, how do we actually deploy a model, like the Serving and things like that? And so we decided like, okay, given our expertise, which is data science and the algorithms, not really the actual programming.

And being good software engineers, that's how.

We decided to work on Nanyamla.

And we also obviously thought it was super important.

So I am very excited to discuss with you your work leading naniml. But before, I'd love to anchor today's discussion and some of the problems that you're trying to solve. So, you know, over the past year, we've had quite a few folks on the podcast discuss the importance of mlops and the different challenges associated with deploying machine learning models at scale. However, another key aspect of MLOPS is post deployment data science work, as in monitoring, evaluating, testing, machine learning models and production. I would also add improving and understanding business impact here. I'd love to start off in your own words. Walk us through the main challenges in post deployment data science.

Yeah, sure. And maybe just a little side note on why we like using the word.

Post deployment data science instead of just machine learning monitoring. Like, I don't want to come off as random people who are just trying to create something new, but the way we see it is that like, monitoring.

Is a very passive activity and if you describe nanoml what it does today, it's indeed a monitoring library.

But we see the work so much.

More than just monitoring.

Like, yes, the first step of post.

Deployment data science is actually monitoring and knowing how your models are performing. But then we see that there's a.

Whole host of other things that you have to do. That's why we like to use the.

Term post deployment data science, because we.

Feel like it is actually real data.

Science and will be the responsibility of someone with more data science skills. But maybe to go into what are the challenges of post deployment data science, I think the first and foremost is knowing the model performance in the first place.

It's not so trivial whether you have.

Ground truth or not. So whether you know after the prediction what actually happens in the real world, which is ground truth, or you don't have it, knowing your model performance is still pretty challenging from an engineering perspective and from a data science perspective. And so just having that kind of.

Visibility is already pretty hard.

And maybe to give an example of, okay, when you have ground truth, it's more of an engineering problem. So when can we get our ground.

Truth and compare it to what our model actually predicted?

But when you don't have ground truth, it's much more of a data science and algorithmic problem. So for instance, credit scoring, where you have a machine learning model that decides if someone should get a loan or not, and the model predicts yes or no, this person should get a loan, and then when do you know when that prediction was correct or not correct?

Either the person has to pay back the loan or they don't pay back the loan, but either way it's very far in the future, so you cannot.

Calculate the performance of your model in the traditional sense. And then the second challenge I would say is models fail silently. That's, you know, one of our taglines. And basically the problem with machine learning.

Models is that if you give it.

Data in the right format, it will make a prediction.

Whether that prediction is right or not. That's not the model's problem. And so, yeah, software for the most.

Part can fail loudly. So like you have a bug, an.

Error, it doesn't run, so you know what, it's not working.

But with machine learning models, you actually don't know when it's not working. And so that silent failure is really problematic. And then a third challenge I would say is that most data drift is virtual. So we didn't get into the nitty.

Gritty technical details yet.

But basically data drift is when you have a change in distribution of your input variables, variables to your models, and most of it is virtual. And what does that mean is that it doesn't actually impact the performance of your models. You have data drifting all the time, but your model can actually handle it. So if you were just detecting when the data changes, you're going to get a lot of unactionable noise and you won't actually be able to do anything with that information. And then I would say finally, which is the most complicated one, and I.

Would say probably a lot of data.

Scientists still don't have that much experience.

With it is the feedback loops.

So you have the relationship between the technical metrics and your models. They might change, but in general, just having a lot of machine learning use cases where you're taking like predictions on a customer base, for instance, and you have a model that takes prediction on a customer and then the a different department does something like imagine a churn model that decides who will cancel a subscription or not, and you predict someone will cancel their subscription and then the retention department sends them a discount. And then the next month, or whenever your model's running again, you make a.

Prediction on that same customer again. So then the model is actually impacting.

The business, and the business is impacting the model.

And you have these like very interesting.

Feedback loops where the model performance will definitely change. And then you can have things where, like when you are building your model, you run a bunch of experiments and that you have a business metric of keeping churn below 5%. And to achieve that, you needed a rock AUC, for instance, of 0.7. But then over time, actually the model performance has to go up, maybe to.

Rock AUC to achieve the same business results.

Because maybe in the beginning you were.

Able to easily detect the people who churn and you took the first steps to stop them from churning. And then the people that are later going to churn, they become harder or other weird things can start to happen.

So there's definitely lots of challenges still once a model is put into production.

I love this holistic list. And I definitely agree with you that this is definitely a problem that is foundational to the industry if we're going to be able to really derive value from data science at scale. You mentioned here the models fail silently component, Right. Can you walk me through the different ways machine learning models fail silently and why?

Yeah, sure.

So, yeah, I already mentioned that most models fail silently. I would say there's two main ways. Data drift induced failure and concept drift induced failure. So basically, data drift induced failure is when the input data to the model has changed to the point where the model has not seen enough data in the new distribution to make good predictions. So for example, the average age of your customers was 25 and now it's 50.

So the age, the individual feature of age, has shifted in distribution.

And maybe your model hasn't seen enough.

50 year old customers to be able.

To take good decisions there, but again.

It will just keep predicting on them.

As if nothing happened. And so that's the silent part. The second part to it is the concept drift induced failure. And this is a change between the relationship of the input variables and the output. Machine learning model is basically just trying to find a function that maps inputs to outputs.

That's basically what you're trying to approximate it as best as possible to the real mapping function that exists somewhere in the ethereal space of reality. So that can be caused by the.

Actual behavior of the underlying system changed most often by a variable that's not included within your model. So for instance, like your actual customer behavior has changed. So maybe now your 25 year old customers are buying cheaper products because the economic conditions have changed and you don't capture economic conditions in your model. And sometimes concept drift induces data drift. So for instance, in this case you.

Would see the average price of the.

Products your customers are buying decrease in time. But sometimes concept drift can also be.

Silent where it actually doesn't impact any of the data in your model, but.

Something did change in the real world that you're not capturing. And so the fundamental behavior of the system is different, and then the performance from that can suffer. And both of these can have either.

Catastrophic failure or gradual degradation of performance.

I love that distinction that you make between concept drift and data drift and harping on the catastrophic failure or gradual degradation of performance. You know, a lot of the problems that you discuss here have those Consequences that can range from harmful to say the least, right? To, as you said, catastrophic for an organization using machine learning and AI. Can you walk us through this range of consequences for badly monitored machine learning and AI systems?

I mean, I guess the first thing can be nothing, depending on how impactful.

Your use case is. That's the funny thing in this kind.

Of space is that your model is only as valuable as the underlying business problem at the end of the day, right? So if it's a model handling some fringe cases or something in your company that doesn't generate a lot of value, then if the performance changes, maybe nobody will care, right? Or maybe depending on your processes in your company, the model doesn't actually do anything by itself. Like maybe it outputs data frames and then they're imported into an Excel and then that's shared with business and then.

Business looks at the results and then there's actually no automated process in there.

So it could be that it's less important to monitor. And I would say that's the lowest thing. Then you have like gradual degradation. So, yeah, maybe just one thing, like monitoring becomes essential, like really essential when.

It'S when your model's in like mostly automated systems, right. It's important it before that because then.

You want to know the business decisions.

You take on top of it, if they're okay or not.

But if it's like in a churn.

System where if a model predicts someone.

Is churning and then an automatic email.

Campaign goes out to give them a discount or whatever, however your company wants to manage churn, that's what monitoring is really, really important.

And so casual degradation is just over time the data is drifting a bit and your model just becomes less and less performant.

That can be a little bit like.

The feedback loop thing that I mentioned before, where over time you're already identifying.

The people who are more likely to.

Churn and you're getting them out or like you're stopping them from churning. And so then the people over time just become harder to detect. Right? And things like that, and your model just gets worse and worse. But it's not anything catastrophic and maybe it causes a 10% loss in performance. And again, this all depends on the underlying use case, right? In some use cases, 10% loss might be like whatever.

In other use cases, 10% loss might Be like 50 million.

It's really depending on whatever business use case you're working on. And then you have catastrophic failure, which.

Is a lot worse. We've seen a few of those in the data science space.

I always point out to Zillow where they basically systematically overpriced 7,000 houses by 300 million and then it just collapsed and actually their market cap dropped by $30 billion.

So that's. Yeah, yeah. And they fired the entire, like they shut down the division that was buying and selling houses and that's fire everybody. So like catastrophic failure.

And also maybe a non financial catastrophic failure could be like the chatbot day from Microsoft.

I don't know if you remember that.

The, the one that was spewing out a lot of harmful content, that was trained on Reddit content.

Yeah, yeah, yeah.

It got racist and terrible real fast.

So these are kind of the catastrophic.

Failure where maybe you're introducing some systematic risk that you don't realize and then all at once it just collapses and.

Lots of bad things happen.

And then finally I would say there's discrimination and bias. And the main impact from that is like, yeah, it's not moral, can be pretty bad pr, it's just not good. And it can be that when you.

Built your model that you didn't see.

Bias in certain demographics because you didn't have enough of them in your data. And then maybe over time, more and more of a certain demographic enters your data and the model can't take good decisions on them. And that would obviously be also from a financial perspective, because if you can't take good decisions on a certain segment, you're obviously not doing the best for the company.

Company.

But it's also just not fair to.

The people that you're taking predictions on. Obviously that whatever it is that's impacting.

Them would be discriminatory and not doing the best it can do.

So I would say those are the.

Impact of what can happen also.

That's really great. And do you mind expanding maybe into that Zillow case study just for a bit? Because I think this is a very interesting case study for data scientists that are deploying machine learning in the wild, especially once you have data science becoming foundational to the company's business model. So do you want to expand maybe on how that failure happened as well as the underlying issues that led to that catastrophic failure?

So that's a good question. We could only speculate because we actually don't know. And also some people claim it's not a data science problem. So you can like postulate, like, how can that happen? So the thing that's interesting with, for instance, house price prediction, which is essentially what they were doing, if I understand correctly, they were like having a machine learning model that decided the price that they should buy a house. And so you essentially predict the price of that house. The problem with that is you don't have any ground truth because the price.

That the model predicts is the price you buy the house at.

So the prediction becomes reality.

So you don't know the quote unquote performance of the model in the real world. Probably when they were building the model.

They had a bunch of house prices and they tried to predict it and then they measured the performance like you would in whatever kind of data science system. But once you put it out in.

The real world, there is no real price because the real price is what.

The model makes the price. And so you can see how if you cannot really calculate that performance and you're introducing these little systematic errors over time that push the house price higher and higher for whatever reason. Yeah.

Eventually you can realize that you have a huge portfolio of very overpriced houses. That makes a huge problem for you.

That's incredible. And it's very interesting, especially when you mentioned here how the prediction becomes ground truth. There's this feedback loop as you mentioned, where the machine learning model becomes reality and companies can't escape that to a certain extent without really proper modeling. So in contrast, I feel like software engineering, it strikes me as an interesting aspect of data science that post deployment work. MLOps in general is still to this day not have been codified and not as yet matured around a set of best practices, tools and rituals taught in data science. Why do you think this is the case?

I would say the main thing is.

That it's still early days.

If you look at data science as.

A field, of course, it's all like.

I don't want to get any statisticians.

Mad, because if I say data science is only 15 years old, I will get a hoard of very angry statisticians that will be like, yeah, I was doing machine Learning in the 70s at a bank somewhere. Yes, we know, we know. But data science as a field in.

Non financial industry, let's just say that that's relatively new. And so I think that there's a big learning curve. And it also comes with the concept.

Of risk as well, by the way.

Because you have these financial institutions who have been doing this modeling for the past 70 years, I think maybe even longer.

And they developed all of these processes.

Around handling the risk that comes with taking decisions based on mathematical modeling.

They have entire risk departments validation, they have this whole process, the government even regulates it.

Right.

If you want to do a model for Credit scoring.

The government has benchmark models that they.

Compare your results on.

Like it is extremely regulated and well known how to handle that. And I think the problem is when you have, for instance, like a grocery store or a media company who doesn't have a risk department and they don't have any kind of inherent understanding of.

Risk, starting to take decisions based on models and like really can really impact their company strategy.

Right. Because if you think of like a churn model, if you see a lot of customers are churning, you might say, oh man, we have to change how we're doing our marketing.

Or you could really take big decisions.

Based on what is coming out of, out of your machine learning systems. But I think it's mostly chalked down to it being the early days and people not having in general a lot of experience with these kind of things. And, and to be honest, as much as us data scientists love to feel that data science is already everywhere, there's not that many models in production. Unfortunately, it's still very early. Like, you know what I like about being a monitoring company or NANIML in.

General is that it's a very good litmus test for whether a company actually has models in production. Because sometimes I feel it's like, yeah, we have models in production, we're doing data science and then, oh cool, so you can use NANyAML or another monitoring library and it's like, oh no, we're not ready for that yet. The models are not actually in production. It's just, it's kind of funny.

I think the latest survey I found is like 10 to 20% of models make it to production only that companies develop. So they definitely agree with you.

Yeah, so it's just the wild west.

And I think that's normal that you see in the early days, like this kind of whole mess of different practices and hundreds of tools and that over time as it matures, I think it will become much more clear on what the best practices are and how things should kind of be handled.

So I think this is a great segue to discuss how a NANIML aims to solve a lot of these challenges. So can you walk us through what NANIML is and how it works?

Yeah, sure. So naniml as of this day is an open source Python library. So data scientists can just pip install it and basically use it to detect silent model failure. Right now the way we see most data scientists using NANIML is like in a notebook. They run an analysis on their models in production, the data they have, but you can Also then of course, deploy it, it can be doctorized, it can be, you know, deployed however you want and have it monitoring in the traditional sense and near real time or batches.

Whatever you want to call it.

But NANIML in general, it has three main components. Performance estimating and monitoring, data drift detection, and intelligent alerting. And so basically with performance estimation, that's the whole reason we're doing naniml.

We spent a lot of time doing.

Research to try and find some sort.

Of methodology that would allow you to.

Estimate the performance of your models in the absence of ground truth.

So instead of having to wait a.

Year for your credit scores to know if someone defaulted or not or paid back the loan, that you can in the middle actually estimate what the performance that your model would have on the current data in production. And that was quite hard and lots.

Of research, but it works pretty well.

Now, I'm probably not the best person.

At NetTieml to go into the gory details of this. Like, I'm a data scientist, I'm not the research guy, but I can, I will try as best as possible to explain it. And then my data scientists will all yell at me and say that I'm stupid. No, just kidding.

But basically my understanding of it is that at least in classification models, your model, it outputs two things, a prediction and a confidence score, basically.

So you basically know if you have.

A class 0 or 1 and how confident your model would be about that score. Imagine you have a prediction one and you have a confidence score like 0.9. You can kind of say that basically the model is correct for 90% of the observations where it predicts one. And you can use some magic from there to basically reconstruct an estimated confusion matrix.

And from there it seems that it.

Captures all changes in performance that are.

Due to data drift, actually.

And from the estimated confusion matrix you can just get an estimated rock AUC or an estimated F1 or any machine learning metric, and it's estimated without the ground truth. And it seems that if there is a change in performance and it's due to data drift, we know that the performance has changed and we know by how much it has changed because you get the actual performance metric. And as far as our experiments can.

Tell, it can do that quite well.

And quite consistently across all use cases and all kind of data sets. And then the hard part is concept drift, of course. So we cannot capture change in performance.

Due to concept drift quite yet, but.

It'S something that we're researching quite well. And also probably that explanation was not extremely coherent.

So please go read our docs and you'll get a much better explanation than what I just gave about our performance estimation.

Yeah, and then the data drift, that's more run of a mill. There's two parts to it. We have like univariate drift detection. So that's basically when the distribution of an individual variable changes. So like age. Then you have multivariate drift detection where nan email gets a bit fancy again, where we developed our own algorithm for detecting multivariate drift. So that's basically detecting drift in the data set as a whole and the relationship between variables. And there it does something basically with a PCA reconstruction error where you basically do a PCA on a reference period and then you do PCA on an analysis period and then you can compare.

How different those PCAs are and you.

Get an error and that error will.

Tell you how different the data in.

Your reference is from your analysis. So then we basically also alert you when changes in performance happen and try to point you in the direction of the data drift that that has potentially caused those changes. Right. So a bit more actionable and try.

Basically it's not caused, so I can't say caused. We don't do causal machine learning just yet. But basically data drift that happened at.

The same time that your model performance has changed. So more correlated. Yeah, and that's it.

Are there any examples of naniml being used in production today?

So since we went open source, we've seen, and this was our hunch all along, is that the performance estimation would be particularly useful in like credit scoring. And so we've identified a bunch of users in the financial industry using Naniml for credit scoring. And when people come to ask us about Naniml, it's often from like that financial industry and credit scoring. But that's.

That kind of goes to the importance of the underlying use case and how obviously you don't want to give loans to people who shouldn't be getting loans and the financial incentive there's is very high. So if we can help reduce the.

Error there, then obviously they're going to be very happy to use an animal. And one of the sad things about being open source is that it's relatively.

Hard to know who's using your software and what they're using it for. We try to identify people using it, we try to talk to who we.

Would define as our ideal user and see who's using it, who's not using it, how are they using it. But in general it's pretty Hard to identify them. And basically we also, because we're still, obviously everyone should be doing this, but in the early days it's super important to work closely with users as well.

So we have a series of design.

Partners where we basically work together with them to deploy an NML iterate on the library and things like that. And there it's really varied like from churn prediction to demand forecasting and things of that nature. So it's really all over the place there. Yeah.

So one thing that I found interesting about NEML is that you decided to make it open source. Can you walk me behind decision making here for why make an EML open source? And what are the pros and cons of going open source as a up and coming machine learning package?

Yeah, so we were doing, like I said, we were doing a lot of research behind our algorithms and we spent quite a long time working with design partners.

So we had like real world data.

And things of that nature to run experiments and make sure that we can build algorithms that work well. But the kind of feedback we were getting from our users were like, okay, we're data scientists and, and if we're using a novel algorithm, we need to know how it works.

We can't just trust you that we're just going to put data into the system and get back some results and.

We don't know exactly how it works and just trust that our performance is all fine and everything is fine. So that was this kind of consistent feedback that we were getting and then from there it was like, okay, our users want to know how this works. And then we're like, okay, if they know how it works then it might.

As well be open source.

But, but in general also for widespread adoption, like before we were open source we were kind of like this no.

Name Belgian startup having our design partners and like getting feedback was very slow.

Basically because we're working with these big enterprises, they don't have that much time, iteration is hard. And then we were like, no, this is not nice, let's go open source. And that's I think what makes the most sense for a lot of data science products. So basically we just wanted to reduce friction, allowing as many people as possible to use it and to give feedback and ultimately building a better product for the data science community. And I would say the main con behind being open source is essentially as a startup you have to find product market fit twice. So for your open source solution, so you have to get mass adoption of your open source solution and then after you've done that or when you're well on your way to that, you have to build a paid offering so you.

Could still exist as a startup. And then you basically have to find product market, market fit for your paid offering as well. So that's definitely a big challenge.

I couldn't agree more. And I love the fact that it's open source and like that, that you're leveraging open source to be able to accelerate the feedback loop. I think the crux of the challenges that NANL is attempting to solve, it's kind of a tension between what data scientists are trained to do versus what is expected of them in the real world. There's a lot of data work around pre and post deployment that is increasingly crossing over to the engineering realm. Realm. And to first start off, I'd love to know how you feel a modern data team should be organized. Do you believe that a one size fits all data scientists that can do both the data science and deployment work, or do you think that these capabilities should be splintered off within the data team?

That's a very good question. I think as most things in data science, it depends.

It depends on the size of the.

Team, it depends what they're working on, it depends how advanced the team already is. I think in bigger companies when you have dozens of use cases, specialization becomes necessary and you already see that you have data engineers, data scientists, ML engineers.

There'S a data analyst, bi, there's this whole slew of roles for the data.

Team and I think that makes a lot of sense. I think as companies become more advanced and more and more models go into production, you're actually going to have it between pre and post deployment data scientists. And I think that like right now you see a lot that monitoring falls under the rules of the ML engineer. But I think that ML engineers are going to move towards mostly the infrastructure and the OPS work and you're going to have this post deployment data scientist who will take over the models in production from a data science perspective. So again, it's like how are our models performing? Are they providing business impact? What are the feedback loops and like really doing analysis and working also to making their models better and to increasing their business impact. And that's a whole set of skills on its own. So you can imagine if you have, I don't know, 10 plus use cases in production, like you can have an.

Individual who's just in charge of all.

Of those use cases once they've been in production. At least that's how I think it will go.

It's the future.

So you never know. I could absolutely be totally wrong about this. But yeah, I think that's what's going to happen.

What do you think are the best skills the modern data scientists should have today?

So putting models into production, I think that will really set you apart. Or if you have any kind of experience with trying to really build a model and actually put it out into the real world and see what's happening, I just think it's one of those important skills that not that many data scientists have. And also really having this feel for the business impact of a model.

Right.

Like a model is not just its performance metrics, like it's technical metrics. It's also like, why are we building this model?

What value does this model add?

How is this value changing over time? How is it impacting other departments and the people around me? And I think it's like not a technical.

It is a technical skill in one head, but the other way. It's also this deep intuition about why.

You'Re doing what you're doing. I find that super important. Yeah.

And just being able to detect these.

Changes in performance, understanding these new concepts like data drift, detection, concept drift, and just really knowing them well and being.

Able to use tools that allow you to do that.

Of course I'm biased in all of.

This because that EVL is what I'm working on, but I genuinely also think that as well.

Yeah.

Now, as we're closing out, Hakeem, what are the top trends that you're looking for within this space that you're particularly excited about, both in post deployment data science, but also in data science in general?

In post deployment data science, it's hard to say.

That's really, really early days.

Maybe in more like mlops type things. I'm really happy to see all of these kind of frameworks coming out that make the engineering side of putting models into production much easier. Shout out to our friends at ZenML. That's like this up and coming framework for MLOps and they're really great and I see a lot of the work they're doing and people actually ask us.

If we integrate with them and things like that. So I really like these kind of.

Of tools and they're also open source.

So plus one for that.

And that makes me really excited that people want to get models into production and that there's people working on these problems. So that's really cool. I think in data science in general, I really like anything that's generative. So like Dall E that came out recently and you see all these insane images or GPT3 in general, like just the text that it can produce and things like that.

That's like, I don't know, it's super exciting. Also, I do not buy all the hype about AI.

And I don't know, there was this Google engineer recently that said that Google AI is sentient.

I'm just like, no, it's not. No, it's not.

Yeah, I don't buy it either.

Yeah, yeah, I don't buy that. But I do find it cool.

The AI community is rarely united over things, but this one was very much so. Like, the AI community is super united over not being. This is not sentient.

Yeah, yeah, yeah. Indeed. Indeed. It's like, dude, it's just.

It's a bunch of functions put together.

Please. Like, you trained it on like, an insane amount of data. Of course it's going to tell you that it doesn't want to die. Or like, it's kind of funny because it's not. Like, it's not weird that AI or.

Machine learning models trained on text or.

Language that's by humans, that it would then exhibit the same behavior as humans when you ask it those questions.

Like, it's very interesting. But yeah, all that generated stuff, I find it's super fun and super cool and also very useful.

Likewise. I'm super excited about what's ahead as well with GPT3 and Dall E. Now, Hakeem, before we wrap up, do you have any final call to action to the audience?

Yeah, I think the first one is a bit cliche, but data science work is very impactful and I think a lot of people don't realize how much it can impact people, society in general. And actually it's a pretty powerful tool. And so it's like this cliche of don't be evil. Like, be conscious of your power as a data scientist and what your models are doing and how it might impact people in society. And really be conscious of it because I think it's sometimes taken very lightly. It's very funny. I was having a conversation recently with a software engineer. He asked, oh, yeah, how come there's not like this machine learning system that just analyzes all the cars on the road and detects if someone is a.

Drunk driver or not? And I was just laughing and I'm like, I don't think that's allowed. And then he was just like, yeah.

As a software engineer, I never have.

To think if my work is allowed or not. I just build and I'm like. Yeah, indeed.

That's why it's important to have AI ethics courses in different programs.

Yeah, yeah, indeed. So I think that's.

Yeah, just be conscious of it. Don't be evil. And then, of course, if you're having models in production and you're interesting to monitor the performance. Yeah. Check out naniml. We're on GitHub. It's totally open source. We'll be open source forever. Our core algorithms will always be open source. Our research will be open source. So, yeah, great to have anybody in the community that finds this interesting.

That's awesome. Thank you so much, Hakeem, for coming on Data Frame.

Yeah, no problem. Thanks, Adele. Thanks a lot for having me.

You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment and share episodes you love. That helps us keep delivering insights into all things data. Thanks for listening. Until next time.