Testing ML systems

Bandwidth for Changelog is provided by Fastly. Learn more at fastly. Com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog Linode is our cloud server of choice. Grab the Nanode plan for just $5 a month. Just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD, and one terabyte of transfer. Let's be honest, you can go a long ways on that five when you do need to scale up, their prices are predictable so you can put your calculator down, you won't need it. We've been running changelog.com on linode for years and we've always impressed by their award winning support team. Check them out@linode.com changelog Once again, that's linode.com changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practical aifm. And now onto the show.

All right, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my CO Chris Benson, who is a principal AI strategist at Lockheed Martin. How's it going, Chris?

Hey, it's going great, Daniel.

How are you doing? Doing pretty good. The winter sickness is still going through our household, so still dealing with that, but otherwise doing pretty good. I think I've avoided most of it at this point or at least got over the worst of it. So that's good. But how are things on your end?

They're going well. I wanted to note that thing from last week about us because I had posted on some social media and people were really surpr so I thought I'd share it with the listeners. So if you are a longtime listener of the podcast, we've been doing this for about a year and a half and Daniel and I actually have never met in person until last week. We were at Project Voice in Chattanooga.

The wonders of the Internet.

No kidding. And so I had commented online how incredibly cool it is that you can develop such a great friendship and collaboration and yet have never met each other in person. So anyway, that's passed now. We've now met in person. It was like meeting family I think for us. And so definitely.

Yeah.

And so anyway, just a very cool thing. I just thought I'd relate that. So yeah, doing great.

Yeah. Awesome. We're starting out. So this is one of our first episodes for the new year. A couple before this, but definitely want to start out this year promoting practical uses of AI and practicality in developing AI and machine learning systems. And I think we've got a great guest today to emphasize a lot of those things. So we're joined by Tanya Allard, who's a developer advocate with Microsoft, a Google Machine Learning GDE, and a Python Software foundation fellow. Thanks for joining us, Tanya.

Hi, It's a pleasure joining you guys.

Awesome. Well, maybe before we jump into some of the things you've been talking about and working on recently, if you could just give us a little bit of your background and how you got into machine learning and ended up at Microsoft, that would be awesome.

Yeah, Well, I started doing machine learning stuff during my PhD. I was using machine learning applied to material science and it was basically to try and identify some materials that could be candidates for tissue replacement. So with a lot of optimization, a lot of materials testing here and there, and over the course of my PhD, I realized that I really enjoyed the computational side of things much more that the experimental and research bit. So after that I transitioned into research software engineering, which was basically research engineering in research institutions in the uk. I then migrated into working for hello Soda, that is company that has machine learning as a service. And I was doing data engineering there, machine learning engineering, research engineering. So it was pretty much everything as you normally do in a small company. And then that brought me into Microsoft because I was already doing a lot of community work, community engagement, doing GDE work with the GDE community in Google, all open source work that I do. And this just seemed like the perfect fit now for me, that's how I got into Microsoft.

That's awesome. Yeah. And you mentioned a few things like you held different positions like data engineering and machine learning engineering, data science and computational research. Sometimes these sort of titles get a little bit fuzzy. I'm wondering from your perspective, I guess one, how you would define what you did differently as a data engineer maybe, versus some of the more data sciencey things. And then also if that more focused engineering experience kind of influenced how you do machine learning.

Yeah, right. I can say I've been working across the machine learning pipeline in all the different roles and as you mentioned, a lot of this roles are very, very fussy or have a lot of things in common. So when people talk about, I don't know, data scientists and data engineering roles in machine learning research, or machine learning engineer rather, they try to use these Venn diagrams. And I found that this is not very descriptive, but rather is, for example, if you're working on the data science thing side of the pipeline, you're focusing much more on the statistics, on developing novel algorithms, or would help your business or your company to get most of your data or get good insights. But then you will probably have or need some software engineering skills as well to take that into a production format with the rest of your dev environment or your dev team. Whereas when you're working on the data engineering side of things, you're focusing much more on all the processes that are extraction, transform, load. But sometimes you still have to know how that data is going to be integrated into your model so that the data is actually usable by the rest of the team. And then the machine learning engineer role is basically the one that binds it all together. It makes sure that everything is robust, accessible, that can be taken to production, that folks that are using the data that is being transformed or are being transformed by the data engineer is actually pulled into a reproducible manner that you can always track where the data is coming to and from at all times so that everything is tightly integrated within your data science infrastructure.

So, you know, I was just looking and I know Daniel had seen it already at your talks that you gave called what is your ML score that you went through at All Things Open and Anaconda Con. And in that talk you were focused very much on kind of QA and testing of machine learning systems specifically. And I wanted to start off by just asking if, you know, as we've kind of talked about the roles there, if you could also talk a little bit about what is a machine learning system? You know, it's kind of a little bit of an ambiguous term. And could you kind of define what that means and what types of things are included in a machine learning system or not?

Yeah, well, it depends a lot on what data problems your company is working on or. Sorry. Or the kind of machine learning projects that you're normally working on. But normally the machine learning system is going to be comprised of whatever during getting your data, which can be a canonical database, then how the data is going to be pulled into your machine learning model or your prediction or classification model, whatever it is you're trying to do with that data, and then how you take that model into something useful by your customer or your teammates. And this can mean Are they going to access it through an API? Is this going to be a standalone web app? Is it going to be on a mobile device, and are you going to be accessing this model? So all of these apparently movable parts that conform to your data warehouse or your database, the infrastructure where you are running, your prediction, where you're training, your model, how you're collating your database, all of that forms your machine learning system. So it's a lot of data bits, a lot of infrastructure, and it could be like things on the cloud, for example, as well, like using the public cloud infrastructure.

And I know when you're talking about testing or validating the ML system, I was just thinking if I was to ask a software engineer and ask them about testing, probably one of the things that comes to their mind is unit testing or integration testing. Whereas if I ask a data scientist or maybe a machine learning engineer what they think about testing, probably the first thing that comes to their mind is testing of a specific machine learning model, the performance of that model in terms of accuracy or whatever the metric might be. And now we have this other kind of category of machine learning system, which it sounds like is more broad than a machine learning model. How is the testing or validation of an ML system or a machine learning or AI system different from the testing of a specific machine learning model?

Right. So I think because the ML, well, the machine learning system comprises all of your pipeline. It has to be a bit more holistic, a bit more integral, and cover all of the parts of that. So if we go back to traditional software engineering, you're testing that your piece of software is returning the results that you're expecting it to do, because you already know what those results or those behaviors are. It's relatively straightforward to design your test cases, create your unit tests, your regression and your smoke test cases, and they.

Should always return the same thing, basically.

Exactly. So it should be deterministic in that sense. When you're testing a machine learning system or a machine learning model, in many cases you don't even know what that end result is because you have your data, you have your labels, if you're doing, for example, a classification problem. But you need to make sure that your system is doing what it's meant to be doing and that it's repeatable that you can repeat all of that. In that sense, you have to ensure that you're testing your data, your features, ensure that the data conforms to that distribution that you're expecting and the behavior that you're going to see. And also the cost that adding more features to your predictive models is adding, because that's a major component, especially when you're doing things on the cloud. Sometimes when you add one feature, you marginally increase that accuracy that you were talking about. But then your compute time or the use of resources that you're using doubles or triples. So you also have to take that into consideration to balance whether that very marginal increase in your accuracy is actually worth all that extra compute costs that you're incurring on. Also, when you're doing your model development, you have to look at things like geometrics, whether the impact of your hyperparameters is also causing impact on your compute resources. Testing for implicit bias, testing for your staleness of your model, because you might then need to retrain your model after a certain period of time if you are acquiring new data or there are significant changes to the API. And then again, you need to test your infrastructure. You need to make sure that you're able to deploy them, deploy your model, your infrastructure, probably using techniques like continuous integration and continuous delivery, because that's essential. Especially if you release a new version of your model, although just tested it before getting into production, it turns out that there is a bug or there's something that needs a rollback. Being able to know how long that rollback or that release is going to take you is crucial. So in that sense, you have to have integration tests against your entire pipeline from data acquisition to data transformation, prediction, and then result serving, whatever that is for your system.

So one of the things I wanted to ask, you know, I know that Daniel and I come from a software development background kind of before we were much into deep learning. And so it's kind of the idea of testing and why you test, and the importance of testing is kind of second nature. But for somebody coming in to this, you know, into deep learning and trying to do these things, it may not be. So I'd like to ask you quite simply, why is testing or validation of machine learning systems so important? And what would be the downside of not doing that?

I think one of the main advantages of you being able to taste your machine learning model is explainability. As we're going into more complex frameworks or more complex deep learning algorithms, it starts becoming increasingly difficult to explain how to reach to a certain prediction and why. Especially when we're releasing machine learning out into the world and is affecting other people, I think it's crucial for us to know that it's actually predicting what we wanted to predict, that it's being transparent and clear and that we can always trace all the predictions that we're doing. Also, test for implicit bias is crucial, especially when we have data sets that are biased toward a certain feature or towards a certain portion of our population. Having test in places throughout all of our pipeline ensures that we can mitigate those biases early on as well. So I think those are some of the most important reasons for us to be testing our algorithms.

Yeah. And I mean, in certain cases, I know you're in Europe right now and there's certain regulations that have come down the pipeline there and of course you're influencing the rest of the world. So I guess it may be partly your own. You know, you're trying to create a development environment that is. Is responsible and you're able to repeat things and actually make incremental progress on things. But also you might be under certain regulations. Right. That you actually have to be able to, you know, give someone an explanation to some degree of what you did with their data. Is. Is that right?

That's correct. I think it was now over a year that all the things, GDPR regulations took place. So one of the most significant things that this brings is that if a customer or someone from whom you're withholding any sort of data comes and tell you, hey, I want to see what you're doing with my data, or I want to have access to all the data of myself that you're storing, you should be able to comply within a standard period of time. So if you don't have mechanisms in place for you to trace this data or trace what you were doing or even to delete, because now your customers should be able to ask you to delete all the data that you have in place for them, you should comply. That's now our regulation. And I think also having a better understanding, as I said before, of where your data sources are and what you're doing with the data and how you're moving it from one place to another is very, very important for, for reproducibility and assurance of our systems.

So I know people that might be coming into AI and machine learning, maybe from a science background or even, you know, there's a lot of different backgrounds, even things like economics or finance or that sort of thing. Some of these things around infrastructure, cicd, monitoring might be sort of intimidating to them. I was wondering if you had any, you know, thoughts as far as who a data scientist needs to work with to make sure that all the right testing is in place for a machine learning system, because it does Impact. Like you said, there's implications for infrastructure for scaling up for changes to data. Who does a data scientist need to be talking with to make sure that all the right testing and quality assurance pieces are in place?

I think a good workflow would be to have the data science team working very, very closely with the machine learning engineering team, if there is one, or otherwise the software engineering team, if you have them sitting together or working very closely. It's easier for both teams or the three teams to better understand what the requirements are, how people are bringing things from research and development environments into production. Because something that I've noticed in some companies or in some teams is that you have the R and D or the machine learners sitting in a corner doing all of their things, developing their models, and then they have to throw things over a wall and hope that the software engineer will take that into production in most cases. And the software engineer doesn't have an idea on how the model works, the canonical database or the canonical data sources, or the transformations that need to take place for that data to be usable. And that's when you sometimes see that folks spend weeks or months working on a model, but then they spend another couple of months or weeks sitting on that model, just waiting for it to be taken into production. Having from day one a collaborative approach where folks define what resources are you going to need, how this algorithm is going to reach out their customers, and what sort of data is going to be accessing to is crucial for both teams to be able to take this from R and D into production in a seamless way. That doesn't mean that you as a data scientist need to do everything or need to be super good at CI and CD and testing and know kubernetes and all of this complex thing. But if you have these teams together, it's easier for both to understand the world of the other one.

Yep.

So, you know, one of the things I'm really getting from you here is that when you're actually working on getting a model into your overall production life cycle and you're integrating with the existing software development and deployment life cycles that you are moving into, it's really part of a larger effort which kind of fits into what a lot of organizations are already doing is they kind of add this in. One of the things I'd like to know is, given that kind of larger team that we've been talking about, how should different roles within that team think about their responsibility for testing? In other words, if you are an infrastructure engineer, what should you be testing if you're a data scientist, what should you be testing? What should the machine learning engineer be testing? How should testing be divided out among those different roles?

I think as you mentioned it, finding these roles and assigning responsibilities is crucial. You as a data scientist, you would mainly be in charge of sourcing your data and your features. So again, this goes back to when you have your data set. Make sure that the distribution of each feature matches your expectations. This is a very basic sanity check, but sometimes because we do it so often, we don't think about documenting it or going in depth into to that. Also making sure that the relationship between your features and your targets and the correlations make sense and that sometimes it needs to go beyond creating just a correlation heat map as well. As I mentioned before, testing the cost of your of your features also something that is very, very important and lines very well with this gdpr. Also ensuring that your system maintains privacy across the entire pipeline. Sometimes we are very, very concerned about privacy of our raw data because that's our most valuable asset. But you need to ensure that your system or every transformation or every data manipulation that you're doing complies with that privacy as well. Also make sure that you are aware of how much time it's taking you to develop a new feature or a new production model because it's also going to help a lot with time churn, which also will prevent you from going into half baked features or having very, very tight data jungles as well. If you are for example the machine learning engineer, you're going to be testing for model development practices and monitoring those models. So again, making sure that everything is checked into a repository, that there is version control, making sure that there is a peer review process, that it not only has to be the data, the senior data scientists, but everyone in the team has to be responsible for them. Making sure again that you're checking your impact metrics, checking the impact of jury tunable hyperparameters, and also check against simpler models. Sometimes because we are so into deep learning and deep learning is the most popular framework or the most popular approach at the moment, or we want to use very sophisticated models. Sometimes it's also good to go back to the basics and compare against a much more simple or a simpler model, just to have a baseline and ensure that we're actually going down the right path and that it makes sense the additional cost that we're going into compute and again test for your implicit bias. And finally the folk that is in charge of our infrastructure, Sometimes it's a DevOps person check for the repeatability of training, making sure that the model specification is up to date and correct and we have properly provisioned all of our data, all of our hyper parameters, all of our models, the training and everything. And integrate the full test for the integration of the full machine learning pipeline. Making sure that again, everything is reproducible, whether during infrastructure. Sometimes you will have infrastructure that is your development, your staging and your production. Making sure that across those three environments you can get reproducible results. Sometimes you'll have changes in infrastructure that will imply changes in your productions that gives you indication that your infrastructure is not reproducible. So you all have to make sure that that's not the case as well. And again, as I said before, test that you can do releases and rollbacks in a reproducible, reliable and robust manner. Because if it's only one person, for example, in charge of deploying things into production and what's going to happen when that person is on holidays, for example, you're not going to be calling them at 3 o' clock in the morning for them to revert back to a previous version. So make sure that there is a robust way to do that without having a bottleneck. There is also very, very important and ensuring that this rollback can be done safely, you know, in a controlled manner so that anyone can do that. Ideally you would have an automated pipeline that would take care of that. So testing all of these little bits will be assigned within the specific role responsibilities and it makes everyone lives Easter as well.

Yeah. And I'm so glad you went into the details of those different areas. It's really helpful and there's definitely a lot there to work on. There's. There's a lot probably for every team, whether a small organization or a large organization, that they can always be improving on. I know a lot of the things that you talked about. I know I have work to do on, on my end, but one of the things I liked about the talk that, that you gave on this was that you developed a sort of very practical, I guess I would call it a rubric for kind of scoring yourself in these different areas, which is what you meant by giving yourself an ML score and then kind of helping you focus your effort on where you're lacking and where you can improve on. I was wondering if you could go into a little bit of the details of that scoring system and that rubric to help me develop my own ML score and make sure I'm putting in effort where I can make the most difference.

Sure. Throughout this conversation, I've placed a specific emphasis in data science testing, machine learning engineering, and infrastructure testing. So within these three areas, you have the different steps that I've already mentioned, the things that you should be testing at a minimum. So, for example, if you have no testing in place for any of the steps, that would give you zero points because you have no testing. If you're testing, let's say, from the data science perspective and you're manually testing the distributions or checking the distributions of your data and then ensuring that the training of your model is reproducible, but all of this is done manually, you can assign yourself probably one point. If you or your company has reached a more mature level where all of these tests are done in an automated fashion, probably through your favorite CI provider can be GitHub Actions, Travis Azure Pipelines, or GitLab CI, then that would give you two points. And as you go creating your test, you can add those points and then compare against the three stages, whether it's data science, infrastructure and machine learning engineering, more than likely is that you're going to be very good in one of these areas and not so good. You'll have a lower scores in one or two areas. And the area where you have the lowest score is the one that you should be paying more immediate attention. Start trying to level up that score.

Yeah, that's super helpful. I was just trying to think through while you were talking what my score would be. I don't know, Chris, if you were doing the same. Yeah, for me. So I guess there's the manual and the automated tests, and then there's the three sections kind of data science, machine learning, development and infrastructure. I think probably at least in my organization, on the stuff I work directly in, I've got a fairly good amount of manual tests going, but definitely not everything's automated. And probably we have been more focused on the data science and machine learning development side than on the infrastructure side. Just because, you know, our, our organization, being a nonprofit, isn't already operating a ton of infrastructure and have some of those things in place, although they do for a variety of projects. So I'm guessing I've got like a one going on and maybe a two in a couple of the first sections. And then maybe where things are needed is more on the, on the infrastructure side. What about you, Chris?

I think it's more or less the same. I was, as you were saying that I think the things that for me personally, and I'm part of a larger team, but the things personally, the closer we are to kind of what I grew up in, in the software development side. I think we're actually. And this is where I'm a little bit different from you, probably more automated and actually attending to tests on the infrastructure side. And as you move over toward the data science, it's probably, you know, pretty decent on manual tests, not. Probably not a lot of automated tests. And so I think it varies a lot for us whether it is a team effort. And I think we probably get a little bit higher, higher score because different people are attending to different parts of it versus when me or somebody else is doing something alone. And I'd say our scores probably fell off. So I think it's definitely probably a testament to throwing people from different perspectives on the team. Probably yields us a higher score since we're combining that. But boy, I'll tell you, after talking through this and then hearing Tanya talk about her scoring, I'm starting to realize all the places that I need a little bit of work.

I think that's cool though, because it's like, you know, if you were to just present all this, like what Tanya presented in general, like all the pieces of the machine learning system, it could be overwhelming and a bit crippling. But if you're able to kind of zero in on where you need to put the most effort, I think it's really helpful in terms of, you know, starting somewhere and at least getting some more testing off the ground. I'm curious, Tanya, as you've kind of gone around and presented this to various groups groups, what's been the feedback that you've got in terms of where people. Is there a consistent place that you think data science teams or machine learning teams are maybe not putting a lot of effort where they need to? Has there been any sort of trends in that sense?

I think something that has come up a lot is infrastructure in the sense of they have a DevOps person that normally takes care of the infrastructure and. But everything is very flat in the sense of they always have the same tests, they always run the same processes without adapting to the specific cases or specific situations for machine learning, because sometimes we would have a bit of different behaviors or we would need something different. If you are serving, for example, a model that is going to be accessed by a lot of people over a web app, for example, or a more simple e commerce app that folks are going to be accessing to buy products or the such. So I think the understanding and the testing of machine learning infrastructure is very often overseen by a lot of teams.

Have you heard of our newest show called Brain Science. Yes, Brain Science. It's a different kind of show. I know, and it's probably one of the ones that reaches the furthest out from our typical listener audience. But this podcast is what we call for the curious. And what's cool about this show is we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and pretty much what it means to be human. If you've ever thought about why you do what you do or why otherwise others do what they do, then this show is for you. Head to changelove.com brain science to listen, subscribe, and learn more about this awesome show. Here's a preview of a recent episode called One Small act of Kindness, talking about empathy and mirror neurons. So it sounds like pliability and flexibility is a pretty crucial role too, in relationships. Because if you're not flexible, bendable, pliable, whatever, however you want to phrase that, if you're rigid, it's going to be difficult for you to flex to enable change or to what you've said before, recalculate, accept new data, analyze that data, make a new plan, and iterate towards a new action.

Yeah. One of the other things involved with this flexibility would be what researchers have discovered as mirror neurons. And so mirror neurons are these neurons within the brain that help us sort of get access to another person's emotional experience. And so there's an action component in it that it was first discovered actually with monkeys, and this sort of mimicry that occurred by watching somebody else do an action? Well, in the same way I can sort of watch somebody else walk through something in terms of an emotional experience. And if I'm holding space for them in my mind, like my body, physiologically, these mirror neurons come. Come to play.

Is that why people cry when they watch movies or certain movies, because their mirror neurons are firing? Because they're watching somebody go through a situation.

Situation.

And they're empathizing with them and can't help but encapsulate themselves into their scenario and feel what they're feeling? Is that why.

Yes.

Okay, so is that why anybody cries at anything when it's like, say, movie related? Because they're. That's what's happening.

Yeah. Think about it. Sort of like this emotional contagion. Right.

So that's interesting to put it that way. And we've said mirror neurons several times. But this emotional contagion, I believe, is actually a better subtitle for mirror neurons.

Yeah. And so some of this emotional contagion or mirror neurons. Like the research has been rooted in aspects of pain because if I can recognize sort of the suffering of another.

All right, to keep listening, head to changelow.com brainscience9 that will take you to the episode titled One One Small act of Kindness. Marielle and I dig into this thing called empathy As a construct. We ask questions like what key brain structures are involved? How can we better understand empathy to be able to better navigate ourselves and our relationships with others, both at home and in the workplace? It's a deep subject, a very Fun subject. Again, changelog.com BrainScience 9 or 6 search for brain Science on your favorite podcast app and subscribe. We'd love to have you as a listener.

So in a slight change of topic here, Tanya, we certainly, I think, would be remiss if we ended our discussion about testing and machine learning systems and integrity without mentioning notebooks. I understand you gave a talk that was called Jupyter Notebooks Friends or Foes recently, and I was wondering what was your conclusion in that talk, especially, you know, given the emphasis in this episode on integrity and reproducibility. Could you share some of your thoughts there?

Yeah, I've given that talk a couple of times and it's been very well received because Jupyter notebooks are a tool very, very commonly used by data scientists. And I'm going to say I love Jupyter notebooks, but I always try to use them within reasons and even with the teams that I work, I try to have these standards on how we work with them, have processes again in place. There are a lot of very, very good things of the notebooks, but there are also a lot of hidden things and caveats. So the more aware you are of this, the better use you can make of this tool. And then again, it comes to having these processes and these workflows in place. For example, something that is relatively easy to do is if you have someone to help you or you spend some time is for example, when you're working with notebooks and then you check them into version control, having, for example, a GitHub that will make sure that all the outputs are cleared out, that you are conforming to certain standards, that your paths are not referencing to, local paths before those are going to be checked into version control. And then again having testing of your notebooks, making sure that your environment is reproducible, that makes a very, very dramatic change in how folks are using it. Because I know some software engineering folks, they absolutely hate it. They absolutely hate not picks because it also allows for a lot of bad practices in the more traditional sense of software engineering. But I think again, if you are dear to this, you know, style guides, having, enforcing workflows that will allow for this quality assurance just goes a long way. And also being smart about what you're using the notebooks for and when it's good or more advisable to move from notebooks into a more traditional development practice as having your scripts and your tests and importing your modules and being able to discern between these two use cases or these two different approaches rather is very, very valuable.

So I'm curious. So I guess first I have a couple follow up questions because there's so much here and of course jupyter notebooks are everywhere. So it really does influence a lot of people's workflow, I guess the first thing. So you mentioned a couple of the checks that you might do when you're checking your notebook into version control. But you also mentioned maybe some, some caveats where notebooks kind of can break down. I was wondering if you would go into that a little bit more maybe. I know, for example, one area that I've seen is where, and this is something that Joel Grus mentions. We had him on another episode and he gave a talk as well, more controversially titled I guess I don't like notebooks. But he was saying, you know, there's a lot of hidden state in notebooks where you know, you might run things out of order and it's hard for another person to jump in and actually recreate that state that you might have actually been a little bit, you might have had some misunderstanding of how you got to a certain place in the notebook. I was wondering if you had similar experience.

Yeah, right. I totally agree with him in that sense. There is a lot of hidden state because it gives you jupyter notebooks give you so much flexibility when it comes to executing cell and getting down with straight away and then going back and forth. But I think especially for people that are getting started or have never had proper software engineering practices, this is not obvious. So sometimes they will work in all books, jump from one cell to another, get to our result, whatever that result is that seems satisfactory for them and they're like, okay, I'm done and just check or insure their notebooks in the state that it is. But then if you go again a step further and you have these practices of okay, I finished this notebook, so I'm going to clear everything, restart my kernel and run all of the cells to make sure Again that my results are reproducible, then you're adding these extra quality checks. There are tools, for example, like nvivao that I love and I've worked with a lot that allows you to do this regression test. And they're very, very useful because you've already have the state of your notebook saved. So it runs again in background yourselves and checks whether what you're getting is the same what you got before. And this is very, very useful for iteration tests for validation of somebody else's work. But then again, something that is very, very obscure is the dependencies that you're using. Unless you are actively sharing your environment through pinned versions in a code environment or you're using PIPM for Poetry or Docker or something of the such, it's very, very obscure. And a friend of mine has done a lot of study around how slight changes in packages version can actually change the result that you're getting in a workflow or in a study. And this is, I think all of the hidden state and all of the weird practices. And then especially when folks only learn to use for example Python through jupyter notebook through jupyter Notebooks, it becomes very problematic because then they're like, okay, so I normally import a library like let's say Pandas. If I develop something in a jupyter notebook, how do I import import this notebook into my notebook so people start misusing the notebooks, if that makes sense.

It does, it does. I think when in my own thought process I think it I come from, I have this bias where I'm coming from the software development background as I, as I had noted before. And so I do think jupyter note are wonderful for kind of experimenting and trying and doing your experimenting with feature selection, stuff like that. But I also know, speaking for myself at the first point where my model starts to stabilize a bit and I'm doing less experimentation and variance and you know, from a minute to minute kind of thing. That's where I'm always looking for that first moment where I say, okay, it's time to get it out of the notebook at this point and really start wrapping it with software development best practices. You know, as we've discussed, I'm kind of curious, what is that point in your own workflow speaking for yourself? And also what would you definitely as part of that not want to see happening in a jupyter notebook? At what point would you be saying I've stayed in a jupyter notebook too long in your personal workflow?

Yeah, I think I very Much agree with what Gigi said. Jupyter notebooks are excellent for prototyping to doing very fast things and get the, the initial part of the R and D process off the ground. They're amazing. Another use case that I found that is really good is parameterizing Jupyter notebooks using tools like Papermill. But then once my model starts stabilizing and I need to start making more consistent predictions or validations or proper training, I tried to go into a more traditional software engineer practice. There is this tool by Fast AI called NBDev where they try to bring all of this literate programming into Jupyter notebooks so you can have your code and your tests and then develop your library from there. I think it's good to start bringing the software engineering practices into the workflow of people. But then again, once I started finding myself that I am reusing and calling a lot of functions or methods that I've declared into a Jupyter notebook and I have to reuse it and probably for the same workflow or other workflows, that's also an indication that, okay, this isn't working. This has to become a standalone module or a standalone package that I can use and share and reuse and maintain separately, rather than, you know, having bits and pieces in multiple code bases or multiple Jupyter notebooks and having to keep that updated if needed.

Awesome. Yeah, I would definitely encourage everyone, all the listeners to go and listen to your talks, both on the ML scoring talk, but also this Jupyter talk. This is really useful and very practical things. So I would definitely encourage people to go there also. Thank you for mentioning Fast AI. We actually mentioned them, you know, a good bit on the podcast because they, they contribute so much. So yeah, I think it's a testament to what they're doing that they're mentioned so much. I was wondering kind of to close us out here. So we've talked a lot about, you know, maybe some things that certain people, certain listeners especially who maybe are newer to the software engineering side of things, might be a little bit intimidated by, whether that's like, you know, Python project structure and tests and automation and all of this stuff. I was wondering, because you're also, you know, in the role as a Python Software foundation fellow, do you have any good recommendations for maybe people that are wanting to level up their ability to code good Python that has a lot of, of integrity and integrate tests and, you know, Python deployments and all those things? Do you, do you have any good suggestions for people in terms of Learning resources and ways for them to level up in that sense.

Yeah, well, there are a ton of resources out there. That is the problem that it's so easy again to fall into this rabbit hole right there is if, for example, if someone wants to get into more DevOps kind of thing. Emily Friedman, that is one of my colleagues in my team, she just released a book called DevOps for Dummies. It's not focused on machine learning. It's a very, very good resource to get you into that devopsy mindset and understanding how you would integrate continuous integration and delivery into your projects. And then you can interpolate some of those things into your own data science stuff. Something else that I recommend is I've talked a lot about collaborations across teams and team members. Sometimes just sitting down with the software engineer and have conversations or pure programming sessions where you both sit and start writing tests or just discussing about continuous iteration, continuous testing, continuous deployment, testing of your programs and your models goes a long way because again, you're learning from the other one and getting things off the ground.

Awesome. Yeah, those are great suggestions. I know for me personally, I was. It was a little bit nervous to sit down and pair, program or be next to some software engineers when I was first getting started out of grad school, but that's probably one of the ways that I learn the most the fastest. And so I can definitely recommend that. And. And hopefully you got some good engineers on your team that are receptive to that. Well, Tanya, it's been super instructive and really great to talk with you today. We'll have links to the various talks and the other things that we discussed in our show notes and really, really appreciate you taking time out of your busy schedule to talk through some of these things with us. And I hope we can meet at a conference or somewhere in the near future. Thank you so much.

Thank you for having me. It's been a pleasure.

All right, thank you for tuning into this episode of Practically AI. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our air errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on linode cloud servers and eleno.com changelog check them out Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.