Running Machine Learning Experiments in Python

Sa. Sam.

Hello everyone and thank you for joining today's codelong. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance chance to join. In the meanwhile, though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments, depending on what platform you're watching on. And tell us something that you'd like to get out of today's session. If you have any experiments that you'd like to try that you are maybe, yeah, thinking of applying stuff to from this session. Then let us know what they are in the chat as well. We are going to be using Data Camp workspace in today's session, so if you don't have a Data Camp account yet already, then please sign up for one. You won't necessarily need a paid account to code along with us today, but obviously to get full access you will do. So. Yeah, make sure you get signed up so you can code along with us. I'll be posting a link so that you can code along with us and I'll also be posting a QR code on screen, so you've got two ways to get on that as well. This session is going to be running for an hour until 12pm Eastern Standard Time, or until 5pm in the UK Greenwich. Meantime, if you haven't already, make sure you get registered for the event on Data Camp so that we can send you the recording and the resources as well. I've posted a link to that in the chat. You can also head over to datacamp.com forward/webinars, but I'll also be posting it again in the chat just so you don't miss it. So, yeah, make sure you get signed up so that we can send you the recording and resources and make sure that you also keep an eye on the chat so you can go along with us as well. Brilliant. I'll be back to repeat these messages for any of you. Join us shortly, but until then, enjoy the background music.

Sam. Sa.

Hello everyone and thank you for joining today's codelong. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about 20 seconds or so. We've just been waiting so everyone has had a chance to join. Let us know where you're joining from using the chat or the comments, depending on what platform you're watching on. And yeah, if you've got any questions or thoughts at any point throughout the session, let us know in the chat as well. We are going to be using datacamp workspace today, so if you haven't got a Data Camp account already, then please sign up for one. You won't need a paid account to code along with us today, but yeah, you can code along with us by scanning the QR code code on screen when it appears. And I'll also be posting a link to it in the chat as well, so keep your eyes out for that. Also in the chat I've sent a link so you can register for today's event. If you register, we can send you the recording as well as the resources as well. I've sent a link so you can do that in the chat and you can also head over to datacamp.com forward/webinars. Brilliant. I think that's everything from me. So now I'll hand you over to your host for today's session. Richie. Richie, please take it away.

Hi there, Data Scamps and Data Champs, this is Richie. Now, a naive perception of machine learning is that your project's done once you have a model that makes predictions. But in reality what often happens is that you need to make several models and then run experiments to see which ones perform better once you expose them to real world data. And that means you need to start worrying about managing all these different models and managing results of the experiments. So in addition to your standard machine learning tools like SciKit Learn, you also need management tools like ML Flow. And so today you're going to create and run a machine learning experiment. Today's guest is Volkert Steinman. He's the founder at Folkert Data. He provides science and machine learning consultancy to services to companies and he previously worked as a machine learning engineer at Engines and he's also the instructor of DataCamp's MLOps Concepts course and the Predicting Temperatures in London project on which this code along is based. Over to you, Polka.

Thank you, Richie. Yeah, indeed. As you mentioned, the general thought about machine learning is that you're done once the machine learning model is done. But of course that's obviously not the case. So today we'll be doing a project on running machine learning experiments in Python which, as Richie mentioned, based on a project which is also in the workspaces on Data Camp. Yeah, so let's dive into it. Before we do the code along, I just want to give a bit of background which I think is useful for getting a bit of context on this project. So today's use case is about London weather data which is sourced from Kaggle and we're going to try and predict the mean temperature in in London. This data source obviously is really hard to predict the mean temperature. There's lots of stuff involved. So just see this as sort of dummy data we use, which is easy to understand. It's not the main focus of today. The main focus is to structure your machine learning experiment. That's what we'll mainly be doing and also learn how to experiment with different models and parameters. Now why is this important? Well, I'll dive into this in a bit. But mainly what you see is machine learning is quite a complex thing to do from start to end. Getting a machine learning to production involves a lot of steps and, and doing this more efficient is very useful. So I want to give a bigger background on where sort of this ML flow tool originated from. And so running machine learning experiments efficiently is part of something we call mlops. And mlops, the definition of it is that it's a set of practices and tools to design, deploy and maintain machine learning models in production continuously, reliably and efficiently. And just as I mentioned, there's a lot of steps involved from the beginning up to the end. Right. We want to plan the machine learning use case, we need to prep the data, find good quality data, then do model development. And then once it's in production, we need to continuously monitor and evaluate it. And so doing this more efficiently can really help. And the focus of today is on this part. So the model engineering and the model evaluation. And yeah, a big thing in doing model development is machine learning experiments. As in any experiment, the amount of configurations can become huge. So see it as sort of turning knobs to create the best machine learning model. We can turn a lot of knobs. Like we can have different machine learning models, different parameters, different versions of the data, different execution scripts and environment configurations. And if we don't have a structured way of doing these experiments, we can lose the configurations and the outcomes we found out before. So imagine you're working at a company, you're doing different machine learning use cases, you end up with a model that works well and then in a couple of months your manager is going to ask you, hey, can we improve on this model that we produced a couple of months ago? And then you might sort of think, I didn't really track or monitor anything I've done in these experiments. And it's hard to find what you did before. And so that's what we'll focus on today is use a tool called MLflow. And MLflow is an open source platform for the complete machine learning lifecycle. So this lifecycle I just showed you today, we'll focus only on machine learning experiments. But MLflow can do much more than that. And using MLflow today, we actively track the configurations and the outcomes to make sure the experiments reproducible. So in the case that your manager asks you, hey, this experiment you do done, can you redo it? You can easily say, hey, I've really tracked the configurations I had three months ago and work your way from there. And so that really structures your process. So yeah, let's dive into the code along, Let's see. So yeah, today we'll focus on the London weather data and again, obviously it's very hard to predict the weather, but we just use it as sort of a dummy data to use it as an example of example data. The focus of this code along is to run machine learning experiments and as in any machine learning experiment, you first want to do a bit of exploratory data analysis and then use, then we're going to use ML Flow to run experiments on what models and what hyperparameters to use. So before we can start, we want to. Let me just make a copy of the code real quick. Yeah, and so before we can use MLflow we need to install it because the datacamp workspace doesn't include MLflow by default. And so we can do this by running the PIP install command here. And then it's hidden here, but it installs MLflow, so that's done. And then we can import all the libraries including mlflow. So we have Pandas here, we have numpy, we have mlflow for the tracking, tracking the experiments. We're going to use Seaborn to do some simple visualizations and then we have SciKit learn for the machine learning stuff. It's all clear. So we're going to be working with data stored in the London weather CSV. This contains the following columns. The date, the cloud cover, sunshine, global radiation, maximum temperature, mean temperature, minimum temperature, precipitation pressure and snow depth and yeah, let's start here. So I think this is something you probably know how to do, Weather CSV. So here we're going to load the data using Pandas read CSV function and we're just going to show the first five rows to make sure we've loaded the data correctly. So here you see the date column and all the columns we just mentioned. So that's great. And then let's just show the information of all the columns Nothing really special here. All the columns except for the date column are floats. Yeah, I think this is not very hard to do. Then we have the exploratory data analysis. So now that we've loaded the data set, let's do some exploratory data analysis to understand the data better. Includes handling missing values, feature engineering and visualizing the data. What we first want to do is convert this date column to something we can work with. So now as you see here, this is still an integer and Pandas has a really nice 2datetime function. So we can easily do some manipulations on the date. Let's start by doing that.

Oh, Volker, could you just bump up the text size one spot just so it's easier for people on small screens to see.

Is this good?

That's better. Thank you.

Yeah, great. So yeah, the date column, we're going to use Pandas to date time function. And then we have the date and we're going to specify a format in which we want the date or in which the date is now. So like as we saw here, the date's currently in this format. So we want to give that as input to the datetime function to make sure it gets the right date. Gets the year, month and date from the column first. And now you can see that Pandas transformed it into this date column. Also want to do is check the number of missing values. Just give it measure a bit how what the quality of the data is. You see here that most of the columns don't have that many missing values except for the snow depth. We missed quite some values there. That's on a total of. Let's check total of columns. That's on a total of 15,000 rows. So about 10% of the rows in snow depth are missing. Then we will group the data by year and month. And this comes in really handy when we want to visualize it. So what we'll do here, and this is why it's so nice to use the to datetime function from Pandas, because we can now easily get the year from the date column. And let's also do this for the month. Now we've created separate columns for both the year and the month. So let's just check that to see if it worked. And now you see here we have the year separated and the month separated. And why this is nice. Well, if we want to visualize for instance the mean temperature, the thing that we're going to predict today, then we can make a line plot using Seaborn using the year. On the X axis and the mean temperature Y axis. Oh wait, I'm going a bit too fast. So now that we have the year and the months, you want to group them. It's one step I forgot. So what we'll do is we group by, we make a separate data frame in which we group by the year and month. List. We use this parameter to make sure that it doesn't group them, uses this group by as an index. And then we can just find the mean per year and month for all these parameters. So I'm just going to run this and then we can see what the result is here. Now what we've done for this data frame is we have the year and the month and the mean, all these mean values per month. And now we can easily visualize this because visualizing it on a day to day basis is probably not that interesting to see. So now we can do what I wanted to do, namely use the year and the mean temperature. And then the data frame we just created. And then we see the mean temperature per month, you can see that it goes up a bit over the years. And this is just to get a bit of sense of what the data looks like. Let's do the same for the sunshine. And then we get the average sunshine per month. And now one more thing we want to do in visualizing is this is important for doing the machine learning experiment. We want to visualize a heat map of the correlations. So what this means is we look at how each column influences the other column. This will probably become clear once we visualize it. So we can make a heat map. We can use the correlation function. And now as you'll see here, we can see the correlations between all these columns. So how you need to interpret this is that for instance, the sunshine has a high correlation with the mean temperature, namely a correlation of 0.4, which is quite obvious because if the sun shines, the temperature will probably be higher. So more sunshine increases the temperature. And we can for instance also see that the precipitation, so the rainfall has a negative influence on the sunshine, which is also quite obvious because if the sun shines, there's probably no rainfall. Sometimes that happens, but quite often it doesn't. And now what we want to do is we want to make a selection of columns that we're going to use to predict the mean temperature. Because as you see here, not each column influences the temperature that much. So you for instance, see that the snow depth has a very low correlation with the mean temperature. Sorry, is that correct? No. So again, what you See is that the month and the mean temperature, they correlate. And for instance, the precipitation and the pressure, they have quite a low correlation with the mean temperature. And so what we want to do is we want to take the columns that are important for predicting the mean temperature as a predictor of the mean temperature. So that's what we'll do here. Let's call this weather metrics and we'll define the columns that we're going to use as predictors for the mean temperature. So we see that the cloud cover has some correlation. Sunshine, global radiation. We're not going to use the maximum and minimum temperature, but because of course that already indicates the temperature. So using this as a predictor would be a bit unfair. We'll not use the precipitation because it's quite low. We'll also not use the pressure, we will use the snow depth and we'll also use the month. Then let's define our new data frame with only these columns. This relevant columns. When I run metrics, Getting a global radiation is not an index. I misspelled that. Now we can see that we've selected only these columns. All right. Oh yeah, there's one thing I forgot. We also want to include the temperature here. That's the thing we're going to predict. So I have to reload the data frame again. One second. So we do all these blocks and then when we again select the relevant columns, Then we have all the relevant columns. So we're going to use these features as predictors to predict this target variable. But before we do that, we want to process the data into training and test set. And because the main focus of today is doing machine learning experiments, I've already put this code in the notebook, but I'll go over it to make sure that everyone understands what's happening here. We have a function here that takes the data frame, feature selection and the target variable as parameters. And then what happens is that we drop the nonce for the target variable because we can predict something that's not there. Then we select the features with which we want to predict the target variable. We put the target variable in the Y variable and then we split this into a training and test set, which we normally also do in training, a machine learning model. And then what we want to do is for the missing values, we want to impute those. So we saw before that we had some missing values in the snow depth column. And here we can impute them using the mean, which means that if we miss the variable in one row, we're going to use the average of the rows next to those to, to fill in this value. Then we're going to split or impute this for the training and test set separately. Because what you don't want to do is impute the values on the whole on both the training and the test set together. Because that would mean that we can have some data leakage. But that means we can also share a source on that from Scikit Learn. That goes a bit more in detail on what that means. But in short, what it means is that if we do this imputing on all the data, it means that we also, it might happen that some of the imputing leaks into the test set. So the mean value that we impute some value with, it's gonna leak into the test set and we don't want that. Then we're gonna scale the values because it helps to scale values to improve the performance of the machine learning model if all the values or all the features are in the same range. And here again, we're gonna do that separately for the training and test set. Yeah, that's what the function does. It's nice to put this into a function because let's say we find out we want to use different features in the future or I find out that my first performance of my machine learning model is really low. I can just easily use this function to use different set of features. It makes it easier to reproduce. So already made this feature selection before a bit. We have the target variable which we need as an input for the function and then we're gonna split, call the function to split everything into a training and test set. Now this all ran fine and now we're going to come to the interesting part here for today, which is the machine learning experiment. If you've ever done training a machine learning model, the simplest form of this would look something like this. So we call the linear regression model from Scikit Learn. We fit the training data, we predict based on the test data and then we can use some metric to test how well our model works. Now what happens if you do this numerous times? So for instance, you do this for a different data set. You do a different training test, split. It means you have to rerun this multiple times. But what happens is that you don't track the results and that's where MLflow comes in. So here I'm showing a way on how to use MLflow. So what we do here is we have, we call MLflow create experiment. For now, we call this example Experiment Run. And what we can do with These functions, if we start MLflow, start run, we call the model, we fit the model, we predict using the test set and then we calculate a metric on, well, on how well our model does. And using MLflow we can log the metric and also log the model so that when we look back in the future, we can easily find what we did there. And this is nice because if we do this multiple times, MLflow automatically tracks the results. So here's the example. For the example experiment we just run. Let me see. Something go wrong, Right? This is what I wanted to show. So what you do when you hit search runs with MLflow and you use the name we just used for this experiment, you can see everything in a nice table here. So you see the run ID and experiment ID the start time. So this is today, just now, the metric that we track using the log metric function, we also had this tag that we used. So all the data from this experiment is in this table now, which is really nice and actually I think it's nice to show what MLflow does in the background. It just saves this run with the things you see in this table here. So it saves the run with the metrics you used. And then we'll come to this later, the parameters and the artifacts. So that's how MLflow works. And that's also what we will be going to do now, using the London weather data. So we will now do this for the London weather data set using two different regression models and we will just use one parameter. But in practice you can use ton of parameters, tons of parameters specific to the model that you're using. So yeah, let's dive in. We're first going to declare the experiment name, something we track the experiment with. Then we do a for loop to go over the depth values. We make sure to use random state, which is a parameter to make sure we can rerun the same experiment and get similar results. And then we fit the models to the training data, get the root mean squared error of each model, lock the parameters and metrics, and then we can easily see these results in the table and mlflow has tracked them. So yeah, let's go first we want to define the experiment name. So what we're going to do is declare variable experiment name. We call this regression v1 and we're going to experiment idea of ID, which we do by calling MLflow's create experiment. And this gets the experiment name as input. Then what we will do is set up a for loop to go over different, different depths that we use as a parameter in our regression models. So let's do that Right. Now we're going to use these depths. So it's going to loop over these numbers and we're going to use this as a parameter. In each model we create a parameters dictionary which contains these max depth, which is one of the parameters that is called in the regression functions. And as I mentioned, we're going to use random state, which is a parameter from Scikit Learn that allows you to rerun the same experiment with the same at the same start sort of. Now let's define a run name for this experiment. So each experiment can have different runs and we're going to use this. IDX value which we defined here in a for loop which just uses enumerate. So we go over these depth parameters and then using this enumerate, we get the idea 0, 1, 2, 3, 4 from this for loop.

Sorry to butt in. Notice that the original bitly link that was posted this workspace, it was posted to the original version, not to a copy. Reese has just posted a new bitly link. So for anyone who has jumped into the original version of the workspace, you need to click the new bit link that was just posted in the chat, go to that and sense you've got your own copy to work on and you can join in codalang coding along. So sorry about the confusion there. Please use the new bitly link, not the old one. All right, Volker, sorry, Carry on.

Yeah, all right, thanks. So, yeah, just as in the example code I showed you, we're going to start the run and here this takes the experiment idea that we've defined as a parameter, and it also takes the run name, the parameter which we just defined. And then what we're going to do is get the decision tree regressor model. We're going to use the parameters we just defined as input, and then we're going to fit the training data on this model. And then we'll do something similar for the random forest regression that we're going to use. Parameters and fit it to the training data. Now what we first do is get a prediction from this each model. So we'll define this as predicting the. Test set using the decision tree regressor. And again we'll do the same thing for the random forest regressor. And then we want to calculate the root mean squared error for each of these models to sort of score how well the model does. We can use this R2 score function. We're going to compare the target variable test set to what we predicted and Again, let's do the same thing for the and forest regressor. All right, now what we want to do is use the MLflow log parameter function to log the parameter that we're trying out here. So parameter that we have in the for loop, the max depth. We're going to log for each run what the depth was that was used in this run. And then we also want to log the metrics for each model. So we can use mFlow's metric log metric function to get the R2 score of the decision tree regressor. And let's do the same thing. Force regression. Yeah, and then this should be all set. So to go over it again, we've made a for loop to try out different parameters for both of these models. We define the parameters here and we use them as input in the model. During each run, we predict the test set, calculate the score of the models, and then we log this to make sure that we track what the results were of each run. Let's run this experiment. This seems to be all done. Then what we want to do, what we can do again using MLflow is search the run that we just did, use the experiment name variable that we've defined before. Now this is the nice thing about mlflow. We see here all the runs that we did. So we loop five times through this for loop, which is why we have five runs here. And then what's great here is that we see here what we looked, we looked the parameter, the max depth that we looped over. So 1, 2, 5, 10 and 20, and we can see for both models what the score was. So for the decision tree regression R2 score, we see that it goes up, up until 10 and then it becomes worse after we go to 2, 20. And for the random forest regression, we see something similar. And this also allows us to easily compare the two models. Of course, you could also do this in an Excel or just note it down using a piece of paper, but that's not very scalable. And if you want to do this for a lot of models, want to do this for a lot of parameters, you can easily do it like this and then compare the best result and take the model with the best result. So yeah, I think that's, that's it for the code along. I think it's now time for the Q and A, right?

Absolutely. So for anyone in the audience, if you've got a question for Volker, please do add them to the chat. In the meantime, I've got a few questions for you. So in this Case the experiment you're running, you had two different versions of the model. So it was just two different types of model. You wanted to compare the performance. Is that a standard format for an experiment or other of the types of experiment that you might want to run?

I mean a bit of an annoying answer to that would be, depends on your use case of course. But I'd say most of the times you sort of iterate over, over different experiments. Right. So you might want to start with three different models just to see what performs best and then zoom in a bit on. Like for instance, the result I had now was that the random forest regression on a certain depth was, was quite or was best on this 10th depth. But what you would, the next step here would be to zoom in and maybe use the depth on 12 or 8 and then find the sort of best score there. I think going over it iteratively, start high, high level and then go more precise to find the best model. I think that's the quite. In my case, that would be the normal way of working.

Okay, so you're looking at like different classes of models and then you're also looking at the. So in this case the depth of the random forest, that's a hyper parameter. Should be looking at hyper parameter as well. It seems like there are lots of, like there are many different possible models you could have. And then also I mean you can look at like the different features that you're putting into the model. So I mean the number of models you can have, it could be hundreds or thousands. So how do you go about selecting like which things you want to include in your experiment?

Yeah, so there's, there's, I mean we, we only mention here the parameters and the models. Of course there's tons of other stuff as well to include. Like you could have problems that, or use cases that involve more that where you need to involve more the data quality. So again, it really depends on the use case, what you want to do for experiments.

Okay, all right. And I just want to make sure that I've understood the workflow correctly. So it seems like basically you want to write your code for a standard machine learning flow. So you're selecting your features, you're fitting the model, you're making predictions, you're crunching some statistics. I think you have like R2 score here as the metric. So you write that code first and you do it for each of your models and then mlflow is a sort of layer on top of that. So you create an experiment, run the code for each and then MLflow is just sort of there to help you compare the metrics. Is that correct or do you want to add some.

Yeah, I think for the machine learning experiments, that's indeed correct. Afterwards there's a bunch of stuff like saving the model, logging the one that performed best, deploying it. That's something that MLflow can help as well. So this workflow we're shown today is just focusing on the experiment, but that indeed would be the workflow.

Okay. All right. And so you mentioned deployment. I think that's perhaps something that's. This is mysterious to some people who haven't worked in a production environment. Can you just give me a bit more detail on, like, what you actually mean by deploying a model?

Yeah. So now we're just running a notebook, right? That's not something I can run to do my predictions every day. Or depending on your use case, it might work in a notebook. You might do the predictions here, but most of the times what happens is that you deploy your model somewhere. Could be as an API, for instance. So how I explain this in layman's terms would be that we could run this model as an API. I'm not sure whether the audience is aware of what an API is, but.

Yeah. Do you want to give a brief explanation?

Yeah. So what you do with an API in simple terms is that you host it on some sort of, what you could see as a website or a web service that I could send a call to and I get a prediction back. So what happens when I call an API is that, for instance, you're still seeing my screen. Right? That would be a bit too small. But an API could just be a URL that I call, for instance, datacamp API. Model. And then what an API call does is that I can send parameters with an API call. And a really simple version would be that I just sent 0.1 as a URL call and it would give a prediction back. And in a browser, that would just be like a bunch of text, but normally that would be returns in a JSON. And so that's what also would happen in this case, that we deploy it, for instance, with the cloud service like AWS or Google Cloud or Azure, and we run it as a web service that we can call using an API call, send data to it. For instance, this is a very simple example that we only send 0.1 within the example that we have here with the weather data, we can send the. The cloud and the radiation and that kind of stuff. And then it predicts based on the Data, the mean temperature. Does that make sense?

Okay, so the idea is you throwing in, giving it some data, some kind of request, is running the model, maybe making predictions for a specific date or whatever conditions, and it gives you the answer back. So in this case, would you be using ML Flow to create this API or would you be using other tools?

Tools, Yeah, I think MLflow is capable of doing it. That would be a tool that can do it for you automatically. Yeah, indeed.

Okay. And I guess another question I have is that how does it work once you go beyond working on your own to working in a team? How do you do collaboration on experiments?

That's a good question. So now we have this mlflow here in our notebook, right. Where it just saves it sort of locally here as files. But what you normally would want to have is that you have MLflow somewhere running centrally. MLflow also has a really nice UI. And with centrally, I mean that it doesn't save it here in my local notebook file, but it sends it on some web server where MLflow is running. So instead of having then the results local here on my laptop, I would have it in the web server where MLflow is hosted, such that we can all work together on experiments and see the results there. And then mlflow, like I said, also has a ui, just something you can click. Like a data camp also has a platform, and an MLflow would be the platform where you click through and you see the different experiments.

Okay.

So you're perhaps getting your data infrastructure team to set up some central ML flow, and that's going to be available for everyone in the team.

Okay, yeah, indeed.

And then also what happens? So, I mean, just in this one example, we've got a single experiment. I imagine if you've got a data team, you can be running lots of experiments all the time. You could quickly grow to having hundreds, thousands, maybe even more experiments. There's ML Flow sort of a scale to this. What, like, how big can you go with this?

Yeah, that's a good one. I. I would assume, but it's an assumption that MLflow is capable of like. And the web server you're hosting mlflow on would need to be scalable, such that it can run multiple instances of MLflow, such that it's easily can. I think it's a scalable solution. Yeah, that's my short answer.

Okay, all right, excellent. Yeah, maybe get someone from ML Flow to figure out the exact details there. All right, yeah. So my next question is for people who are wanting to get started with experiments, like, what's a good first experiment to run.

Yeah. We just say go do it, go try different stuff. Take some dummy data. There's tons of stuff on Kaggle, but also on Data Camp that you can use and try to see if you can make it as structured as possible so that, that if you want to turn the knobs to get the best model that you can easily turn those knobs. So for instance, the function I created here for doing the feature pre processing, that's a really nice one to have because you can easily rerun it with different feature selections and such. And so if someone would be starting doing experiments, try to set it up in a way that you can easily run a lot of experiments at the same time to get, to get good results. Yeah. Does that answer?

I think so, yeah. Is this sort of example of having just like a couple of different models and comparing the performance? Is that like the sort of the simplest kind of experiment you can do or what. What is like what's the easiest possible thing you can get value out of ML flow from?

Yeah. So now what I just said as well is that we're really focusing on just the model and the parameters. But of course there's also a lot in data and data quality, like data is often the first step where you start improving your model because if you have bad data, you can have the most fanciest of models, but it's not going to work. So I would say start with the data first, making sure you have good data, looking at the correlations and seeing, making sure you don't have a lot of missing stuff and seeing if the correlations make sense. Because mlflow you can also. But that's something we didn't touch upon today. But you can use different log different data versions, so I think that would be a good place to start.

So try running the same model on two different versions of a dataset.

Yeah. So taking all the knobs you can turn, like the data, the model, the parameters, make sure you can tweak them easily such that you can see from running multiple experiments where the best results come from. And then like we had in the first question, iterate from there on what the next step.

Okay. So I'm also curious as to whether this works for things like machine learning computer competitions. So I think, well, data camp competitions are generally structured slightly differently, but for like Kaggle competitions, where it's all about can I get the best possible performance?

Yeah.

And like most accurate predictions, would something like ML flow and running experiments be useful for trying to win those competitions?

Yeah, Definitely. I mean I remember someone that started on Kaggle competitions when Kaggle started and it was really hard for him to monitor all the results he got. And this is like a great solution to see where the best results are and then zoom in on how I can tweak it in the best possible way.

Okay, cool. So for anyone who's interested in competitive machine learning then it sounds like this is a very useful thing. And beyond that for careers, what sort of, what sort of companies, what sort of industries tend to use this sort of techniques? Like who's doing machine learning experiments?

Yeah, I would say that definitely like tech heavy companies are using this. So I mean you would probably have multiple machine learning use cases if you want to use these kinds of tools like MLOps becomes interesting once you have multiple use cases working on it with multiple people. So I think like really tech heavy companies, data intensive companies that would be really interested in using these kinds of.

Okay, could you maybe give me an example of like some business use cases for these sorts of experiments?

Well, yeah, there's tons. I mean you have this big Dutch company called Adyen who does a lot of machine learning in fintech use cases. So for instance detecting fraud, that's a clear business case. I mean in retail I've seen usage of machine learning in predicting how well a product is going to sell and then doing different machine learning models for different categories of products that would all be use cases of machine of these kinds of.

Okay, so it's really pretty broadly applicable across like a lot of different industries then.

Yeah, but then, then still your company needs to be quite far in its, in its data journey, so to speak. So, so a company that's just starting out, seeing, getting insights from their data, doing some data analysis, maybe having dashboards would probably, or maybe even experimenting a bit with machine learning wouldn't yet use a tool like this I think.

Okay, so there's a sort of order to this. So you need to get like maybe your descriptive statistics done first in your dashboards and then a few tried just one off machine learning models and then you get onto experiments later.

Is that the flow? Yeah, I think this becomes interesting once you have at least one use case in machine learning and you have two or three which are doing multiple experiments for then it becomes interesting to use this.

Okay, and it seems like the start of this code along was all about processing your data, making sure like all the dates were in the right format and things like that. So what do you need to know about data quality before you start doing your experiments?

Yeah, I think just like I said, like, data quality is often the most important thing in doing the experiment. Again, bad quality means a bad model in any case. So having high quality data and making sure you understand it is really important in improving your model. I mean, you can run tons of experiments with tons of parameters and different models, but if the data is bad, the data is bad and you're going to get a bad performance. So I think the data is key. And we're also seeing in the past couple of years that there's way more attention in getting good quality data. Because we all say there's big data and there's so much data, but we don't know if it's relevant, if it's good enough, if the quality is high. We just know there's a lot. And that's also something I hear at companies often as well. Like, we have so much data, we can get so much insights out of this. But that's often not the case because the quality of the data needs to be good. So knowing how to load data, also how to, for instance, impute it like we did and scale it, those are really important skills, I think.

Okay, Jermaine, you want to talk a little bit more about imputation then? Because we glossed over that like fairly quickly in this, but it's incredibly important technique. So do you want to talk about, like, what imputation is and why it's important?

Yeah. So what you often see and in the data set we use, it's quite, the data is quite okay in terms of missing values, but what you often see is that you miss values. And this can have numerous reasons in different use cases. But imputing means that we can, or what imputing results in, and that we can use more data. So instead of a way of imputing or a way of handling missing values would also be to just disregard all the rows that we don't have values for. But that means that, for instance, if we throw away all the rows in this data set that we didn't have to snow depth for, that would mean that we throw away 10% of the data. And so imputing it means that we just find the value that we expect to be the row to be alike. So for instance, with the mean, we just take the mean of this value or of this column to fill in the row. And so we can still use the rest of the row and not throw it away because it's missing, but still uses data.

I do love that is like you basically you're making up data, but you're doing it in a sort of a scientifically rigorous manner. So it's not just I've randomly generated some numbers. Okay, two more questions then I think we'll wrap up. You also briefly talked about data leakage. I have to say for people who are doing the data camp certifications, not understanding data leakage is one of the big failure points. So this is quite an important thing to understand. So can you just talk me through what is data leakage and how do you like when does it occur, how do you avoid it?

Yeah. So here we do the imputation and the scaling and for instance, when we use the impute with the mean, what we actually do is we take the mean of this column to fill in the values that are missing. Right. And so if we do this for the complete data set before splitting it, it means that the, the test set sort of knows the same thing as the train set but they need to be separated because you want to train your model on or yeah, maybe to give a sort of a more intuitive example, if I want to learn myself something, I'm going to learn all these examples that I'm going to learn from what the relationship is between the variables. And then for me to, if I make an exam and I already have seen all the questions and the answers before, then I'm going to make a perfect exam. But it really didn't really test my skills. So I think that's also a good example of what happens when you impute on the whole set. You're sort of giving me already the questions and answers that I'm going to be examined on, which means there's data leakage.

I really like that analogy of just looking at the, the answer page to, to an exam before you start taking the exam. Yeah. So I think, yeah, the secret is really just split into your training and testing sets right at the start of your workflow. That actually makes it clear why you've got this pre process data frame code as a function. Because you've got to run the same code then for your training set and your testing set. So you've got to do it twice, I guess. Guess that's why I use the function there.

Yeah, yeah, indeed. And also to like here it's nice that if you want to experiment with this, easily rerun it instead of having to change the whole code again.

Nice. Okay, so last question from me. So we're focused on machine learning experiments today, but there are all sorts of other data experiments. So perhaps A B tests are maybe the most popular kind of experiment. Can you use the same sort of tool set like mlflow for a B test and other data experiments, or is it just for machine learning experiments?

Yeah, thinking out loud. Like, you can use different data versions in this experiment, so you can use MLflow to do different you different versions of the data. So that could also include that you use one set of data in the A test and a different set of data in the B test to see how well the model does. So I think you could maneuver your way of using ML Flow for a B tests. Indeed. Yeah. Yeah. But I'm not sure whether it's really set up for that task.

Okay, okay. Maybe use a dedicated tool, but then. All right, so we've got one question from the audience. Before we get to that, I just want to say we've got loads more. We've got three more webinars this week, so I'm going to be back again tomorrow for a webinar on all the announcements that OpenAI made last week. So you're going to find out about GPT4 Turbo, custom versions of GPT, the Assistance API, all the good stuff that they, Sam, Alton and the rest of the OpenAI team announced on Thursday got a session on revolutionizing decision making using AI. So like just going beyond like using data for stuff like decision making is like the big new frontier that's definitely worth turning up for on Thursday. And then on Friday we've got another AI sessions. This is on fine tuning your own llama 2 models. So llama 2 is one of the, the, the GPT rivals is like, it's a very big, large language model. So loads of stuff for the rest of the week. Please do come back. I hope to see you again in future sessions. And now we have a question from Javier. So Javier, you'd like to know about the pros and cons of ML flow versus dbc. I think DBC has data version control, if I remember my acronyms right.

Oh, that's something DVC I haven't heard. Okay. Does he mean dbc? Maybe.

Well, maybe it's worth talking about. Like, are there any alternatives to ML float that are worth knowing about?

Yeah, there are indeed. I mean, there's. Each day there's new MLOps tools coming out, so there's tons of alternatives there. I think mlflow is one of the bigger ones. There's also NEPTUNE I know of and a bunch of others, but not on the top of my head that I know.

I mean, I think certainly for experiment, tracking weights and biases may be the most popular. And then There's Comet ML, I think is another one. There's a few of them.

Yeah, yeah, there's loads. There's loads. If you, if you look for it, MLOps tools, I think it's, it's, it's growing every day.

You're probably gonna have to, if you want to buy one of these things, do your due diligence and just like see if the, the features that meets your needs. But mlo, it was free in this case, actually. Is there a paid version of ML Flow? I'm not sure.

Maybe for deployment. That's often what these products do is that they have a free open source version. And if you want to, like for instance, with hugging face, you have a free version. But if you want to deploy it, you can pay for it. But I'm not sure whether that's the case with MLflow. It seems all open source.

Well, anyway, you managed to use MLflow for free today, so it's probably a good place to get started.

So.

All right with that. We're at time, so thank you once again, Volker. I was really nice just like seeing like what the Flow is for running a machine learning experiment. So excellent stuff. Thank you to Rhys for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. Don't forget I want to see you tomorrow for the GPT4 Turbo stuff. It's going to be a lot of fun and hope to see you all again in future webinars.