Introduction to Experiment Tracking

Foreign.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'm going to be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has the chance to join. In the meanwhile though, we'd love to hear from you, so let us know where you're joining from using the chat or the comments, depending on where you're watching from. And yes, maybe tell us something that you'd like to get out of today's webinar. We're going to be using dagshub for the code along today, so sign up for an account. I've put a link in the chat so please go find that. I will also be reposting it for anyone that's missed it. And we'll also be using Google Colab today, so if you don't have Google Colab, get signed up for that as well. But again, I will be posting links to these again. I'll be back to repeat these messages for any new joiners shortly, but until then, enjoy the background music.

It. Foreign.

And thank you for joining today's webinar. My name is Rhys and I'm going to be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join. In the meanwhile though, let us know where you're joining from and yet tell us something that you're looking to learn in today's webinar. Just a note, we are using Dags Hub today, so please sign up for an account there. And we'll also be using Google Colab as well, so if you haven't got access to those tools, please get them sorted now. I will be posting links to them in the chat and the comments shortly. I'll see you again shortly.

Sam. It.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator for today's session. We've just been waiting so everyone has had a chance to join. So nice to see that everybody is joining from a variety of locations. Lots of people from lots of different locations today. Quick note from me before we get started. We're going to be using Dagsub today as well as Google Colab. I posted links to those in the chat so get signed up to both of those before we get started so that we can all code along together. Brilliant. I think that's about it from me today. So I'll hand you over to your host for today's session, Richie. Richie, please take it away.

Hi, everyone, this is Richie. Thank you for joining us. As we said, it's really amazing to see everyone from around the world. I just noticed someone saying they're from Middlesbrough. That's just down the road from the town I grew up in. So hello, Olia, and hello to everyone else. All right, so we're talking about experimentation today and experiment tracking. And. And while we often think of experiments as being a thing that only scientists do, actually every business needs to perform experiments too. And a common example of this is A B tests for improving websites and for improving products. And just as a consequence of this ubiquity of the A B test, more or less, every data scientist needs to have at least some understanding of experimentation. And an extra complication is that once an organization is running lots of different experiments, it can become tricky to manage them. And if you can't manage your experiments, you're going to get the wrong answer, they're going to get confused, and that's just a disastrous situation where your data science is all nonsense. So today we're going to learn about experiment tracking. And our guest is Jinen Setpal. He's a machine learning engineer at dagshub and Jenan has worked extensively on challenges in natural language processing and more recently, machine interpretability and Dagsub. His work involves making it easier for everyone to systematically develop and iterate on their MLOps projects. So I'm going to hand it over to you, Jenin. Take it away, please.

Awesome. Thank you, Riz. Thank you, Rishi. That was awesome. And I think Rishi started off by summary summarizing the entire challenge with data science in general and experiment tracking very, very beautifully. The primary goal within every single organization's data science team is to ensure that everything that they work on is managed and can easily be accessed by someone with no experience in programming. You should not have to open up a terminal window to get started with tracking your experiments or identifying something new or something different. Mflow. It does a fantastic job in enabling that and I am super excited to take you all through that entire journey from zero to get started with tracking experiments using MLflow today, just to get started. Can everybody see my screen, yes or no? All right, cool. So awesome. Thank you. And this is the collab notebook that we're going to be using today and basically getting started with everything. So I believe that the link has been posted in on the chat as well. So if you would like to follow along. Please do so. They will be using DAX Hub for a part of this workshop because it provides a remote host for mflow and we will cover that today. But to get started, MFlow we integrated MFlow into DAX Hub almost two years ago now and we provide a zero configuration MFlow remote with built in access controls like MFlow's tracking registry and deployment functionalities. We have handled many of its specifics and have managed to ease that process as much as possible. So we are hoping to help you guys also leverage the zero configuration functionality of the MLflow server. Here's what we will talk about today. First we will discuss about why we need MLflow, what MLflow fundamentally is, and then we begin deep diving into the attracting functionality. We understand their nomenclature for runs and experiments, logging runs and experiments. So actually doing the practical logging of the experiments and the runs and understanding how and where runs are recorded. Finally we go hands on with using MLflow with a sample project that we've adapted and hopefully you guys would also like to follow along through that. So to get started with why we really need MLflow, there are these three means that illustrate the idea really nicely. Data cleaning is a significant part of machine learning, especially with large language models. There was this paper called lima, which is less is more for model alignment and they used 1000 fantastic data samples and were able to perform at a level that was possible at par with GPT4 on the task that they align their language model to. Mastering. Data cleaning is a massive part of the machine learning process. The effort and the time that is spent towards things like this is often very very underestimated. And doing things like hyperparameter Turing manually is a major challenge because it means that that you would have to undo redo a bunch of times in order to identify what hyperparameters work well. And if you are trying to simultaneously optimize a bunch of parameters, that makes the challenge exponentially harder. Before I used MLflow and before I really knew about MLflow, my colleague used this notion table to keep track of all of the experiments that they conducted. This is a table that contains research logs. There are details about the experiment, stuff like parameters, batch size, the epoch, then the status of the experiments, information about the data set, the outcome, and just general comments. Finally, there's also a log file that contains all the verbosity that you are doing. So that's the general idea with regards to the experiment. However, you can pretty easily identify that why notion is a fantastic tool for just keeping random notes, but when those notes become highly numeric in nature and require you to perform a lot of queries over them, it becomes significantly harder to be able to use it at scale. One way to look at mflow is like a notion for machine learning experiments. It is an open source tool to manage the entire machine learning life cycle. It supports the live logging of parameters, metrics, metadata and artifacts when you run any machine learning experiment to manage the post training stage. It also has a model registry with a deployment functionality to custom serve tools. Today we will be covering the live logging functionality part of MFlow. It was created to reduce the complexity in monitoring experiments is the reproducibility of results, which today is especially important because results conducted badly can have lasting consequences on huge populations depending on the impact of the machine learning model. And finally, it also caters to a need of standardization for registering and deploying models into production. So here's a little bit about the organizations that use and contribute to MFlow. The really cool thing about this or about MFlow in general is the fact that it is open source. And what that means is that all organizations can contribute and use it completely freely. And initially a lot of companies developed their own tools for model and experiment tracking just because it was so important. But all of these developing it themselves was no match for the community. And when databricks introduced basically was adopted by a lot of them, they themselves ended up contributing to it, which makes it even better. And that's kind of the. I don't know, I love open source and I'm super, super happy that they're successful because of that. So there's that MFlow has a couple of components. The first one is MFlow tracking which is the locking of parameters, metrics and artifacts. MLFlow projects, which is packaging and reusing data science code. The MLflow registry, which manages the lifespan of a model. So taking it from the experimentation stage to deployment to finally retiring, retiring it once you develop a new state of the art model and MLflow models, which is just packaging and actually deploying MLF models that are deemed to be be in production. So let's get started with the MLflow tracking functionality. There are runs and experiments and these are two very, very important components within MLflow. The experiment unit in MLflow can be considered as a project or an approach. For example, hyperparameter tuning the learning rate of the model. This term, it is kind of an umbrella term and houses multiple runs in and you can house multiple runs in a single experiment. So for instance, if we were looking at optimizing the learning rate of the model, we could say that we have one run with a learning rate of 10 to the power, negative five all the way to one. And we basically compare and see how the model is trained with each of the supplied learning rates. And more than one run may be associated with a single an experiment. And each run, like the individual run itself, is a single execution of your data science code and that records the following information. It records the source of the execution. So it contains the commit hash if your code is tracked, reacted and the original line of code that was utilized for the run. So if you use Python train py or Python source train PY that will also be logged within the run. It also logs artifacts. These are output files that are recorded towards the run. So if you're running an evaluation stage, the images that you generate that have maybe the graphs that show how well the model performs on unseen data, that could be an artifact. The actual model data itself could be another artifact. Parameters. These are stored in the key value format and they are basic, they're the hyperparameters that are used in the run. So learning rate is one of them. The type of loss function that you use could be another, and so on and so forth. Finally, there are metrics. These are used for evaluation like root, mean square, error or roce oc. These are an accordant through the extent of a run, so epochs. So every single epoch itself also counts as metric and it's basically any data that updates through the extent of a run, while parameters are data that do not update through the extent of a run. So if you have a variable learning rate, that would become a metric, not a parameter. All right, so that's the runs and experiments and I hope that makes perfect sense. And if it doesn't, please shoot questions in the chat and either Rishi would let me know and we can resolve that as soon as possible. There is that. Next we will talk about how and where runs are recorded by mfl. MFL runs can be stored locally in files, either remotely in a tracking server or inside a database that is compatible with SQL Atom, that is an open source SQL toolkit and object relational mapper for Python. The first scenario or the default scenario that MLflow uses in order to set up its runs or experiments or both, is the completely on local host scenario. What it does is it creates a directory called mlruns inside your local system. As soon as mflow logs a run or an artifact, then you are able to view view the artifacts using the MLflow UI command. It will bring up a web page and you can browse all of the experiments along with the graphs. Through that it's a good starter mechanism where everything is handled completely locally. However it performs or it gives several restrictions on the actual collaborative aspect of the experiment tracking. So if someone wants to load at the experiments that we have conducted, it becomes extremely hard because it only exists locally inside your repository and this also cannot be tracked via git because artifacts generally are significant in size. So you would probably want to use another versioning tool for that and that complexity scales up very quickly. The UI itself contains the following features. It has experiment based run listing as well as comparison. You can search for runs by parameter or metric value. You can visualize metrics and finally download results from the run as well. The next scenario is MLflow or Localhost with SQLite. The only difference between this and the previous one where it stores stuff on localhost, it still is on localhost. It's just that it is using a SQL database instead of just your file system. So it is a little bit more efficient in terms of the way it stores the data. The next scenario is MLflow on localhost using a tracking server. This scenario again is pretty similar to the first scenario, but over here you set up a remote server using MLflow server and that starts a tracking server at port 5000. This tracking server is continuously listening to any HTTP request that it gets in order to log experiments. Everything is basically proxied through that. Now the next scenario is MLflow with a remote tracking server, a backend as well as artifact stores. The tracking server backend store and artifact stores are all located on different hosts in distributed architectures. So it is the most efficient possible. I'm just going to make this screen a little bit smaller so everything fits into the single image. But the setup is as follows. This diagram describes it pretty well. We have your localhost that actually runs all of the code. You have a remote host that is a tracking server, so you can have a separate tracking server that unifies with all of the projects that are running at a simultaneously within your organization. You can have a separate S3 bucket for actually storing artifacts as well as a PostgreSQL server to actually query any of the metrics that you are using. This allows you to perform a lot of very efficient optimization routines as well. For instance, your images or artifacts. Generally if you store it into a SE remote host, that is much better than storing it through SQLite or any SQL server. In general, it is a distributed architecture, which means you can scale different components differentiably depending on the actual requirement that you have at hand and the one caveat with this is the fact that it requires you to set up. It requires you to set all of these up differently and link them as well. This also means that you will will require to have some DevOps knowledge beforehand. Additionally, since both of the S3 and HTTP requests are sent through the single local client, it also requires that entire setup to be done client side as well. So that is the caveat. Two caveats of this scenario. Scenario five is the MLflow tracking server enabled with a proxy artifact storage. In this case we eliminate the need for directly making S3 requests to the remote host and instead proxy everything through the HTTP server wherein if we are using or sending models, images, configs, anything that is to be sent through S3, it is proxy through the HTTP host and the remote host itself manages the S3 deployment. This eliminates the second caveat, however it still does maintain the first one wherein DevOps strategy is required in order to use MFlow. That's the best performing scenario that we have that MFlow provides so far because it is the most scalable and works well as well. So it is the most efficient. So removing that last caveat from what we have is the scenario 6 which is Mflow xtaxel. So going through all of this could be a bit of a hassle even for people who do have a DevOps background. To simplify this, Daxlab decided to do all of the MLOps HeavyDistring for you. We provide a free MLflow server with every single repository. You can log your experiments with MLflow to it, view it with all of its information under the Experiment tab, and manage your trained models from the full fledged MLFL UI built into your Dagshub project so you can access MLflow as though you are setting it up locally through our client. When you create a repository on DAXA, a remote MLflow server is automatically created and configured with the project. The repositories mflow tracking server will be located at the taxa git repo and you just add mlflow similar to how you would identify a git host with the git added to the URL you would simply add MLflow and that gives you the MLflow remote. That sums it up for the theoretical portion. Now we can deep dive into the actual programming part of this. That's fantastic. At this time, if there are any questions I'd love to take it. And if not, I guess I hope you all have your notebooks up and running. And yeah, we can start off with the hands on.

Yeah, not many questions yet. If you do have any questions for Jill, please do ask them in the chat now. Otherwise I think we're probably okay to get started.

Fantastic. Okay, sweet. So to get hands on, let's start off by forking this project by Eric Robinson. He trains a neural network to classify images as either Mario or Vario. So I forked that and trimmed it, made it, just optimized it a little bit for the purpose of this workshop. And if you press down that URL, it would be redirected to the Mario vs. Wario Git repository. So from there, just for that, to your own personal account. You can rename it if you like, but I will just keep it the same and from there you should be good to actually start working on integrating MFload into an actual project yourself. So I'll wait for this to process and in the meantime I will install taxhub's client repositories and add my information. So feel free to do the same for yours as well. This is you would want to add your Tax app username over here and the email that you use to sign up with Tax app over here and the DAX hub repo if you changed it, you could change it if you didn't keep that the same. Once that is done, we will generate an OAuth token for for improved account security and Fantastic. It did not request me to authorize it separately, but you should get a link over here which you press and put redirect you to a DAXA page. Once you press authorize that should set it up for you. Fantastic. Next we have this actual repository. That is the base repository on which we were going to be working on. Now that we have that in our DAX remote, I'm also going to clone that locally. I will run our first configure git and then clone that repository. I already have the repo clone. Fantastic. That is that once that is done we can actually install MFlow, which is what we are here to work on. We have two flavors that MLflow provides. The first is the default one and the second is MLflow Skinny. So it reduces the set of dependencies that MLflow uses and cuts down also on a couple of features, but it is good for training on an edge device. For now we are not on an edge device because Colab provides us with a lot of hefty compute and a nice gpu. So I just install the default version of Microsoft for now. All right, fantastic. Next we'll explore the actual project itself. So let's actually understand what's going on before we just start executing random code. We have a couple of stages over here. There is a data stage or a data folder where we have all of the data for our experiments. There is a configuration file as well as a training file that are both fed as inputs to this training stage. And this training Stage runs the Python 3 source train Py and generates out our models for us looking at the data that we have. Very quickly, let's just take a quick look at what we are actually working with here. The images are taken from a. A video game and they are completely in black and white. Hopefully this opens soon. I'm not having the best Internet today. All right, so in the meantime, let's just run the chain py, because that was the pivotal script that generated the code file system is already mounted. Oh, it's running Installix again. All right, just a second. If you run into this, which I do not think you will, you would just need to remove the. Where is this at? Trained by line 86. There you go. I will just comment this out. This is for the file system streaming to ensure that we don't have to pull all of the data from the remote again. Once I remove that, that should work just fine. It begins the actual training process. This manage to load. We have two distinct folders, one for Mario and the other for Wario, and they each contain images of the images of or screenshots from the video game. If that ever opens up, I will make sure to show you guys as well. So we completed our training. It's a binary classification task, so we basically want to find or identify if a single image contains Mario or Wario. So we have all of these images and I will pick one at random and hopefully hope that looks good. But yeah. So the training completed, we ran only five epochs and we have an accuracy of 0.64%. So that's not fantastic given again, it's a bilateral classification task, so even if we guess that random, we would get about 50, but that's fine for now. Now let us sort of work or begin to integrate mflow to this project that we have. The first is locking experiments locally, right? So in order to do that, we start off by actually importing mflow into Train py. So I will open Train PY and remove that for space and actually import mflow in here. And in order to. So in order to create an experiment, I need to use this command from the Command. This command from the. If I'm using the mlflow Cli but if I'm using Python, I can use mlflow createexperiment and give it a name. So I created the Simple Helper folder on the Simple Helper file that gets an experiment id. We basically give it a name as the input and it gets experiment by name. If from the mflow either present in localhost or the remote. And if it does not exist, then we receive none. And if it is none, we want to create a new experiment and return the id. And if it does exist, we just want to return the experiment id. And that's basically running right here. So if I run this very quickly, I will get zero, that is the new experiment ID that I have gotten from the creation process. Next, I can allocate a run to this actual experiment itself. So if you remember from before, the experiment is the large umbrella that contains all of the runs within it. If we want to add a run to a single umbrella experiment, we want to use this API function where we just provide the experiment ID right here. This experiment ID is what we have gotten from this function. So in order to start an MLflow run, we use the MLflow Start Run Python API call and we paste this into our train PY with the experiment ID that we obtained from the last execution. And importantly, we need to provide a context to the MLflow start run, so you would not be able to run it outside or you should run it within the with the block for context. Finally, let's discuss about how we can actually log information. I'll cut this out and add it back again once we need to actually input stuff. But yeah, here is how we start or actually log information into MLflow. We will first import MLflow to our notebook or by file. Then we use manual logging commands to log all of the features that we discussed earlier. So parameters, metrics, artifacts, and just general text. So for a single parameter, we set up logparam or use a logparam and provide a key value pair and it logs that to the relevant run. If we want to log multiple parameters in one shot, we need to provide a dictionary of key value pairs and that will give us the final set of parameters in one go. For metrics, it is much of the same. We provide a key as well as a value pair. We also optionally provide the step argument, which is basically be the point on the x axis for the graph itself. So if you have 10 epochs, your first epoch will be step one, and so on and so forth. For multiple experiments, it is much of the same. You would provide a Dictionary of key value pairs and optionally a step keyword argument for the position on the X axis. Next, we have artifacts, so you can log a single artifact by setting the local path of the artifact itself. For text, you can use the log text functionality. Provide a string of information that you want to include and a local path for a string. This is the actual file or the text file that is generated and the text is stored into that text file. That is the basic logging capabilities. MLflow has a bunch of others. If you would like to read more, Please refer to mFlow's tracking docs. Recently they also started experimenting with LLM and prompt tracking stuff for prompts. So that might be something to check out for the future as well. All of the MLFO code in the session should be under the context block, as I mentioned earlier. But this contains the actual process of logging all of our things. So I'm going to go ahead and copy all of this very quickly and go into train and before that actually I will import from source. What was it? Get or Create. Experiment? I think it was the Get Experiment ID function. All right, so I will log this helper function as well. And let me expand this a little bit and go down to the actual training part of it. So this is the. This is the driver for the code itself. We have a Get data set function that gives us all of the data sets and creates data loaders out of it. With the batches of 22 and a validation ratio that I cannot see, it's 0.2. That is that we have the get model function and that gives us the model itself. We then fit the model on our training data, compare it against our validation data for five epochs, and then we set up the evaluation part of things where we evaluate it on our test set, get the loss accuracy and report none of those actually, and print the final model. So that is that we have the. I mean, oh, it is printed during the model evaluate part. So that's that. All right, I stand corrected. Anyways, here is the actual code for the logging of the experiment. So I am going to use the Start run functionality for Experiment id. I'm going to get. I'm going to call the Get Experiment ID functionality function from the. From this Python file and provide it the name. So over here it's Mario underscore vario. Fantastic. And with colon paste. Sorry, I will return it back to the normal size in a second like that and paste that right here. Make the text visible. All right, there we go. Okay, now from here all I will do is make sure to align this properly. And that should set up all of our logging here. This sets up our actual parameters and the metrics. Metrics we actually do not have at this time. So I will move this entire bit to the bottom for after we complete our training as well as evaluation. Sorry, just a second. I don't see the evaluation code one. I'm just going to reset it. Very, very sorry, but. Source, I think I messed something up. Yeah, okay, there you go. I did actually remove that. So this is here and I need to perform the imports again. Oh no, I made the same mistake as earlier. All right, awesome. Now fix this and fix imports. All right, that done. I will run Train py. Oh, I need to comment it out. I forgot.

Give us a quick summary of what you updated there, just so everyone's clear on what's happening.

Absolutely. I'm sorry, getting back from the Beginning, we imported MLflow to actually use the package and imported the get experimentid function from source. Get createexperimentid py. This is what we had used initially to get the ID of the experiment that we need in order to define where the run actually goes. So which experiment does the run actually go to? So from there we set up the entire training process that we already had from before and start the logging part of things. We use the start run python function get experiment ID and we get Mario Wario's Experiment id. Then we log all of the parameters. So we have the single parameter logging, the multiple parameter logging, the single metric logging, and the multiple metric logging. Then we also log the artifacts. So the artifact is what we got after the model save is complete. And I realize I need to put this up here as well. So we have this and now everything should work in the correct order. We have the image size, the learning rate, as well as the epochs that we are logging. We log the test set loss and also the test set accuracy right here within our metrics section. Next we log the artifacts and the general text, so general information about the run and we save it in runinfo.txt. so that's the general set of changes that we made. It is exactly the same as this under the with context block at the end of the entire train PI script. So that is the extent of the changes. I'm sorry about being confusing. Now if we run Source py, we should be able to train everything as before. But now, in addition to just training it, we should also be able to see the result of our experiments that we Drive within the 0 directory under mlrun that contains all of the actual information about our training run. Assuming the experiment ID do not update in the last two times, we should be able to refresh this once the training run is complete and identify all of the logs and everything that we have actually identified or actually logged using mflow. It will follow the structure that we have. Over here we have the the run hash that we get from the model run. Then this is within the actual experiment id. Within that we have the artifacts, so models as well as the run information, the general info that we included, the metrics, test set accuracy, test set loss epochs, image size, learning rate, everything that we had included from before, and tags which is metadata about the actual run itself. So the head of the command that was used in order to track the experiment, the source, the experiment name type, and the user that tracked the experiment. So this says that it is done. I will refresh it real quick and check for the updates. Okay, what happens if I run get experiment ID for this? All right. I'm not sure why I can't see that, but I'll move forward for now. So that is that now this basically saves all of our data locally, but in order to avoid the process of setting up the remote server, we will use Dagstub's integration with Mflow. When you create a repository on Dagshub, a remote MLflow server is automatically created and configured with the project. The tracking server is located at the URL. This is the same as we mentioned above, and in order to actually set it with our machine, all we need to do is configure set tracking uri function within MLflow and give it the remote host that we had mentioned earlier. We can set up our credentials as OS variables, and just like that we are ready to start actually logging experiments to a remote host. Setting up the environment variable. All I need to do is execute this cell. We set our tracking username as the dagsub username, the password as the dagsub token that we obtained during the OAUTH process, and the mlflow trackinguri. Normally it is not recommended to set the tracking URI as an environment variable simply because it changes from project to project. However, since it's just one project on our notebook, that's not a problem. From that, let's update the get or Create Experiment PY folder in order to actually integrate remote experimentation. So first I'm going to import OS because we want to get the OS variable for MFlow tracking URI and copy this line of code that will actually set the tracking uri Right. It's the Python function which we provide as input the remote UR URI that we got via Taxa. Now once we run this, we can go to our actual Taxa page and open the experiment session. And this experiment section should now contain or be populated with a single experiment that is completely empty and contains no information. So let's check that out. Oh, I guess that our experiment from earlier was actually logged to the MLflow mode. But yeah, I will check that out. That kind of explains why it was not present locally. All right. Anyways, so that sets up a default experiment. Since the experiment already existed, we do not see anything blank and we actually saw the populated version of a SET experiment. Let me see if I can. Yeah, break this. Now should be empty and from there I should be able to run this again and hopefully we should get an empty mfx. Yep. Okay, so that is that. And in order to actually begin logging our experiment to the remote post, like we saw earlier, all we need to do is go to train PY and do the exact same thing that we did earlier. Where is this getter create experiment? We take the SET tracking URI code and paste it within the runner for the script. And now it will log all of those metrics that we had earlier into the into the MF flow into the MLF remote that is present on the tabs. And with all of the Oops. I need to import OS with all of the added functionality and optimizations for performing machine learning experiments both collaboratively and at scale. So I will do that, run source train PY and it should get started. I don't expect the training to, but we should be able to do it reproducibly. So that is our plus point. Now, final epoch. We're still at 60% validation accuracy. Okay, I expected it to not get better. I did not expect it to get worse, but unfortunately it did. Anyways, our test accuracy that we report is 0.55, which is a tad over rolling a dice. So it's okay, I will still take it. But if we go to our MFL remote and refresh this very quickly, we should be able to identify the MFM experiment within this. Within this remote. Awesome. That is our actual run. We identify the fact that we do not get a fantastic test accuracy, but that is okay. Now we have covered the basic general understanding of setting up the logging portion of an experiment as well as logging it remotely. But there is still one very important concern. What if I forget something? If you're a forgetful human logger like I am, I tend to forget a lot of information that I added and is relevant for the training run itself, but I just did not fail to log in at the time and I'm also lazy so I don't know what to set it up manually. MFlow basically has set up this auto logging functionality where you can do all of the parameter logging automatically and and not have to worry about. Exists for a lot of frameworks like SciKit, Learn, TensorFlow and Keras, Gluon, XGBoost, LightGBM, Stats Model, Spark, Fastai, Pytorch. It's basically everything that you would need. You can use MLflow Autolog functionality by itself to enable logging for all of the supported libraries. However it is still better to specify the library manually and for now we are going to stick with that because it is a lot less volatile. I will add MLflow TensorFlow Autolog because we are using TensorFlow for this project right above the with Start run call and rerun all of our code in order to see the log metrics. So I will do that very quickly. I will go to Trainer PI, I will remove all of the manual logging that we have set up so this so we can like objectively compare the importance or the difference between logging manually and logging automatically. I will set the width call just before model fit and add or and indent this by 1. Once I do that I will use mflow tensorflow autolog and save this and that is all of the autologous configuration done. Now on top of this, if you do find there are some features that are not set up or identified by Manager Flow, you can manually go ahead and set that up and it will do it for you. I save this just to be sure and I will run this same training script one more time and hopefully it should not give us any issues. There are warnings, but warnings are not errors so it is no problem and that is training. While that happens, I also wanted to mention something. MFlow also places a focus on reproducibility of experiments and with every single run it actually identifies the hash that was used in order to run the code. This commit ID is basically just a piece of the information that was logged. You know what your code was at the time of running. If you update the code like create an additional loss function or something, you can specifically configure that into your code or know that it was that specific thing or that specific loss function that you added that caused the divergent behavior. That is also a very important reason to come in as much as possible and ideally with every Single run whenever anything is changed between runs, that is done. I am going to refresh this in order to identify the additional experiment that we had logged. Then we can compare those two experiments and see the difference between the logging capabilities that we have. Right off the bat we can see that there are a lot of things that we already missed. But if I just compare it very quickly that extent becomes all the more obvious, all receptive. No. 208 is the information that we had and enthused Grub is the information that autolog set up. We have this. There is image size and learning rate that they actually miss. I think LR is mentioned somewhere up top, but I'll have to check that in either case, if we find this information missing that we do want, we can definitely add that manually and that will also fall within the same run, provided it is within that context window. So we can also see some visualizations for the accuracy of the model, the loss of the model, how well it worked, and if we had two experiments that ran with all of this data, it could also overlay it on the singular graph so you would be able to compare that manually as well. Oh, so the images that we actually trained on finally decided to load up. Here are the Mario images and here is the Wario image. Wario looks way more menacing than that. Anyways, there is that. And now that we are done with this, I will commit and push the MF flow the changes that I have made to my repository and I should be able to check that as well in our actual Git part of the DAX app repository. And that is it. Congratulations. You can now integrate MLflow's experiment tracking into auto all of your projects. So that's all that I have for today. I hope you had a good time and were able to learn a little something. And if there are any questions at this time I would be very happy to take them.

Brilliant. We've got a couple of questions from the audience, we'll get to those in a moment. So thank you very much Jill. And that was really informative, like very cool sort of workflow there. Before we get to audience questions, I just want to say we've got a few upcoming webinars that you may want to attend or tomorrow we've got people from Spec Saves, the Opticians talking about how they use data in their company and how they've been making use of Data Camp. On Friday we've got a training on using the Chat GPT code interpreter for Data Science. And then next Tuesday there is a session on building AI skills with Data Camp. So please do sign up for those@datacamp.com webinars Jen, I also have a question for you. So a lot of the stuff you're doing today involves quite a lot of code and when you're running experiments there are often a lot of non technical stakeholders involved. So for example, like product managers probably want to see how their experiments perform, things like that. So can you just talk a bit about how non technical people can make use of the tracking features that you've been talking about today?

Yes, absolutely. If I can go back to my screen, we can basically set up the DAX app repository. Now over here, this is your repository and it is hosted by daxa. This is present on the cloud. In order to add someone like a project manager or anyone on your team that wants to access these experiments, they can simply access this through the link. If it is private, you can add them as a collaborator and they will be able to see it. And from there they can open up the Experiments tab. And through the Experiments tab they are able to pick out an experiment, for instance, let's say this one, and if they want to identify, I don't know, test accuracy for the last three, three epochs, they can do so by just setting up the filter for here and then they can make that identification. So if they, if there were a lot more experiments over here, they could work on it themselves and compare all of the experiments for themselves and identify which one works best. And accordingly, they could also use the mflow UI that is present directly in that's up. So I'll pull that up as well in a moment. But this relates to the model registry component as well. So for someone who is Maybe on the DevOps side of things or like the front end or project management side of things, who wouldn't want to really worry about all of the experiments, but would want to know which is the model that they want to pick up for actually deploying it. They could use the mflow Registry part of things and open the actual integrated mflow ui. Then within models they could pick out the model which we haven't talked and we will cover that within our model Registry part of things and set it up with a production stage or move it to the staging stage or maybe an R type stage, whatever is relevant to their problem and from there they are able to utilize that UI and basically pick out what is relevant for the management side of things.

Okay, so you've basically got like a built in dashboard there that's going to explain all the metrics and that's auto generated then. So you don't need to do anything to create that.

Absolutely. As provided that you're using the logging functionality. Everything from that is auto generated.

Super.

And just to provide a bit more motivation, can you talk a bit about like what are the most common problems if you aren't tracking experiments properly? What are the common things that goes wrong?

Yeah, of course I am myself. I'm pretty recent to using MFLO and stuff for experiment tracking. Before this I used to just completely use Jupyter notebooks for all of my experiments because they work so well. But one challenge with Jupyter notebooks is the fact that their execution order is not sequential. If you are working on a challenge, you do not want to be thinking about the fact that about the execution order of your notebook. You may work on it for say a couple of hours, save your project and the next day you open it up. And the execution order is not the same. You have multiple locations where you have defined the similar variables. You don't know which variable I actually accounted for the run. You don't know which one actually gave you the result that you wanted. And now you have to run those two things again or guess. And neither of those are ideal for science. Science needs to be as non guessable as possible. You have to be as sure as you possibly can. So mflow allows you to be as sure as you possibly can without having to take the tedious effort of making sure that everything works or is logged or can be ran sequentially. So that's really cool about it.

Thank you. And just one last question from me before we go to audience questions. Often a company is going to run lots of fairly similar experiments. Like think about, it's going to be a dozen experiments going on on a landing page. So is it possible to reuse code from one experiment to the next? How do you go about recycling things?

Absolutely. So suppose you were to say the one example that I gave from the actual workshop was the fact that you maybe want to change or test the learning rate of the model Linux. And that would be one experiment. Suppose you were to change that and say I would like to test the epsilon that I'm using with an add in in order to see how well that model performs. What you could do is keep all of the code basically exactly the same. You would open up train py and over here you would basically set up a for loop that iterates through a series of epsilons. And this experiment you would change it to say I don't know Epsilon optimization, and that will now change that experiment completely and you can test which epsilon works best for your specific purpose, and that becomes completely different. So you can reuse a large part of your code. The primary thing that you would have to change is your driver code, and that's about it.

Okay, so if you're running very similar experiments, pretty straightforward to reuse a lot of your work. Excellent. All right, let's get on to some audience questions. So, first one is actually, it's a request from Wallace saying, because you made some changes to the notebook during the session, just make sure those are updated. If people want to reuse your work, then they can do absolutely.

I'll just save this right away and make sure that if you refresh the original link, that should work right off the bat. The only main change that I made to the actual notebook was actually checking out the original training pycode. I accidentally erased training the model, evaluating the model parts of this code, and in order to get it back, I changed that part of it. But besides that, nothing actually is different. If you want to see the difference within the actual integration part of MLflow, the commit that I pushed should have that information there. That is not the default repository. I can type the link over here or post it in the chat, whichever is more convenient. But it is definitely this link that has all of the code or all of the updated code with the integration already ready.

Perfect. Thank you. Question from Alex saying, how big was the training data that you were using in the example?

How big was the training data? Let's find out. I guess. Four megabytes. Not a lot. Yeah, it's four megabytes.

Ah, very good. Okay, so we've got time for two or three more questions. So this one comes from an anonymous LinkedIn user saying, how can you save a custom Python model, not Scikit Learn or Pytorch using logging with all dependencies?

Okay, Right. Awesome. That is actually pretty awesome. Part of what MLflow allows, MLflow has this thing called the Model Registry that I touched a little bit on throughout this session. But the basic idea with it is that you. One of the functionalities that the MFlow model registry has is that it allows you to pick or create a custom pyfunc class within mlflow and integrate that reproducibly. So you would set up the INIT function of the class with all of the dependencies, loading anything or importing any modules and loading up the data and running the infrastructure difference code so that one function itself that would allow you to save or run A custom Python model that has already been logged and registered for actually logging and initially registering the model. As long as you're saving it to a local directory, you can use the save there functionality and save that directory into the mlfloor run or log into the mlfloor run. One thing to note, with things like wheel, like saving custom wheel, you can host that on a git repository or something and PIP can manually install it within your requirements txt file. You can just set up git HTTPs and the URL of the git repo that has the Python view file and it will install it from there. That will probably work.

Nice. And we've got. We're sort of at time. There's two more interesting questions I want to go through. So if we do this quickly. So one of these. Wallace again. So Wallace is asking what happens if you're running multiple experiments at the same time? How does that affect the tracking?

Oh, so that's. Each run creates a specific hash. So if you are using something like Ray that runs multiple, that performs hyperparameter optimization in parallel, by default, it will not affect it in any way because it will create a specific hash for each run and log all of the data to that hash within your MLFLoat remote or loop loss, depending on what scenario you're using. So it will work completely fine in parallel as well. You can run parallel runs simultaneously and find out what works best. Much quicker than just doing everything sequentially.

Okay, that's good to know that you can run multiple experiments at once. And final question, another anonymous LinkedIn user. So this is you covered using mlflow, you covered using dagza. So can you just explain where the similarity, where the overlapping functionality ends?

Awesome. So the thing is, I wouldn't say DAXA versus mfl, right? It's daxub and mflo. They're effectively the. We integrate mflow completely inherently. So it is not us versus them, it is us and them. Mflow requires you to set up, if you want to set it up scalably for either your own projects or for an organization, it requires you to perform a lot of DevOps beforehand. Right. For instance, since. Alchemy and the PostgreSQL server, all of these components sort of have to work together and if something breaks, you will have to fix that and your setup will not work correctly until you fix that. So that can either be managed by a DevOps personnel that is sort of working to ensure that nothing goes wrong, or you could create a repository on dagsub and we will make sure that that is not a problem at all for you. So it's just optimizing the MLflow process. Besides that, it is entirely similar as just using MLflow yourself.

All right, super. That's good to know. And it just seemed like it's a pretty powerful combination, the two tools together. Excellent. Okay, so thank you once again, Jenin. That was a really cool session. Very powerful advanced stuff being shown there, so I hope it's something our attendees can try out for themselves. Thank you to Reese for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today, and I hope to see you again in future webinars.

Thank you.