Machine Learning Automation: The Next Generation

So hi everyone, I'm Timothy Verdier from the Machine learning team and thank you for joining me to this talk about those functions in the Wolfram language that you throw garbage in and get probabilities out. So the point of today's talk is basically to answer every machine learning model question, which is why me and that's what I'm going to guide you through. For the supervised task like classify and predict self supervise, which is task in which you do provide the data itself is the untarget and unsupervised when you have no target like clustering or learn distribution. And so first of all, we will review together how a single model is trained and see how we can then automate the process. So when you, when you start a training for cluster classify with a given method, or with classify providing a performance goal, direct training, it returns a classifier function. And how it works internally is. It takes the argument you provide inputs in both case targets only in classify case case. And we will first extract some, some, some feature, engineer some feature. And you can see that it's, it's light blue, transparent blue, because this is data dependent. The feature extractors contain parameters that are trained on the data. It receives a target, a set of hyper parameters, so we call it configuration. And it has to train the parameters that correspond to this configuration, receiving the features and the targets. And when the trainer is done, then you get the feature extractor and the model wrapped into the predictor function. And when you have trained the model, then you can actually evaluate it by providing just the input data. And it's even simpler in this case. The input gets fed to the predictor function, the feature extracted with the parameters that are now fixed, fed to the model evaluated on the model with this parameter. And it return in all cases, it return in all cases internally at least a probability distribution from which you can compute decision or properties. Decision is the best candidates from the probability distribution in this case. And we all know that evaluation is a synonym of control. So know that you can evaluate your model, you can also validate it, that is testing it either if it's an unsupervised task just on some internal metric of the results, or if it's supervised, providing some ground truth data and calling for a metric which compares the target distribution expected guessed by the model to the ground truth targets. So you can compute figures like silhouette value for the clustering measurement or accuracy. And with all those elements together, we can start building what is known as automatic machine learning, which is a system that will also search for the best configuration, the best model according to your data, to your specific case. So as we need to validate the model in order to decide which is the best, we need the model utility function which we add. And we need a way to generate configuration I.e. hyper parameter sets that defines the models. And so we add a configuration sampler. And with those two elements, last operation to do is to split input and targets in order to keep to hold some part of the data set that will be used for the validation to compute the model utility function. And then it's exactly the same workflow repeated over and again over and over with some information flow from the utility to the sampler and a search strategy for the sampler. Okay, so just. Just a summary of the of the challenges that that we are tackling in this. In this chat with automatic machine learning. First, as you may have seen, we have an arbitrary mixture many many different input types to first figure out and to conform. Why figure out? Because we are not interested in the encoding itself. We're interested in the meanings that the input convey, which means that one can stand for an integer, which can be a class. Actually for nominal value, it can be some numerical value. It can also encode some logical value, a boolean. And based on the context you have to figure that out. So just here is a little memo about all the types we accept. And those can be vector, tensor or write array like sequences. Then we need to basically join all those inputs together and to generate to engineer the features that we will provide the model which can only take numerical or nominal train the model. So we need good optimizer and then define a strategy to search the best model given the data provided. And obviously if we want some common framework for the automation, we have very different tasks here at Ant. And that means lot of metric or utility function to accommodate. And obviously this is still not completely fully. It's not a fully clean problem. Because in the real world you're not asking your computer for the best model like pending high classify guy. Here are some data. Just provide me the best model ever. Because you're probably not ready to sit and wait in front of your computer for 10 more years waiting for Classify to find one more improvement in the model that provides 0.0001 accuracy better than the current model. So we need to aggregate the resource usage which is tied to performance goal in our system of the training times, the evaluation times, the memory and on top of the model performance. Right? So let's review all those challenges step by step and see how we answered Those and what we are currently doing, hopefully what we want to achieve. So when you provide any function like classify with a complex input with mixture of data, first you go through the type inference I already spoke about. Then we train a learned distribution on the data in order to impute the missing values that you can have here. Because many feature extraction mechanisms, they don't accept missing values. Then comes the confirmation. All the colors set to the same color space. And in order to have encodings that are all compatible together, we extract the value. There are some rescaling that can help for the model in some cases, and dimension reduction in order to actually compress the information as much as necessary for the model to work. It's a bit more elaborated than feature selection. And again you see there are many steps that are learned, trained and computed based on the data. So it's mostly a fixed and data driven with a lot of heuristic pipeline. And what it produce is a numerical vector, a nominal vector or a mix of both, which is what the model can actually take in. We're doing quite a bit of efforts to expose it. It's currently a bit blackboard ish. You can now extract information from the classifier using the property feature extractor which gives you this pipeline that is used prior to the model to extract the data. It's not that we want to hide it, you can access it by taking parts of the classifier. It's just we are really thinking on the best design to make it the most useful. And before we prior to exposing to the users. Right. And now that we have the feature, we can feed the model with at least the trainer with configuration. And so it's time to discuss a bit about the search strategy. So the most natural way when you have those hyper parameter to set and to decide is basically to start enumerating possibilities for say L2 parameters going to be 1 or 10 or 5. And this is called grid search. When you do a grid of possibility, and this is probably the most natural and underperforming way to do the same, especially because the space you are searching is huge and the relevance on a given data set of every parameter is highly fluctuating. So when you have large dimensional space and importance of parameters that is very different, random search is much better. You're much more likely to find an optimum with a random search than doing grid search. So with this in mind, we will basically start with a random search and try to beat this baseline. So the issue with machine learning is that generally the training are super expensive in time and computation. So you don't want to like you would do in a grid search or basic random search, train everything and see what happens. You want to explain exploit the information of the training you've done so far. That's one direction to improve on random search. And you also want, and that's what we are trying to keep and maintain to have on time anytime algorithm which you can stop whenever you want and get a result. And try to also be parallelizable. Also the criterions that we are using, excuse me, which are the validation, they are computed. They are dependent on the choice of how you split the test and the training data. So they are fluctuating based on those choice and those fluctuations in the measure you're making. They're basically setting the worst case scenario when you take decision on extreme cases. So here are the two directions we're going to see together. Try to find something that improves on random search and try to limit to evaluate the variance of our validation and and to limit it. So regarding the core of the of the automation, what's possible direction in bold you see those that we already implemented is the first and this is a very very huge trend in automatic machine learning is to start with priorizing a good portfolio of configuration. So you train on many data sets and it's basically the expert knowledge, you know that those configuration are somewhat. It's a Swiss army life, they are somewhat working well on different data sets. And you know that starting with a warm start of the optimizer with those candidates, you will already start with good proposal. So it's a prior on the sampler. And then another idea is if you have any time training and if you can interrupt your training is to use early observation of how your model are performing to prune your search space and remove candidates. And so it's called multi resolution optimization. Like you get an image and you have a few pixels and more and more pixels. The resolution is getting better and better and you can decide whether you want to carry on with the download or not. It's very like Internet ish pre fiber time classic algorithm for this are success evolving which is close to what we are doing itotens which is some variation of it which I will describe now. So you start with a lot of configuration and you train them on very few data and so you see how they behave. And based on the validation you have here, you select the best one only given fraction like 1/2 or 1/3 and you move on to train of 10 times or 5 times more data and you carry on in this which we call round again and again and select only the best. And so hopefully you will select just. You will train only a handful of configuration on the full data sets. So we will gather as much information as possible at the early stage and stop configuration that we deem useless. And eventually. So each learning curve here is a configuration. But you may see also that they are error bar because we are estimating not only the validation mean, but the distribution. We will discuss about that later. That's important to when you're interested in the rank of your candidates to know how much they fluctuate and. And finally another direction is to gather final results, because you can see here that I have several blue lines which correspond to logistic regression. And I may think that probably the L2 regularization parameters that I have here, I can find a soft spot by just looking at how they perform ordered together. This is something that is that I can optimize with a surrogate model, say based on those what I've seen here. I think that next time I should sample more around 10 than around 100,000. This is Bayesian optimization, we don't have that yet. Right. You saw the error bar. So you see that we estimate the variance of the validation. We do that with scaffold validation. So when you have the data set, instead of doing a single split training testing, you do several of them and you rotate which you call fold, and you rotate the one that you keep for testing in successive training. So you train on the blue one and you test on the orange one, which leads several utilities for the same size in your rounds. And you can estimate the mean and you can estimate the mean and standard deviation from that which is much more informative than a single value, though at the expense of more training, more computation. And of course, because you want those fluctuation to be as small as possible, you want every fold to be as representative as possible of your population. So in the case where you have very unbalanced and then ordered inputs at the beginning, you may want. We do use automatically stratified sampling and randomization to shuffle the data and make sure that we have elements of every class in the proportion of the total population in every fold. And now let's magnify a bit the screen to look at how it goes in practice. So I'm launching a training on toy model MNIST data sets with a subpar 20,000 example of handwritten digits with a time goal of three minutes. And what you can see here is that we are training on 700 of the total. So 4%. And for every round here the summary is giving us the winning configuration which can evolve from one step to another. And you see the error bands which comes from free from the cross validation and let you know how much the model actual cross entropy in this case which is the metric for classification varies. And obviously you have this stop button here you can stop anytime. And. If you're happy enough with the data, here is the accuracy which shows that it's plateauing. So it's probably an interesting state to stop. But you see that it found a new winner and the curve evolved. And those curves are actually the minimal envelope of the curves that we discussed on previously changes that we made from previous time. So you have all the Euro bar here we found out that it was probably more interesting to take decision to stop or not based on what's happening at the very or between the best configuration so far. So we are zooming in even though you don't see the huge amount of configurations that have been trained on the first rounds. And also we are adding colors schemes so you can see that you know can tell which method works, Which method is used. You have random forest here in green blue are logistic regression and so on. And let's stop it actually because we are getting already a decent. Okay, so it's nearly any time you need to wait for the current training to finish in order to get things. So here are the classifier and all this you can retrieve with the information in the information panel. You see that you get the cross entropy estimate of the variance for every metric that we measured during validation for free same for the loss. And all those information also comes from the validation step, right? And you can see that now you can plum for the best configuration for every data set size which might be of interest also if you want to invest in more data somehow. And as we are working on the automation we are also this is not exposing to null but we are also keeping track of every iteration of the algorithm for every every model and the value which you can be interested in. And in conclusion then the the in progress projects this on what we this is what we commit on which is basically improve the reporting as you as you may have seen on the on as the trainings goes on, but also expose more about what happened during the automation to let you check whether the current set the settings that you have for class before for the function correspond to your actual goal. If you're interested in the last person of performances or not. We want to extend this automation scheme. We think you can give good results for unsupervised function like clustering which are currently using a simplest system. And those are the challenges that we have not yet. We are working on, but we're not yet completely clear. It's more research. So a big makeover of the processing pipeline with high performance data structure underneath and user exposed. As I told you, out of core training, which is just get the data just when you need it from the hard drive to not put everything in memory as it is happening now. Online training, which means basically training with a stream of data instead of one data set first and also being able to resume training with a given classifier and carry on with as more data come. Bit like active training. And to get the best during the out of the training you just let me just change the magnification because. Yeah, right. Yes. To put some Bayesian optimization in the sampler in order. It's important in the heavy resource usage case when you're training on very, very large data sets because then you invest a lot in training and you have a lot of information and makes sense to train a surrogate an additional model to try to sample in a more efficient way. And I will put this online now for you to access it and there is a lot of resources and selected material that you might be interested in looking at and that's it.