The Lottery Ticket Hypothesis - Live Coding

The Lottery Ticket Hypothesis: Finding Sparse Trainable Neural Networks. Today we're gonna reproduce the experiments done in this paper with the LeNet model on the MNIST dataset. I will put a link in the description, um, following to my personal GitHub where I reproduce other experiments done with Conv6, Conv4, Conv2, and even the VGG architecture on the CIFAR-10 dataset. So to start, if you find any type of mistake or anything I did wrong in this video, or that could improve this video, feel free to let me know. Otherwise, let's start by creating first our neural network. Architecture, which we will call it LeNet as the model. So yep, LeNet right here. Let's open it. Import torch, obviously. Import torch.nn as nn. We can create our class, which we'll call again LeNet, which inherits from nn.Module because it's a torch.model. We create the init function, and now we do have the LeNet 300/100 version, which is simply some linear layer. So let's create our classifier, which is sequential because it's a sequence of different layers. It starts by a simple linear layer which takes in inputs the MNIST image, so it's 28 by 28, which outputs 300, which has so 300 neurons. We have as activation function a ReLU. As, as second layer, we do have right here 300 obviously as inputs, 100 as outputs, and again a ReLU. And finally, our output layer, which takes as input 300— 100 features and has 10 output features. Well, it's already done for the initialization of our neural network. We can do the forward method, which will be really straightforward. If you take torch.Tensor, which is our input data, it will output again a torch.Tensor, and we can simply do self.classifier t, and obviously returns the results. Maybe, yeah, t right here will be an image, so we will have to do a flatten first. So torch.flatten, our inputs will be t in all cases, start_dim will be 1 because the first dimension is always batch size. And yep, okay, so our tensor is flattened. We give it to our classifier, a sequential of layers, and we're ready to go. To verify everything works well, which should be, we can create a simple main Yep, so torch.rand will create so a simple random-like images, so 128 by 128. So it will be our tensor, our model will be so LeNet obviously, and we can No Siri, I was not calling you. And yep, output will be of one method. And right here. So let's see if everything works well.

Yep.

And here we go. So yep, everything works well. I guess now we can— nope, I was not calling you. I guess now we can create our data_handler-like file. So this file will be responsible to import the MNIST dataset, transform it, and giving—

oh, sorry—

and giving us back the different data loaders we need. So again, let's create a simple file called data_handler.py. My bad. Yep, right here. So what do we need? First, we'll need the MNIST dataset. So for from, I guess it's torchvision. So yep, torchvision.dataset import MNIST. We're gonna need from torchvision transform right here, and because we will need to normalize our dataset. And I guess we can start to create or get data loaders. So, our DataLoader, we take as arguments the batch size we want. In the paper, they use 60 as batch size. Here we'll set the default num_workers as 1. In my cases, I will set it to 4 later. We take kwargs, and I think we're ready to go. First, let's import our train set. So it's MNIST dataset. Roots will be— let's get from QWorks. By default, we will set to data, I guess, data folder. It's the train set, so train =True. The transform, we will do it later, and we want to download it to not have to re-download it again each time. We can do the same for the test set, so just the only difference here is False. We'll need to import from torch torch imports randomperm. Yep, and we can call it as torch.randomperm so we know it's come from Torch. What we want is to take 5,000 samples, random samples from the test set, from the train set to create our validation set. So we just, I'm gonna create some, some random indices. So shuffled_indices equal torch.rand and it will, yep, be as long as our train set. So we just have all the indices of the train sets but in a shuffled way. Now we can create our valset, which will be subsets of this dataset. So from torch.utils.data import Subset. Yep, my bad. So Subset, it's a subset of or train sets, obviously. And the indices will be shuffled indices. Yep, and we will take only the 5,000 first samples. Obviously, we will have to do the same for the train sets, but by removing the, the first 5000 samples because we don't want our validation set to have any value coming from the test set, obviously. And now we can return the different DataLoader, which I didn't import. It's right here, DataLoader. So DataLoader for our train sets, which will have batch, batch size equal to batch size. It will have num_workers equal to num-workers, and we want to be sure that our sets, our datasets, is shuffled each time we reuse it, as explained explained, and as said in the paper. So we're just gonna do the same for the other sets. So here it will be test sets, and the last one will be validation set. Just checking if everything is okay. Okay, so now we do have our data, but it's not normalized. So we just gonna create another function we could call get_transform because it will give us any transformation we need to apply on the datasets. So yeah, my bad, transform. Yep, so we want to have the normalize transform. Transform, which will normalize datasets. Transform— yep, I imported it. Normalize, mean— in all cases we are using MNIST, so it's 1307. And for the standard deviation, it's 0.03081. Okay, now we can return a ComposeTransform because we'll need to apply two things. Oops, my bad. The first one will be to transform our data to a tensor, obviously, and the second one will be our normalize transform. And I think we're ready to go. Yep, yep, yep. So here we have our data, we have our model, it's okay. So I think we can now go to create a simple dummy main function, main.py. So what we're gonna do in our main.py, I guess First, we can import our model. So from lenet import lenet. From dataloader, yep, right here, import, we called it, he's not very happy, get_dataloader. Oh yep, I forgot to use this function get_transform. So obviously for any data we will use we want to transform it, so transform will be equal to get_transform. Same in our test sets. And now we're ready to go. So yep, get_data_loader, get_transform could be a private function. So yep, get_data_loader, no problem. And now what we could do is create, yep, our main function. So our main function will take as parameters the number of pruning iterations we want to do, so it will be a simple int, the number of training iterations we want to do. Again, we will use— no, I think we will use a lesser value compared to the paper. Again, too, we don't have to wait very long. We want to have the initial weights of our model, and we want to know if we want to apply the, say, lottery ticket hypothesis. So apply lottery ticket hypothesis, which will be bool, my bad, and by default it will be equal not to True but to True. I think we're ready to go. So yep, we can select our device. So let's import torch again. Our device will be equal to torch.device cuda if torch.cuda is— yep, yep, is available. Right here, she's not really happy again. Else it will be CPU.

So why are you, are you not happy?

That's a good question.

Oh, okay, my bad, I just forgot the half else, the else keyword.

So yeah, my bad, we can remove this. We can create our model, model equal linet. We want to move it to our device. We want to— we want it to load this state_dict, which will be the initial weights. To be sure that we don't change it anytime, we'll import from copy— uh, deepcopy, and we'll do a deepcopy. Of this one. Like this, we are sure we never change the initial weights here at any time. We want our criterion, so in our cases it will be cross-entropy, so torch.nn.CrossEntropyLoss. Yep, we want our optimizer, optimizer.

My bad, nope, which will be equal to torch.optim.adam as stated in the paper.

And we will use, okay, so model.parameters right here. As the learning rate, we'll use the same as the one, the describing the paper, so 1.2e-3. And we can import our data, so train_data_loader, loader. We can do the same, yep, for the test_data_loader, for the val_data_loader, set get_data_loader. Dataloaders, so we retrieve our different data. As stated before, we will have a batch size of 60, and we will have num_workers equal to 4. I still don't know why he's not really happy to to use that app. So let's try to run it all to just see if everything works. Expectate statics. Yeah, got none, obviously. So yep, just gonna give you a lunetz.static. Don't worry, no problem, my bad. And yep, it's downloading our data. Our data now should be in data folder. Perfect. And now we can straight start training iteration. We're just gonna import from tqdm— tqdm to— import, my bad, import tqdm to have a loading bar. So we will do as many training iterations as given in parameter. So for n in tqdm number pruning iteration, we will do it range, we want to start from 1, pruning iteration plus 1. Yep, I think we are good. We just want to set it as leave to false because we will run it 2 times. One to see the effect of the lottery ticket hypothesis and one to see what happens if we don't apply the lottery ticket hypothesis and we just randomly initialize our weights. So, okay, okay, I think we we could, we can create a variable named training_iteration equal to 0. While training_iteration is less than max_training_iteration, we will keep training our model and testing our model obviously. So now we do need to create the function to train a model, which will be straightforward. So we just create our file training, our function train— yep, we'll call it train_model, train_model, my bad, which which will take as arguments first our model, our DataLoader, our optimizer, criterion obviously, what— yep, the current training iteration to know if we're gonna stop, so if we need to stop, So the max training iteration again, and the device. I think that's good for now. We just import torch to have some type hinting. So our model is a torch.nn.Module. Yep. Optimizer is same. Nope, optimizer is an optimizer, my bad. So torch.optim.optimizer, or Criterion is a module, or DataLoader is torch.

Utils—

nope, it's—

I noted it some—

yep, nope, it's torch.utils, I was right—.data. Well, I better do from torch.utils.data import DataLoader. It will be less verbose, more readable. Yep, training_duration, which is an int. Again, and here is a torch.device.

Nope, not work. Okay, so I think we're ready to go.

Yep, first we want to be sure that our model is in training mode, so I set it to train.

We want—

so for data in DataLoader, we want to retrieve both the inputs and the labels from data. Inputs, not in-puts. We want to move it to device. Yep. So if you're using any GPU like that, it will use it. We want to be sure that our optimizer is ZeroGrad. We will do the forward pass. So outputs equal model.inputs— inputs, yep, right here. We'll calculate the loss using our criterion. So it takes both the outputs at plural and the labels. We'll do the backward pass and our optimizer step, and I think we're ready for, uh, dummy training. I'm starting to believe this video will be longer than expected. Obviously, each time we do this, we want to add 1 to our current training iteration. And yep, yep, yep. Yep, I just gonna check something. Yep, and if training iteration is more than the one expected, we do a simple break. Okay, so we want to retrieve basic data like, or I don't know, loss or accuracy or anything. So let's create step losses, which will be an empty array. Same for the accuracies. And right here, loss.backward. Mm-hmm. We want to do it before the break to losses and step.accuracy, and we'll add accuracy. So currently we don't have the accuracy, so we just gonna create a simple function that will calculate for us the accuracy. Like this, we can reuse it for the testing part. So let's create name it "calc accuracy" which will take as arguments our prediction and let's call it labels and it will return us a float. So what we want to do, we just want to if things are equal so So torch.argmax because we want to take the argmax of each prediction and the— starting from the first one because again, you know, you have batches and you want to see if the argmax of our prediction is equal to our labels.

We want to do the sum.

Which we can do because it will be just some bools, booleans, so if you add two true and one false it will be equal to 2, you know it, divided by the length of the prediction, and we want this to be a simple float, so we take the item. Now we have, or we can of our accuracy. So yep, we have a loss right here. Just gonna do this after the— yep, accuracy equal calc_accuracy, prediction will be outputs, labels will be labels, and we're ready to go. At the end, obviously, we want to return our step_loss and our step accuracy.

Okay, so now we can already start to train our model to see if everything works, which should be. So, um, from training, training imports train, yep, what's it called, it's train_model, my bad.

Again, calc_accuracy, I think it could be a private function without causing any harm. And okay, train_model, yep.

Train_model which takes this as arguments. Yep, yep, don't worry, take, take, take, take, take, and remove. Perfect. So model equal model, DataLoader equal training DataLoader, optimizer will be optimizer obviously, same for criterion. Training iteration will be our current training iteration, and max training iteration will be max training iteration, our device will be our device. So train, train, train_model returns step_loss and step_accuracy, but I see we do update training iteration, but we don't return it, so we we want to return it, and we will retrieve all data from train_model. Yep. So, and we will call it train_loss, train_accuracy, because we want to do the mean of those values, so it will be simply the sum divided by the length of, well, the thing. So again, sum training iteration—

no, not for you—

divided by the length of step. I will— yep, copy paste. And now we do have the mean of those values, so we are ready to go. For n Bam! Okay, I think we can create another loading bar to just see how many times we have to wait for our model to be trained. So yep, let's create a new progress bar. So p_bar is equal to tqdm, the total will be max_training max_training_iteration. Again, we don't want to leave it, so leave it to False. And at each training, we want to set a description. So it will be— first, we want to know if we are applying— yep, it will be an f-string. First, we want to know if we are using the lottery tickets. Hypothesis, or if you are applying till lottery ticket hypothesis, or if we are just randomly initializing weights. So it will be— my bad— LTH if apply lottery ticket hypothesis, else it will be equal to random. We want to know our train loss, so train_loss equals, and we want to just I think it's— yep, nope, I forgot a dot. Want to do the same for the loss. Nope, it was— yep, I did the loss. Now we want to do the same for the accuracy. And yep, I think we are good. We just need to update our progress bar to know how much it needs to be updated. And for that, We want to know how many training iterations we did, so let's call it last_training_iteration equal training_iteration, and like this we can do training_iteration, training_iteration minus last_training_iteration, so we know how many steps we did. I think we're good to go. For first test, number pruning iteration, I do it right here, max pruning iteration. Okay, so yep, number, we'll set it to 1. Here we can set it to, yep, 1800, and let's see if everything works. Well, okay, everything seems to work, to work fine. So, and now we can do the same, um, but for the test data. So it will be here, uh, not train, it will be test. Yep, nope, we'll just do test. Okay, we will not need optimizer, we will not need the training duration, the max training duration, only the device. And we will put it in training. So it will be test_model. We don't need this, we don't need that either. So we'll want to be sure it's in eval mode. We want to do it in a no-grad way. Just checking if I forgot anything. Yep, see loss.backward, we don't need it. See optimizer step— why did I took— see, nope, we do need the criterion, but we don't need the optimizer. Yep. So, okay, loss.backward, bam bam, training duration, nope, step_loss, okay, okay, okay, okay. And we don't need to return this either. So I think we can simply do with torch.no_grad. And yup, just to be sure we don't have any gradients. And to put everything a bit faster. I think we're ready to try again. We'll just update our, our description. So here we'll do the same but for test. Yep, right here. TestModel, which I forgot to import. So yep, don't worry, test_model is right here, and let's try again. Uh, testing— well, I did here a mistake, my bad. So yep, don't worry, we do have our Criterion, our device, our data loader, our model, so everything should be fine. It works, so great. Now we can start to, well, I guess apply the pruning at each iteration, so like states in the paper. Yep, yep, so we'll do, I think it's, let me check to be sure, yep, from torch.nn.utils, yep, no, okay, I didn't import it before, import prune, and at each time we want to prune. So yep, for n_time, we want to prune each layer in an unstructured way like stated in, in the paper. So the first layer we want to prune right here will be this one. So it will be our model.classifier . So model.classifier index 0. We want to prune the weights and amount will be our pruning, pruning rate, which I need to create. Our pruning rate, we will say it's equal to 0.2 like States in the paper, power 1 divided by n, because we are doing iterative pruning. So again, just, uh, reading the paper, we want to do it the same for, um, this one. Yep, so it will be, uh, index 2, and we want to do it the same for the last one, but Again, let's stay like stated in the paper. The last one will be pruned by pruning rate divided by 2. Okay, now it's where we want to apply the reset, the weights either in a random way if we don't apply the lottery decay hypothesis or to their initial value. If we want to apply it. So if apply_lth, uh, weights_to_reinit2 will be initial_weights, else— oh, we can do it. Yep, else we can do it in one line, can be more readable. Else it will be, well, a simple lunetz.state_dict. Yep, just checking if I forgot something. Nope. So now we were ready to go. We want to Yep, so reset_weights equal a deep copy of our current model state dict. Again, I just do a deep copy to ensure we don't change anything we want— we don't want to change. So I think we good. Just checking again. Yep, so for each for each param we do have in resetted_weights.keys because we want to retrieve each params. If param, so it will be here the name of the param that splits with dots, we want the last one because it will be name, so either weight or anything. .replace, we want to replace our rig by this. I will explain it in a few seconds. In, well, I can just do equals, wait, the value, yep, in, wait, right here. So, okay, why did I apply a replace here? If you don't know, when you apply pruning using Torch, Torch will add a new value to your state dict. It will be called weights_orig, which will contain your original weights before pruning. You will also have a mask and your weights param will no longer be parameters, Torch parameters, it will be just values. So if this is the case, we want to reset the weights, we want to set it to a deep copy again just to be sure, weights to reinitialize to Well, I— did I do the deep copy? No, okay, so again, okay. Again, we'll do it— we'll do sys. I don't think sys will work. No, my bad. param.replace. And I forgot here something. Nope. Why are you not happy? Because, yep, that don't do any sense right here. Okay, so here we do apply all pruning. We do check which weights we want to reinitialize the weight to, so either the initial weights or random weights. Here we do have the targets weights, and now we just have to do model.load_state_dict resetted_weights, and that should be right. So now we can increase the number of training iterations. Let's set it to 3. In the training, I, well, I think we should here show how much sparse our model is. So we'll create a variable called current spore_city. Again, it will be simpler. And current_spore_city will be— I guess we can set it at the, at the beginning. So yeah, right here we see rest of the other variables. current_spore_city, we will create a function, so gets Yep, get_sparsity, I guess, simply will take a model as parameter. We can create here a function get_sparsity which takes a model. We'll just do a simple, really dummy function which could be really smarter in much ways. But let's go straightforward. So, torch.sum, we want to check any— again, I will just, yep, copy paste this, it will be faster. Here,.wait right here equal O, so yep, just to be sure, we want to check any— wait, that's all— pruned divided by floats— see, same thing, but here we remove equals O and we say that's number elements. And here we don't have to do `torch.sum`, okay, just, yep, do this. And I think it's, yep, it's, uh, no, it's not `current_parsee`, it's `get_sparsity`, my bad. So here you're good, and again let's run this to check if anything works. I'm starting to think if the next time I should be better to split this format in multiple, in more videos, like first step creating C model, second step creating C data handler, again and again. You understand my idea? So let's Current sparsity, okay. Okay, I expected it to be higher or lower in a way. Current sparsity, okay. Oh, did I do anything wrong? Okay, okay, okay, right, perfect, perfect. So, okay, now can we— what can we do? We can, I guess, First, there is two things. We want to compare the results when we are applying the lottery ticket hypothesis and when we are not applying, where we are just doing random things. So we want to set, okay, here apply, bam, apply equal True equal true for the first one equal false for the second one. Here we are, yep, we are setting the number of training iterations, we are setting the number of max training iterations.

Initial weights.

This. And yep, just let's, let's put it on multiple lines. Yep, right here and here, here, here, here. And yep, perfect. So true/false. Here we will do initial weight equal those one, and we'll just— yep, set the initial weights. We'll set, uh, nbpruning— so nbpruning_iteration, uh, we'll set it to 5, I guess. And here we do want the exact same number, max max_training_iterations here to 5000. Yep, and I think we are ready. Now our main function needs to return some values because, hey, how can we compare those values if we don't do it? Hey, I forgot, I did a mistake right here. I did provide the train_loader to the test_model function. So my bad. Just doing this, we'll set it just for debugging or for developing to 2, just to be sure if it works. And right here, now we do want to, to have some values. We do want to use our Valentin De Matos loader. We do want to know the evolution of the test loss of the training iteration again and again. So what can we do? I guess right here we can create some dict. So it will be test losses, and we want to know the losses for depending of the pruning. And the current training iteration. Same here, so test accuracy, and later we'll do it the early stop iteration step. That's plural. And I think— oh, I didn't watch— my bad. So I guess it works. I will just run it again just to be sure. So now, at the end of each— yep, test accuracy, okay, great. So at the end of each training, we want to current_sparsity— yep— update or, you know, retrieve value training iteration and it will be equal to test loss. Great. Accu— yep, and here it will be "sys" and here will be "c". I think we are good. We can— well, what can we do with this? We can just simply return it. So at the— okay, yep. So at the end, return test_losses, plural, test accuracies, and later we will want to check the early stop. So, um, let's retrieve it, uh, just to check the difference. Let's call it the first one LTH, and the second one will be RUND. Okay, perfect. Now we want to plot the evolution of the losses, of the accuracy, depending the current sparsity of the model and the training iteration. So let's create a new file which we can call plotting, I guess, plotting.py here. Plotting.

And we want to plot a few things.

First, let's import matplotlib.pyplot as plt. We'll want to plot the test, different test losses. It will be, it will take as arguments the test losses when we're using the lottery ticket hypothesis, when we are doing random initialization, and we want to know where to store the input image, input image path. So yep, just some string. Uh, PLT will start to be— oh, uh, yep, here don't seem really happy.

Why?

Why are you not happy? Was not close? I'm pretty sure it's close, Betty. Like, really? Why did I do it wrong again? I don't know. I really don't know. Well, let's Let's see.

Python, RSSI plotting.

Nope. Okay, so it's just my VS Code, which is a bit dumb. Just to be sure. Okay, str, and here you go. Okay, so my VS Code is really dumb, uh, which is a bit bad. So first we want to clean our current figure. We'll want to create a new figure. We'll set a fixed figure size equal to 20 by 10, just to have something a bit nicer, tight. Layout. Yep, didn't do any typo. We want to have the same color for, you know, okay, so we want to compare the loss when using the large-scale hypothesis and when we randomly reinitialize all weights. So we want to have the same color for each sparsity test. So we'll just create a simple iteration. We need to import from matplotlib.pyplot import, I think it's called, yep, cm, so for column map. We want to have a simple rainbow, rainbow colormap. From numpy import linspace. I'm just gonna close it and reopen it again to see if he's happy. Not he, he's like really not happy. I don't know why. Okay, so we do have a rainbow. We want linear spaces starting from 0 to 1 with as many steps as we do have, and I guess we good. Current sparsity, or more sparsity t levels, because we have two different sparsity levels. We're just gonna round it so, you know, it will not be— it will not have too many figures. We want it to 2 for sparsity level in, in last— that's case. Because it's a dict again. Okay, it's good. Now for sparsity levels, uh, k in— hey, don't worry— uh, in zip both sparsity levels and the real sparsity levels with all the figures. The color will be the next color in all_colors. We'll set plt.plot. We want to plot the losses— yep—.keys. I think it's good. And for here, it will be values, obviously. We want to set the first one as a full line, and we can set a label, a label key, which will be an f-string. Yep. Lth because it's Latent Reticle Hypothesis, and here we will set 100 minus sparsity— hey, sparsity level, singular, singular, sparsity level, it will be nicer— and c, my bad, and c equals c, the same but for— hey, where did I use— oh, I really don't know what my VS Code has currently, but he definitely have an issue. Here it's random, I think it's good. We'll set xlabel, so it will be training iterations. We want to set the y label, which will be loss on the test— yes, it's a test set. I will set a nice title, so "Model loss regarding the fraction of weights remaining in the network." 'after pruning'. And just add the legend because we put some legend. The location, we'll let matplotlib decide which one is the best. And plt.savefig, we want to save it where the user asked it, and I think it's good. So, yep, we're gonna try if anything— everything works. Yep, yep, plot. It was this and this one, and my bad, for I forgot the name of the parameters. This one will be in images/loss.png. Yep, just gonna create this folder. mkdir images. Yep, right here. And we can test to see if everything works. Now we can do the same for the accuracy. K-ERO. Yep, you know, don't seems really happy because I didn't create it before, so we need to create it here. Yeah, don't worry, it will be before. Yep, I think here equal a dict. Same here, equal a dict. Yep, you created, created, current species you created. Okay, so everything is good, I guess. And we still have to do this one and this one. And now we're ready to go. For the plot side, let's do the same but for the test accuracy. Uh, train, uh, so accuracy, and the Tesla's model accuracy. Here, it will just be everywhere it's written loss. Just have to set accuracies. And I think everything is the same. Here, plot_test_loss doesn't exist. That doesn't exist? I'm pretty sure it exists. Oh yeah, maybe I didn't import it. Like from loading, uh, imports. Yep, yep. And here— oh, same thing, um, my bad. Yep, nope, nope, it's right. So plot_test_accuracy right here, and we can do the same but with plot accuracy, which will be accuracy here. Accuracy, it's exactly the same function. I shouldn't create two different functions, but well, okay, no problem. Plot test accuracy. And let's run it again. And you're not happy because, uh, why? Run. Okay, add comma, my bad. Invalid again. Uh, plotting line 20, my bad. And yep, so I guess here same mistakes. And now we're ready to go. Okay, while it's training We can start to talk about how we can evaluate the early stop. So first we're gonna have to create a new file which will contain classes called early_stopper.py. Are you happy? You're not happy. Can't convert CUDA— oh, my bad. Can't convert CUDA device type tensor to NumPy, blah blah blah blah.

So yeah, yeah, yeah, yeah, don't worry.

Can't convert CUDA device type tensor to NumPy. Use tensor CPU to copy the tensor to host memory first. Hmm, okay, I should here teslas.cpu.item and you should be happy. Now let's rerun it. A list wrapper, so it will be name— hey, you don't happy here? Okay. Oh yeah, it's already— yeah, my bad. I'm here, it's already a float, so no problem. So yeah, I was saying we will create a list wrapper which will be a simple dummy, really dummy. Class. Let's create init. So self, we want to know, to set the patience. So it's an int. By default, we will set to 1. self.patience. So the patience is how many steps we accept the validation loss to increase before considering we should stop the training. So patience patience, self.counter 0 because, well, the current iteration equal to None, so when did we stop, and self.min_validation loss is equal to float Infinites. We want to set the call method which takes self, the current validation loss. We'll simply take, hey, if validation_loss is less than that the current mean validation loss. So self.counter equal 0 and self.validation_loss equal validation_loss. I think it's good. Uh, elif validation, uh, loss is higher than the min_validation_loss, well, self.counter 1 if counter is more than all patients, more equal, Well, we return true, otherwise we return false, and yep, yep, yep, I'm missing that I don't need iteration, I guess. Yep, self of— yep, no, we— okay, okay, we will just use it to store information, I guess, but we'll keep it to none. So we do have now our images, so we can check the accuracy. Oh yeah, I forgot, here in the plotting, we want those ones to be in in Deutschland, otherwise we can't compare it. But here we can already see some results. Okay, in the paper they do— in the plots they show the mean of 5 experiments. We're here to not have a video 2 hours long, and I think we will be pretty near since this time range, we'll just do one, one experiment. And again, you will have a link in the description linking to the GitHub project for this one. So yep, now we can see the difference of the accuracy when we are using the Large Reticket Hypothesis and when we are not using it. And we can see it's pretty random at this point. So, yep, I think we're good. Yeah, I was saying, now we want to check the early stop, and this is where we will use it, early stop and Vail Loader. So, In the train model, we'll have to add the DataLoader, key is equal to DataLoader, the— my bad— the EarlyStopper, which will be equal to EarlyStopper or EarlyStopper Yep, we want to initialize here, right here. So early stopper equal early— well, I can just do this. Hey, I did a typo here. From early stopper import EarlyStopper, early_stopper EarlyStopper with patience=2, because why not, and I think it's good. Now in all— yep, just save this— in all_training, this will start to become a bit messy, this function, but yeah, you know, we want, as stated in the paper, to check the validation loss every 100 training iterations. So yeah, just right here, if training iteration module 100 is equal— oh, we'll set model.eval, set it back to model.train after with torch.no_grad. Yep, I have it right here. With torch.nograd, we can, well, just call test_model. We want to take the validation loss. We don't need to take the validation accuracy. Model equals model, obviously. Data_loader equals val_data_loader, criterion equal criterion again, my bad, and device equal device. I think we are good. So, oh, now we need to use or early_stopper. So, yep, okay, I will add it later. If early_stopper iteration is None, so we do not set it, and early_stopper.value, so if we need to stop, uh, elly_stopper.iteration equal training_iteration, which is right here. And now we're good. So we set it back to model.train, okay, and if needed we break. And now, okay, we do have everything here. We just need to add the new parameter which was First one, after device, yep, right here. So ValDataLoader, which is DataLoader, a ListOperator. Hey, okay, you really don't seem really happy today. From early stopper, just for typing, type. Hinting import EarlyStopper, so hey man, it's an EarlyStopper. And now I think we're ready. Okay, we want to check the evolution of the early stop. So where was it?

I'm just checking my notes.

Yep, yep, it's at the end after the updates. Yep, right here. If a list opener— or where is our list opener? Right here. If we did add_to_stop earlier, early_stopper.iteration is not None, well, we set this one current_sparsity, hey, current_sparsity, thanks, equal to Alistair Bird iteration, else I could do a one-liner, I guess, else is equal to training iteration. I think it's good. Now we can return this one here.

And here we want to retrieve it.

Unlist up iteration steps, uh, LTH, run, and just run to see if anything— everything works fine. And while it's doing its work, we'll just create a new function. So Funny, I didn't create a train. Shouldn't work. Plot test. Oh yeah, no, I did it. Okay, so plot early stop, and we do have here early stop when using LTH and early stop when not using LTH. So, uh, yep, we have a table layout here.

We don't need color, we don't need everything here.

Uh, we just have percentage of weights remaining, remaining is equal to rounded of 100 minus sparsity level. Thanks. And we want to round it to 2 digits for sparsity level in in a list of LTH.case right here. So we do have our different percentage of weights remaining. Here will be this. Hey, my bad. Nope, nope. Okay, I understood. Right here. Okay, you seem Happy. Now here, I think we just have to set here to early stop, and we don't have any keys. Yep, yep. Okay, here value, we don't— here. This, so it don't have to be an f-string, just a normal string. We don't have any color, so we can remove the color. It will not be this, it will be the percent of weights remaining, and here it will be early stop iteration on the Valset. And title, we can call it Early Stopping Iteration of the— yeah, don't worry, don't worry— of the Iterative Lottery Ticket experiment. And I guess again should be fine. Now we want to plot, say, evolution of the early stop, and we can put this in the main function, early stop Yep, is equal to this one, is equal to this one, and this one will be equal to this. Early stop. Okay, I think it should be good somehow. Yep, I think it's good. Just checking if I forgot anything. Um, I forgot to close the progress bar, but it's not really that important. Um, yep, here it's running. Here I did a typo. It's not this one, it's training, training, training iteration. Thanks. Here, pbar.close, want to close our progress bar. Yep, everything should work. Yep, yep, yep. And just, we just have to wait. Okay, so forward to import CIF function. So yeah, just restarting and see you at the end of the training. So, the training ended. Now we can retrieve our results. Forgot to set the same color, but it's okay. Just gonna reset the plotting here, and we want to set the color, let's say, to blue. I I guess. So here, label_color equal to blue. And now we finished everything, so we can start the number of training iterations to 5, set the max number of training iterations to 5,000. I think in the paper it was 15,000, but again, just so you don't have to wait too long, we'll just set this to 5,000. So let's train again and see you at the end of the training. And now training stopped, so let's, let's Check the result. First, we can see that our model do indeed need less training iterations to train compared to random weight reinitialization. You understand me? Again, we should do more experiments and do the mean of the experiment to have really concurrent results, but again, you should really check out the link in the description to have the full test. The accuracy of the model applying the Large Read Ticket Hypothesis again is higher in general compared to a random rate reinitialization. Again, you understand me? And the same applies for the loss. You can see that the loss is much lower when applying Lottery Ticket Hypothesis, even more when using high level of sparsity. So I think this video is already long enough, so feel free to subscribe if you like the video, to like it, and to write in the comment section any idea you have that could improve this format of video. So take care and see you next time.