Deep Learning with PyTorch Full Course | Master PyTorch, Tensors, and Neural Networks

What's up everybody? Today's Videos is a collection of course videos coming from our Deep Learning with Pytorch skill track. It covers everything you need to know about working with Pytorch from building deep learning models in Pytorch using multimodal output and input, as well as dealing with issues like overfitting, vanishing gradients, and a lot more. If you enjoyed this video, make sure to like and subscribe to the channel. And if you're interested in going beyond watching videos but actually practicing in the browser, check out the link in the description below.

Hi, I'm Yasmin, a senior Data Science Content developer at datacamp, and I'll be your instructor for this course on deep learning with Pytorch. Deep learning is everywhere. It powers many recent and exciting innovations, from language translation to self driving cars, medical diagnostics and chatbots. Deep learning is a subset of machine learning where the fundamental model structure is a network of inputs, hidden layers and outputs. As shown, a network can have one or many hidden layers. The original intuition behind deep learning was to create models inspired by how the human brain learns through interconnected cells called neurons. This is why we continue to call deep learning models neural networks. These layered model structures require far more data compared to other machine learning models. In order to derive patterns, we are usually talking about at least hundreds of thousands of data points. While there are several frameworks and packages out there for implementing deep learning algorithms, we'll focus on Pytorch, one of the most popular and well maintained frameworks. Pytorch was originally developed by Meta AI as part of Facebook's AI Research Lab before it moved under the Linux Foundation. It is designed to be intuitive and user friendly, sharing many similarities with the python library numpy. We can import the PyTorch module by calling importtorch. The fundamental data structure in Pytorch is a tensor, which is similar to an array or matrix. It can support many mathematical operations and forms a building block for our neural networks. Tensors can be created from Python lists or Numpy arrays using the Torch Tensor class. This class converts the data into a compatible format for deep learning. We can call Tensor Shape to display the shape of our newly created TensorFlow and Tensor DType to display its data type. Here, a 64 bit integer checking the shape and data type ensures tensors align correctly with our model and task and can help us in case of debugging Pytorch. Tensors can be added or subtracted provided that their shapes are compatible. Two tensors are compatible if they have the same shape, meaning their dimensions align exactly. When shapes are incompatible, we get an error. We can also perform element wise multiplication which involves multiplying each corresponding element from two arrays of the same shape and many other operations involved, including matrix multiplication. Matrix multiplication is a way of combining two matrices to make a new one. Each value in the new matrix comes from multiplying a row from the first matrix with a column from the second matrix and summing the results behind the scenes. Deep learning models perform countless operations like addition and multiplication to process data and learn patterns. Let's build our first neural network using Pytorch tensors. A neural network consists of input, hidden and output layers. The input layer contains the dataset features, the output layer contains the predictions, and hidden layers lie in between. While a network can have any number of hidden layers, we'll begin by building a network with no hidden layers where the output layer is a linear layer. Here, every input neuron connects to every output neuron, referred to as a fully connected network. This network is equivalent to a linear model and helps us understand the fundamentals before adding complexity. We'll use the Torch NN module to build our networks. It makes neural network code more concise and flexible and is conventionally imported as nn. When designing a neural network, the input and output layer dimensions are predefined. The number of neurons in the input layer is the number of features in our data set. And the number of neurons in the output layer is the number of classes we want to predict. Say we create an input tensor of shape one by three. We can think of this as one row with three features or neurons. Next, we pass this input tensor to a linear layer which applies a linear function to make predictions. NN linear takes two arguments. In features is the number of features in our input three and out features is the desired size of the output tensor. In this case, 2. Correctly specifying in features ensures our linear layer can receive the input tensor. Lastly, we pass input tensor to linear layer to generate an output. Notice that the output has two features or neurons due to the out features specified in our linear layer. When input tensor is passed to linear layer, a linear operation is performed to include weights and biases. Each linear layer has a set of associated weights and biases. These are the key quantities that define a neuron. The weights reflect the importance of different features. The bias is an additional term that is independent of the weights, providing the neuron with a baseline output. At first, the linear layer assigns random weights and biases. These are tuned later. Let's Imagine our fully connected network in action. Say we have a weather dataset with three temperature, humidity and wind. We want to predict whether it will rain or be cloudy. The humidity feature will have a more significant weight compared to the other features as it's a strong predictor of rain and clouds. The weather data is for a tropical region with a high probability of rain, so a bias is added to account for this baseline information. With this information, our model makes a prediction. So far we've used one input layer and one linear layer. Now we'll add more layers to help the network learn complex patterns. We'll stack three linear layers using NN Sequential A PyTorch container for stacking layers in sequence. This network takes input, passes it to each linear layer in sequence and returns output. In this case, the layers within NN sequential are hidden layers. Here, the first argument in the first layer represents the number of input features, and the last argument in the final layer represents the number of output classes. Classes both defined by the dataset. We can add as many hidden layers as we like, as long as the input dimension of each layer matches the output dimension of the previous 1. In our three layer example, the first layer takes 10 features and outputs 18. The second layer takes 18 and and outputs 20. Finally, the third layer takes 20 and outputs 5. A layer is fully connected when each neuron links to all neurons in the previous layer, as shown in red in the figure. Each neuron in a linear layer performs a linear operation using all neurons from the previous layer. Therefore, a Single neuron has n 1 learnable parameters, with N being the output dimension of the previous layer plus one for the bias. Increasing the number of hidden layers increases the total number of parameters in the model, also known as model capacity. Higher capacity models can handle more complex datasets but may take longer to train. An effective way to assess a model's capacity is by calculating its total number of parameters. Let's break it down with a two layer network. The first layer has four neurons. Each neuron has eight weights and one bias, which results in 36 parameters. A second layer has two neurons. Each neuron has four weights and one bias for a total of 10 parameters. Adding them together, this model has 46 learnable parameters in total. We can also calculate this in Pytorch using the Numel method. This method returns the number of elements in in a tensor. By looping through the model's parameters and summing the number of elements, we can confirm that the total is also 46. Understanding parameter count helps us balance model complexity and Efficiency. Too many parameters can lead to long training times or overfitting, while too few might limit learning capacity. Welcome back. So far we have seen neural networks made only of linear layers. We can add non linearity to our models using activation functions. We'll discuss two activation sigmoid for binary classification and softmax for multiclass classification. This non linearity allows networks to learn more complex interactions between inputs and targets than only linear relationships. We'll call the output of the last linear layer the pre activation output which we'll pass to activation functions to obtain the transformed output. The sigmoid activation function is widely used for binary classification problems. Lets say we are trying to classify an animal as a mammal or not. We have three pieces of information. The number of limbs, whether it lays eggs and whether it has hair. The latter two are binary variables, 1 if yes and 0 if no. Passing the input to a model with two linear layers returns a single output, the number 6. This number is not yet interpretable. We pass the number six through the sigmoid function, transforming it into a range representing probability between 0 and 1. We are now ready to perform a binary classification. If the output is closer to 1 or 2 or greater than 0.5, we label it as class 1 mammal. If it were less than 0.5 the prediction would be 0, not a mammal. Let's implement sigmoid in PyTorch. Here NN sigmoid takes a one dimensional input tensor of value 6 and returns an output of of the same size, meaning it is also one dimensional. The output is now bounded between 0 and 1. Typically NN sigmoid is added as the last step in NN sequential automatically transforming the output of the final linear layer. Interestingly, a neural network with only linear layers and a sigmoid activation behaves like logistic regression. But adding more layers and activations unlocks the true power of deep learning, which we'll see later. We use softmax, another popular activation function for multiclass classification involving more than two class labels. Let's say we have three Berd0, Mammal1 and Reptile2. In this network, Softmax takes a three dimensional pre activation output and generates an output of the same shape one by three. The output is a probability distribution because each element is between 0 and 1 and the values sum to 1. Here the prediction is for the second class mammals which has the highest probability of 0.842. In PyTorch we use NN softmax. Dim minus 1 indicates that softmax is applied to input Tensor's last dimension. Similar to sigmoid softmax can be the last layer. In nnsequential, we've explored tensors, small networks and activation functions. Now let's dive into generating predictions. This process is called running a forward pass through a network. When input data flows through a neural network in the forward direction to produce outputs or predictions, it passes through each network layer. Calculations transform the data into new representations at every layer, which are passed to the next layer until the final output is produced. The purpose of the forward pass is to pass input data through the network and produce predictions or outputs based on the model's learned parameters, also known as weights and biases. This process is essential for both training and making new predictions. The final output can be binary classifications, multiclass classifications, or numerical predictions regressions. We'll look at an example for each. Say we have input data of five animals with six features or neurons per data point. We create a small network with two linear layers and one sigmoid activation function in sequence. The first layer takes six features as input outputs four and the second layer processes this into a final probability. The output of our binary classification is a single probability between 0 and 1 for each of our five animals. Recall that we commonly use a threshold of 0.5 to turn these probabilities into class labels such as 1 or 0. This output will not be meaningful until we use backpropagation to update layer weights and biases. More on that later. The model would be similar if we wanted to run multiclass classification. Say we are predicting three mammal, bird or reptile. We specify our model has three classes. Setting this value as the last linear layer's output dimension. We use softmax instead of sigmoid. We with dim equals negative one to indicate the five animals have the same last dimension as the last linear layer's output. Using the same input as before, the output shape is 5 by 3. When we print the output, each row represents probabilities for three classes which sum to one. The predicted label for each row is assigned to the class with the highest probability. In our example, the first and second rows are mammals, the third is a reptile, and so on. The last model we'll look at is regression predicting continuous numerical values. We'll now use the same data to predict the animal weights based on their properties. This time there is no activation function at the end, and the last linear layer's last dimension returns an output with one feature. Output dimensions are five by one, five continuous values, one for each row. We've generated predictions by running a forward path. The next step is to see how good our Predictions are compared to the actual values. The loss function, another component of neural networks, tells us how good our model is at making predictions during training. It takes a model prediction y hat and true label or ground truth as inputs and outputs. Afloat Lets use our multiclass classification model that predicts whether an animal is a mammal, bird or reptile. Our dataset contains animal characteristics. This example is a bear. Therefore the class is 0 mammal. If our model predicts that the class equals 0 it it is correct and the loss value will be low. An incorrect prediction would make the loss value high. Our goal is to minimize loss. Loss is calculated using a loss function F, which takes the ground truth y and the prediction Y hat as inputs and outputs a numerical loss value. In our animal example, possible values for our true class of y are integers 0, 1 or 2. Y hat is the raw prediction before applying the softmax function. It is a tensor with the same dimensions as the number of classes n. If n is 3, the softmax output is a tensor of shape one by three. We use one hot encoding to convert an integer Y into a tensor of zeros and ones so we can compare to evaluate model performance. For example, If y equals 0 with three classes, the encoded form is 1,00. We can import torch nn functional as f to avoid manual 1 hot encoding. In the first example, ground truth is 0, the first class. We have three classes. So the function outputs a three element tensor with 1 at the first position and zeros otherwise. If y equals 1, the second class, the output tensor has 1 in the second position and zeros otherwise. Lastly, if y equals 2, the third class, the output tensor has one at the last position and zeros otherwise. With the encoding complete, we can pass it along with our predictions Y hat to a loss function. Here, Y hat is stored as the tensor scores. The most commonly used loss function for classification is cross entropy loss. We start by defining our loss function as criterion. We then pass it the double method of the scores tensor and the one hot target tensor. This ensures tensors are in the correct format. For the loss function, the output is the computed loss value. In summary, the loss function takes the score's tensor as input, which is the model prediction. Before the final softmax function and the one hot encoded ground truth label, it outputs a single float the loss of that sample. Recall that our goal is to minimize this loss. In the next video we'll see how to do that with backpropagation. Excellent job handling those loss functions. Let's now see how we can minimize loss. We know a model Predicts poorly. When loss is high, we can use derivatives or gradients to minimize this loss. Imagine the loss function as a valley. The derivative represents the slope, how steeply the curve rises or falls. Steep slopes shown by red arrows indicate high derivatives and large steps. Gentler slopes, represented by green arrows, have smaller derivatives and smaller steps. On the valley floor shown by the blue arrow, the slope is flat and the derivative is zero. This point is the loss function's minimum, which which we aim to reach. Convex functions have one global minimum. Non convex functions such as loss functions have multiple local minima where the value is lower than nearby points but not the lowest overall. When minimizing loss functions, we aim to locate the global minimum when x is approximately 1. During training, we run a forward pass on the features and compute loss by comparing predictions to the target value. Recall that layer weights and biases are randomly initialized when a model is created. We update them during training using a backward pass or back propagation. In deep learning, derivatives are known as gradients. We compute the loss function gradients and use them to update the model parameters including weights and biases, with backpropagation repeating until the layers are tuned during backpropagation. If we consider a network of three linear layers, we can calculate local loss gradients for with respect to each layer's parameters. We first calculate loss gradients with respect to L2 parameters to L2 and L1 and repeat until we reach the first layer. Let's see this with Pytorch. After running a forward pass, we define a loss function here, cross entropy loss and use it to compare predictions with target values. Using dot backward we calculate gradients based on this loss which are stored in the grad attributes of each layer's weights and biases. Each layer in the model can be indexed starting from zero to access its weights, biases and gradients. To manually update model parameters, we access each layer gradient, multiply it by the learning rate and subtract this product from the weight. Learning rate is another tunable parameter. We'll discuss this and the training loop later in the course. We use a mechanism called gradient descent to find the global minimum of loss functions. Pytorch simplifies this with optimizers like stochastic gradient descent. We use optin to instantiate SGD dot parameters returns an iterable of all model parameters which we pass to the optimizer. We use a standard learning rate LR Here. The optimizer automatically calculates gradients and updates model parameters with dot step magic. It's time to train our neural network. Efficient data handling is key to training deep learning models. Our animal classification data is in a CSV file and can be loaded using PD read C CSV. We'll use hair, feathers, eggs, milk, predator, legs and tail as features to predict an animal's type. The animal name column isn't needed since names don't determine classification. Note the type column has three bird zero, mammal and reptile two. We'll use iLock to select all columns except the first and last type, giving us our input features. These are converted into a numpy array for easier handling with Pytorch. Similarly, we can extract the last column type and store it as an array of our target values, which represent the class labels for each animal. We'll call this Y. We'll use Tensor dataset To prepare data for Pytorch models, we first import Torch and Tensor Dataset from Torch Utils data. This allows us to store our features X and target labels Y as tensors, making them easy to manage. We convert X and Y into tensors using Pytorch's Tensor method and pass them to Tensor Dataset. To access an individual sample, we use square bracket indexing. Dataset 0 returns a tuple containing the input features and label, which we unpack into Input sample and Label sample. Once we've created our dataset using Tensor Dataset, we can pass it to DataLoader to efficiently manage data loading during training. We start by importing dataloader from Torch Utils data. Next, we define two key parameters. Batch size determines how many samples are included in each iteration. Since deep learning models require large datasets, batching helps process multiple samples at once, making training more efficient. Shuffle randomizes the data order at each epoch, helping improve model generalization. One epoch is a full pass through the training data loader, and generalization means the model performs well on unseen data rather than just memorizing the training set. We then create a dataloader instance with these parameters, making it easy to iterate through our dataset in batches. Let's iterate through dataloader. Each element in the dataloader is a tuple, which we unpack as batch inputs and batch labels. Since our dataset contains five animals and we set a batch size of two, the first iteration randomly selects two animals and their corresponding labels. On the second iteration, two more samples are randomly selected with their labels. Finally, the last remaining sample is returned. Since our dataset has an odd number of samples, the last batch contains just one item. In real world, deep learning datasets are much larger with batch sizes of typically 32 or more. For better computational efficiency, we now have the core components to train a Pytorch deep learning model. Once we create a model, choose a loss function, define a dataset and set an optimizer, we're ready to train. This involves looping through the data set, calculating the loss, computing gradients and updating model parameters. This process, called the training loop, repeats multiple times. A training loop allows for greater flexibility and control, giving us the option to customize different elements. We'll work with a data set of data scientist salaries to see this in action. Features are categorical and the target is salary in US dollars already normalized. Since the target is a continuous value, this is a regression problem. For regression, we'll use a linear layer as the final output in the instead of softmax or Sigmoid. Additionally, we'll apply a regression specific loss function. As cross entropy is only used for classification tasks, we can use mean squared error loss for regression problems. The MSE loss is the mean of the squared difference between predictions and ground truth as shown in this Python implementation. In Pytorch, we use the NN MSELOSS function as a criterion. Note that both predictions and targets must be float tensors let's put everything together now we have two NumPy arrays, features and target containing our data and labels. We start by passing these to Tensor dataset to organize our features and targets into the right data types floats. This casting is required and float is the data type used by the parameters of our model. We can now load our dataset into the dataloader class to enable batching. Here we use a small batch size of 4, but selection of batch size is customizable depending on the use case. We create our model next. This dataset has four input features and one target output. We won't need one hot encoding as this is a regression problem. Finally, we create the MSE loss criterion and the optimizer. A 0.001 learning rate is a good default learning rate for most deep learning problems. We can now loop through the dataset multiple times. Looping through the entire dataset once is called an epoch and we train over multiple epochs and indicated by numepochs. For each epoch we loop through the data loader. Each iteration of the dataloader provides a batch of samples which we saw earlier before the forward pass. We set the gradients to 0 using optimizer 0 grad. Because the optimizer stores gradients from previous steps, by default we get features and targets from each sample of the data loader. We use the features for the forward pass of the model and we use the target for the loss calculation. Finally, we use the optimizer to update the parameters of the model. We've seen how activation functions introduce non linearity to help neural networks learn complex patterns, and we've learned about gradients and their role within the training loop. Some activation functions can shrink gradients too much, making training inefficient. So far we've worked with two activation functions, sigmoid and softmax, which are typically used in a model's final layer. We'll begin by understanding some of the limitations of the sigmoid function. The sigmoid's outputs are bounded between 0 and 1, meaning that for any input the output will always fall in this range. Sigmoid could be used at any point in a network. However, the gradients of the sigmoid shown in orange are very small. For large and small values of x, this phenomenon is called saturation. During backpropagation, this becomes problematic because each gradient depends on the previous one. When gradients are extremely small, they fail to update the weights effectively. This issue is known as the vanishing gradients problem and it can make training deep networks very difficult. The softmax function, which also produces bounded outputs between 0 and 1, suffers from saturation in a similar way. Therefore, both of these activation functions are not ideal for hidden layers and are best used in the last layer only. We'll discover two widely used activation functions designed for use between linear layers or in hidden layers. Here is the rectified linear unit, or ReLU. ReLU outputs the maximum value between its input and zero as shown in the graph. For positive inputs the output equals the input. For negative inputs the output is zero. This function has no upper bound and its gradients do not approach zero for large values of x, which helps overcome the vanishing gradients problem. In Pytorch, RELU can be used through the Torch NN module. It's a reliable default activation function for many deep learning tasks. The leaky RELU is a variation of the RELU function. For positive inputs, it behaves exactly like relu. For negative inputs, it multiplies the input by a small coefficient defaulted to 0.01 in PyTorch. This ensures the gradients for negative inputs remain non zero, preventing neurons from completely stopping learning, which can happen with standard relu. In PyTorch, the leaky RELU function is implemented using the Torch NN module. The negative slope parameter controls the coefficient applied to negative inputs. Learning rate has come up a few times, the time has come for us to dig deeper. Training a neural network means solving an optimization problem by minimizing the loss function and adjusting model parameters. To do this, we use an algorithm called stochastic gradient descent or SGD, implemented in PyTorch Recall. This is the optimizer we used to find the global minimum of loss functions. The optimizer takes the model's parameters along with two key learning rate, which controls the step size of updates, and momentum, which adds inertia to help the optimizer move smoothly and avoid getting stuck. Understanding their impact helps us optimize efficiency. Let's try to find the minimum of a U shaped function. We start at X minus 2 and run the SGD optimizer for 10 steps. After these steps, we observe that the optimizer is close to the minimum. We can also note that as we approach the minimum, the step size gradually decreases. This happens because the step size is the gradient multiplied by the learning rate. Since the function is less steep near zero, the gradient and thus the step size gets smaller. However, if we use the same algorithm for a learning rate 10 times smaller, we realize that we are still far from the minimum of the function. After 10 steps, the optimizer will take much longer to find the function's minimum. If we use a high value for the learning rate, we observe that the optimizer cannot find the minimum and bounces back and forth on both sides of the function. Recall that loss functions are non convex. One of the challenges when trying to find the minimum of a non convex function is getting stuck in a local minimum. Let's run our optimizer for 100 steps with a null momentum on this non convex function, we see that the optimizer gets stuck in this first dip of the function, which is not its global minimum. However, when using a momentum of 0.9, we can find the minimum of the function. This parameter provides momentum to the optimizer, enabling it to overcome local dips. As shown here. The momentum keeps the step size large when previous steps were also large, even if the current gradient is small. In summary, two key optimizer parameters impact training, learning rate and momentum. The learning rate controls the step size, and typical values range from 0.01 to 0.0001. If it's too high, the optimizer may not find the minimum. If it's too low, training slows down. Momentum helps the optimizer move past the local minimum. Without it, the optimizer may get stuck. Here, typical values range from 0.85 to 0.99. We've explored how neural networks learn by updating weights during training. This final chapter will look at techniques to evaluate and improve model performance efficiency. Before we get started, please note the topics here are more advanced and we'll cover them at a high level. Data normalization scales input features for stability. Similarly, the weights of a linear layer are also initialized to small values. This is known as layer initialization. Let's create a small linear layer and check its weight range. We observe that the weights are between -0.125 and positive 0.125. Why is this important? The output of a neuron in a linear layer is a weighted sum of inputs from the previous layer. Keeping both the input data and layer weights small ensures stable outputs, preventing extreme values that could slow training. Layers can be initialized in different ways, and it remains an active area of research. Pytorch provides an easy way to initialize layer weights with the nn.in IT module. For example, here we initialize a linear layer with a uniform distribution. As you can see, the weights values now range from 0 to 1. In practice, engineers are rarely training a model from randomly initialized weights. Instead, they rely on a concept called transfer learning. Transfer learning takes a model that was trained on a first task and reuses it for a second task. For example, we trained a model on US Data scientist salaries. We now have new data of salaries in Europe. Instead of training a model using randomly initialized weights, we can load weights from the first model and use them as a starting point to train on this new dataset. Saving and loading weights can be done using the Torch save and the torch load functions. These functions work on any type of Pytorch objects. Sometimes the second task is similar to the first task, and we want to perform a specific type of transfer learning called fine tuning. In this case, we load weights from a previously trained model, but train the model with a smaller learning rate. We can even train part of a network if we decide some of the network layers do not need to be trained and choose to freeze them. A rule of thumb is to freeze early layers of the network and fine tune layers closer to the output layer. This can be achieved by setting each parameter's requiresgrad attribute to false. Here we use the model's named parameters method, which returns the name and the parameter itself. We set requiresgrad on the first layer's weight to false. We've done a lot of training. Now let's evaluate our models. In machine learning, data is split into training, validation and test sets. Training Data adjusts the model's parameters, such as weights and biases. Validation data tunes hyperparameters like learning rate and momentum, and the test set evaluates the model's final performance. We'll track two key metrics, loss and accuracy during training and validation. Let's begin with loss. Training loss is calculated by summing the loss across all batches in the training data loader. At the end of each epoch, we compute the mean training loss by dividing the total loss by the number of batches. We begin by setting training loss to zero. We iterate through the train loader, run a forward pass and compute the loss. As usual, the model computes gradients and updates weights using backpropagation. We add each loss value to the total using dot item, which extracts the numerical value from a tensor. Since one epoch is a full pass through the training data loader, and we compute the mean loss by dividing training loss by the number of batches in the train loader. After each training epoch, we run a validation loop. First, we set the model to evaluation mode using Eval, as some layers behave differently during training and validation. To improve efficiency, we use Torch Nograd, which disables gradient calculations since we don't update weights during validation. We then iterate through the validation data loader, run a forward pass and compute the loss, summing it across batches. At the end of the epoch, we calculate the mean validation loss. Finally, we switch the model back to training mode with train, preparing it for the next training epoch. Keeping track of training and validation loss helps us detect overfitting. When a model overfits, training loss keeps decreasing, but validation loss starts to rise. This means the model is learning the training data too well and won't perform well on new data. Loss tells us how well a model is learning, but it doesn't always reflect how accurately it makes predictions. To measure that, we track accuracy using torch metrics. For multiclass classification tasks, we create an accuracy metric with torchmetrics accuracy. As the model processes each batch, we update this metric using its predictions and the actual labels. Since the model outputs probabilities for multiple classes, we use Argmax DIM minus 1 to select the class with the highest probability. This converts one hot encoded predictions into class indices before passing them to the metric. At the end of each epoch, we calculate the overall accuracy using dot computer. Finally, we reset the metric with reset to clear its state before the next epoch. This process is the same for both training and validation. Previously, we learned how to detect overfitting by looking at the training and validation losses in this video, we'll discover a few ways we can fight overfitting. Recall that overfitting happens when the model does not generalize to unseen data. If we do not train the model correctly, it will start to memorize the training data, which leads to good performance on the training set but poor performance on the validation set. Several factors can lead to a small data set, a model with too much capacity, or large values of weights. To counter overfitting, we can reduce the model size or add a new type of layer called dropout. We can also use weight decay to force the parameters to remain small. We can get more data or use data augmentation. Let's explore these strategies. A common way to fight overfitting is to add dropout layers to our neural network. Dropout is a regularization technique which randomly deactivates a fraction of neurons during training, preventing the model from becoming too dependent on specific features. Dropout layers are typically added after activation functions. The p argument determines the probability of a neuron being set to zero. In this example, 50% of the neurons are dropped. Dropout behaves differently during training and evaluation. During training, it randomly deactivates neurons, while during evaluation it is disabled, ensuring all neurons are active. For stable predictions. To switch between these modes, we use model train and model eval. The next strategy to reduce overfitting we will discover is weight decay, another form of regularization in Pytorch. Weight decay is added to the optimizer using the weight decay parameter, typically set to a small value, e.g. 0.001. This parameter adds a penalty to the loss function, encouraging smaller weights and helping the model generalize better. During backpropagation, this penalty is subtracted from the gradient, preventing excessive weight growth. The higher we set the weight decay, the stronger the regularization, making overfitting less likely. Collecting more data can be expensive, but researchers have found a way to expand datasets artificially using data augmentation. Data augmentation is commonly applied to image data, which can be rotated and scaled so that different views of the same face become available as new data points. While we won't discuss how to augment data here, it remains a valuable method for combating overfitting when additional data isn't available. In our final video, we'll bring everything together and learn a recipe for tackling any deep learning problem. First, we create a model that can overfit the training set. This will ensure that the problem is solvable. We also set a performance baseline to aim for with the validation set. We then need to reduce overfitting to increase performance on the validation set. Finally, we can slightly adjust the different hyperparameters to ensure we achieve the best possible performance. It's useful to start with a single data point before overfitting the entire training set. This ensures the problem is solvable and helps catch potential bugs. We modify the training loop to repeatedly train on a single example rather than iterating through the entire data loader. When the model is set up properly, it should quickly reach near 0 loss and 100% accuracy on that data point. Once this step is successful, we scale up to the entire training set. At this stage, we use an existing model architecture large enough to overfit while keeping hyper parameters like the learning rate at their defaults. Now we need to create a model that generalizes well to maximize the validation accuracy. We know a few strategies to reduce overfitting, such as including dropout layers, data augmentation, weight decay, or reducing the model capacity. We need to keep track of the different parameters and the corresponding validation accuracy for each set of experiments. Reducing overfitting often comes at a cost, as applying regularization can significantly impact model performance. The original model overfits the training set, achieving high accuracy but failing to generalize well to new data. In contrast with too much regularization, the updated model shows a drop in training and validation accuracy, limiting its ability to learn effectively. This highlights the importance of balancing overfitting reduction strategies while closely monitoring key metrics to find the best performing model. Once we're satisfied with performance, the final step is fine tuning hyperparameters. This is often done on optimizer settings like learning rate or momentum. Grid Search tests parameters at fixed intervals, for example, momentum values from 0.85 to 0.99 and learning rates from 10 to the power of negative 2 to 10 to the power of negative 6. Random search takes a different approach. Instead of testing set values, it randomly selects them within a given range. The MP random uniform function, for example, picks a number between two and six, allowing us to explore a wider variety of learning rates. Random search is often more efficient as it avoids unnecessary tests and increases the chance of finding optimal settings. This introduction to deep learning in Pytorch is coming to an end. You've learned so much and I hope you've enjoyed this course. In the first chapter you discovered deep learning, learned how to create neural networks, learned all about linear layers. In the second chapter you were introduced to more neural network components like activation functions and loss functions. You used Pytorch to calculate derivatives and used the backpropagation algorithm. In the third chapter you trained your first neural network and learned the impact of the learning rate and momentum. Finally, in the last chapter you learned all about improving and evaluating your model to reduce overfitting and more. Here are some courses you can take next to continue on your learning journey. If you want to push further and expand your deep learning knowledge, I suggest strengthening your understanding of probabilities, linear algebra and calculus. Practice is essential here, so remember to apply your skills. Here is a hands on project in Datalab, your playground for deep learning and machine learning challenges. Finally, take your expertise even further by using your skills to create something tangible like a smartphone app or an innovative project of your own.

Welcome. My name is Michal Olesvak and I will be your instructor for this course. We will learn how to train robust deep learning models in Pytorch. We will cover how to improve the training process with optimizers, mitigate frequent training issues of vanishing and exploding gradients, build convolutional neural networks for image data, recurrent neural networks for sequence data, and models with multiple inputs and outputs. Before starting, you should have a basic understanding of the training process for neural networks including forward pass loss computation and back propagation. You should also be able to train and evaluate basic models in Pytorch using datasets and dataloaders as taught in the following course. Object Oriented Programming, or oop, is used throughout the Pytorch community to define objects. With more flexibility, we will use it to define Pytorch datasets and models. OOP is a way of writing computer programs where we create objects like virtual entities, each with abilities called methods and data called attributes. Take this class called Bank Account. We implement the INIT method inside the class definition which is called when the bank account object is created, it is written with two underscores on either side, takes two arguments self and balance and assigns balance to the object itself using self balance. This way the balance becomes the object's attribute. We can now create a new account object with a balance of 100 by calling bank account with argument 100 and assigning it to account. We can then access it with account balance. Classes can also have methods. These are functions to perform various tasks. We'll add a deposit method that updates the account's balance. It's written like a normal Python function with the addition of self. We can now use this method by calling account.deposit to increase the balance. In this chapter we will be working with the Water Potability dataset. The task is to classify a water sample as potable or drinkable 1 or 0 based on its chemical characteristics. All features have been normalized to be between 0 and 1. To train a model, we need to build a Pytorch dataset, set up a dataloader and define the model. Let's build a custom dataset for our water potability data using oop. We start with the INIT method, which reads a CSV file into a dataframe and stores it in the data attribute as an numpy array. The super INIT command ensures our water dataset class behaves like its parent class and storage dataset. Next, Pytorch requires us to implement the LEN method that returns the total size of the dataset, which we access as the zeroth element of DataFrame's shape. Finally, we add the GetItem method, which takes one argument called IDX, the index of a sample, and returns the features, all columns but the last one, and the label the final column for that sample. With the water dataset class defined, we create an instance of the dataset, passing it the training data file path. Then we pass the dataset to the Pytorch dataloader, setting the batch size to two and shuffling the training samples randomly. We use the next ITER combination to get one batch from the data loader. With a batch size of two, we get two samples, each consisting of nine features and a target label. Pytorch models are also best defined as classes. We may have seen sequential models defined like this before. That's fine for small models, but using classes gives us more flexibility to customize as complexity grows. We can rewrite this model using OOP. The net class is based on the NN module, PyTorch's base class for neural networks. We define the model layers we want to use in the INIT method. The forward method describes what happens to the input when passed to the model. Here we pass it through subsequent layers that we defined in the INIT method and wrap each layer's output in the activation function. Welcome back. Let's look at model training and evaluation and the optimizer's role in the training process. Let's review the Pytorch training loop. First we define the loss function, conventionally called criterion, and the optimizer. We'll use binary cross entropy or BCE loss, commonly used for binary classification tasks. We use stochastic gradient descent or sgd, as the optimizer and tell it which parameters to optimize. Here it's all of NET's parameters. Then we pass it the learning rate of 0.01. We start the loop by iterating over epochs and batches of training data. Next, we clear the gradients to start from zero for the new batch, followed by a forward pass to get the model's outputs. Then we compare the model's outputs to the ground truth labels to compute the loss. We reshape the labels with the view method to match the shape of the outputs. We compute the gradients of the model's parameters for the loss using the backward method. These gradients contain information about the direction and size of the changes required to minimize the loss. Finally, we pass the gradients to the optimizer, which performs an optimization step, that is, it updates the values of the model's parameters based on the gradients. Let's take a closer look at the optimization step. In practice, neural networks can have billions of parameters. Let's consider an example with only two Imagine we have the following parameter values and gradients. They are passed to the optimizer, which computes an update for each parameter. The updates are applied to the parameters and the optimizer step is finished. But how does the optimizer know how much to update and in which direction? The direction depends on the gradient sign. The first parameter, for example, has a positive gradient, so it should be decreased in order to decrease the loss. Hence, the parameter update is negative. What about the size of the update? Different optimizers use different approaches to decide how much to update. In stochastic gradient descent, or sgd, the size of the parameter update depends only on the learning rate. A predefined hyperparameter SGD is computationally efficient, but because of its simplicity, it's rarely used in practice. Using the same learning rate for each parameter cannot be optimal. Adaptive gradient or adagrad improves on it by decreasing the learning rate during training for parameters that are infrequently updated. This makes it well suited for sparse data, I.e. data in which some features are not often observed. However, adagrad tends to decrease the learning rate too fast. Root mean square propagation, or RMS prop, addresses adagrad's aggressive learning rate decay by adapting the learning rate for each parameter based on the size of the previous gradients. Finally, Adaptive moment estimation, or adam, is arguably the most versatile and widely used optimizer. It combines RMSprop with the concept of momentum, the average of past gradients where the most recent gradients have more weight. Basing the update on both gradient size and momentum helps accelerate training. ADAM is often the default go to optimizer and we will use it throughout the course. Once the model is Trained, we can evaluate its performance on test data. First we set up the binary accuracy metric from TOSH metrics. Then we put the model in the evaluation mode with net evaluation and iterate over the data loader test with no gradients calculation and do the forward pass to get predicted probabilities, which we then transfer into predicted labels based on the 0.5 threshold. Finally, we update the accuracy score. Let's compute and print the overall accuracy. We got over 67%. Not bad for a basic model and small dataset. Welcome back. Neural networks often suffer from gradient instability during training. Sometimes their gradients get smaller during the backward pass. This is known as vanishing gradients. As a result, earlier layers receive hardly any parameter updates and the model doesn't learn. In other cases, the gradients get increasingly large leading to huge parameter updates and divergent training. This is known as exploding gradients. To address these problems, we need a three step solution consisting of proper weight initialization, good activations and batch normalization. Let's review these steps. Whenever we create a torch layer, its parameters stored in the weight attribute get initialized to random values to prevent unstable gradients. Research showed that initialization should ensure that the variance of the layer's inputs is close to that of its outputs and the variance of the gradients is the same before and after passing through the layer. The way to achieve this is different for each activation function. For RELU or rectified linear unit and similar activations, we can use he initialization, also known as kaiming initialization. To apply this initialization, we call kaiminguniform from torch nn.in it on the layer's weight attribute. This ensures the desired variance properties. To implement it, we need one small change in our model's INIT method For each layer, we call kaminguniform on its weight attribute. For the last layer, where we use sigmoid activation in the forward method, we also specify nonlinearity as sigmoid. This is what it looks like within the full model. Lets discuss activation functions now. The ReLU, or rectified linear unit, is arguably the most commonly used activation. It's available as NN Functional ReLU. It has several advantages but also an important drawback. It suffers from the dying neuron problem. During training, some neurons only output 0. This is caused by the fact that the RELU is zero for any negative value. If inputs to a neuron become negative, it effectively dies. The LU, or exponential linear unit is one activation designed to improve upon ReLU. It's available as NN functional L thanks to nonzero gradients for negative values, it doesn't suffer from the dying neurons problem. Additionally, its average output is near zero, so it's less prone to vanishing gradients. A good choice of initial weights and activation functions can alleviate unstable gradients at the beginning of training, but it doesn't prevent them from returning during training. A solution to this is batch normalization. Batch normalization is an operation applied after a layer in which the layer's outputs are first normalized by subtracting the mean and dividing by the standard deviation. This ensures the output distribution is roughly normal. Then the normalized outputs are scaled and shifted using shift and scale parameters that the batch normalization learns. Just like linear layers learn their weights effectively, BatchNorm allows the model to learn the optimal distribution of inputs to each layer before it is applied. This speeds up the loss decrease and makes it more immune to unstable gradient issues. To add batch normalization to a Pytorch model, we must define the batchNorm layer using nn batchNorm1d in the init method. Here we call it bn1. We pass it the input size, which needs to be equal to the preceding layer's output size, in this case 16. Then in the forward method, we pass the linear layer's output to the batchNorm layer and pass the result to the activation function. Welcome back. Let's learn about deep learning with image data. We will work with the Clouds dataset from Kaggle, containing photos of seven different cloud types. We'll build an image classifier to predict the cloud from an image. But first, what is an image? Digital images are comprised of pixels, short for picture elements. A pixel is the smallest unit of the image. It's a tiny square that represents a single point. If we zoom in into this cloud picture, we can see the pixels. Each pixel contains numerical information about its color. In a grayscale image, each pixel represents a different shade of gray, ranging from black to white, which would be an integer between 0 and 255, respectively. A value of 30, for example, represents the following shade of gray. In color images, each pixel is typically described by three integers denoting the intensities of the three color channels red, green and blue. For example, a pixel with red of 52, green of 171, and blue of 235 represents the following shade of blue. Let's build a Pytorch dataset of cloud images. This is easiest with the specific directory structure. We have two main folders called cloudtrain and cloudtest. Within each, there are seven directories, each representing a cloud type or one category. In our classification task, we have JPG image files inside each category folder. With this directory structure, we can use image folder from TorchVision to create a dataset. First we need to define the transformations to apply to an image as it is loaded. To do this, we call transforms compose and pass it a list of transformations. We parse the image to a torch tensor with two tensor and resize it to 128 by 128 pixels to ensure all images are the same size. Then we create a dataset using image folder, passing it the training datapath and the transforms we defined. Dataset Train is a pytorch dataset just like the water dataset we saw before. We can create the dataloader from it and get a data sample. Notice the shape of the loaded image. 1x3x128x128 1 corresponds to the batch size of 13 to the three color channels, and 128x128 represents the image's height and width. To display a color image like this, we must rearrange its dimensions so the height and width come before the channels. We call squeeze on the image to eliminate the one dimensions of the batch size and then permute the order by replacing the original order of dimensions 012 with 1 to 0. This way we place the channel dimension at the end. For grayscale images, this permutation is not needed. This lets us call plt imshow from matplotlib followed by plt show to display the image. Recall the dataset building code. We said that upon loading, one can apply transformations to the image, such as resizing. But many other transformations are possible too. Let's add a random horizontal flip and rotate by a random degree between 0 and 45. Adding random transformations to the original image is a common technique known as data augmentation. It allows us to generate more data while increasing the size and diversity of the training set. It makes the model more robust to variations and distortions commonly found in real world images and reduces overfitting as the model learns to ignore the random transformations. Here's a sample of augmented images using rotation. Welcome. Let's discuss neural networks for image processing. Let's start with a linear layer. Imagine a grayscale image of 256x256 pixels. It has over 65,000 model inputs. Using a layer with 1000 neurons, which isn't much, would result in over 65 million parameters. For a color image with three times more inputs, the result is over 200 million parameters in just the first layer. This many parameters slows down training and risks overfeeding. Additionally, linear layers don't recognize spatial patterns. Consider this image with a cat in the corner. Linearly connected neurons could learn to detect the cat, but the same cat won't be recognized if it appears in a different location. When dealing with images, a better alternative is to use convolutional layers. In a convolutional layer, parameters are collected in one or more small grids called filters. These filters slide over the input, performing convolution operations at each position to create a feature map. Here we slide a 3x3 filter over a 5x5 input to get a 3x3 feature map. A feature map preserves spatial patterns from the input and uses fewer parameters than a linear layer. In a convolutional layer, we can use many filters. Each results in a separate feature map. Finally, we apply activations to each feature map. All the feature maps combined form the output of a convolutional layer. In pytorch, we use NN conf2d to define a convolutional layer. We pass in the number of input and output feature maps here, arbitrarily chosen 3 and 32 and the kernel or filter size 3. Let's look at the convolution operation in detail. In the context of deep learning, a convolution is the dot product between two arrays. The input patch and the filter dot product is element wise multiplication between the corresponding elements. For instance, for the top left field, we multiply one from the input patch with two from the filter. To get two, we sum all values in the outcome array, returning a single value that becomes part of the output feature map. Before a convolutional layer processes its input, we often add zeros around it, a technique called zero padding. This is done with the padding argument in the convolutional layer. It helps maintain the spatial dimensions of the input and output and ensures equal treatment of border pixels. Without padding, the pixels at the border would have a filter slide over them fewer times, resulting in information loss. Max pooling is another operation commonly used after convolutional layers. In it, we slide a non overlapping window marked by different colors here over the input. At each position, we select the maximum value from the window to pass forward. For example, for the green window position, the maximum is 5. Using a window of 2x2 as shown here halves the input's height and width. This operation reduces the spatial dimensions of the feature maps, reducing the number of parameters and computational complexity in the network. In pytorch, we use NN MaxPool 2D to define a max pooling layer, passing it the kernel size. Let's build a convolutional network. It will have two a feature extractor and a classifier. Feature extractor has convolution, activation and max pooling layers repeated twice. The first two arguments in conf2d are the numbers of input and output feature maps. The first conf2d has three input feature maps corresponding to the RGB channels. We use filters of size 3x3 set by the kernel size argument and zero padding by setting padding to 1. For max pooling, we use the MaxPool 2D layer with a window of size 2 to half the feature map in height and width. Finally, we flatten the feature extractor output into a vector. Our classifier consists of a single linear layer. We will discuss how we got its input size shortly. The output is the number of target classes, the model's argument. The forward method applies the extractor and classifier to the input image to determine the feature extractor's output size. We'll start with the input Image's size of 3x64x64. The first convolution has 32 output feature maps, increasing the first dimension to 32 0. Padding doesn't affect height and width. Max pooling cuts height and width in two. The second convolution again increases the number of feature maps in the first dimension to 64, and the last pooling halves height and width, again giving us 64 by 16 by 16. Welcome back. In this video, we will train the cloud classifier before we proceed to the training itself. However, let's take one more look at data augmentation and how it can impact the training process. Say we have this image in the training data with the associated label cat. We apply some augmentations, for example Rotation and horizontal flip, to arrive at this augmented image, and we assign it the same cat label. Both images are part of the training set. Now, in this example, it is clear that the augmented image still depicts a cat and can provide the model with useful information. However, this is not always the case. Imagine we are doing fruit classification and decide to apply a color shift augmentation to an image of the lemon. The augmented image will still be labeled as a lemon, but in fact it will look more like a lime. Another example classification of handwritten characters. If we apply the vertical flip to the letter W, it will look like the letter M. Passing it to the model labeled as W will confuse the model and impede training. These examples show that sometimes specific augmentations can impact the label. It's important to notice that an augmentation could be confusing. Depending on the task, we could apply the vertical flip to the lemon or the color shift to the letter W without introducing noise in the labels. Remember to always choose augmentations with the data and task in mind. So what augmentations would be appropriate for our cloud classification task? We will use three augmentations. Random rotation will expose the model to different angles of cloud formations. Horizontal flip will simulate different viewpoints of the sky. Automatic contrast adjustment simulates different lighting conditions and improves the model's robustness to lighting variations. We have already used the random horizontal flip and random rotation transforms to include a random contrast adjustment. We will add the random autocontrast function to the list of transforms. In the clouds DataSet we have seven different cloud types, which means that is a multi class classification task. This calls for a different loss function than we have used before. The model for water potability prediction we built before was solving a binary classification task for which the BCE or binary cross entropy loss funct is appropriate. For multiclass classification we will need to use the cross entropy loss. It's available in Pytorch as NN crossentropyloss. Except for the new loss function, the training loop looks the same as before. We instantiate the model we have built with seven classes and set up the cross entropy loss and the ADAM optimizer. Then we iterate over the epochs and training batches and perform the usual sequence of steps for each batch. It's time to evaluate our cloud classifier. First we need to prepare the dataset and dataloader for test data. But what about data augmentation? Previously we defined the training dataset, passing it training transforms including our augmentation techniques. For test data, we need to define separate transforms. Without data augmentation, we only keep parsing to tensor and resizing. This is because we want the model to predict a specific test image, not a random transformation of it. Previously we evaluated a model based on its accuracy which looks at the frequency of correct predictions. Let's review other metrics. In binary classification, precision is the fraction of correct positive predictions, while recall is the fraction of all positive examples that were correctly predicted. For multiclass classification, we can get a separate recall and precision score for each class. For example, precision of the cumulus cloud class will be the fraction of cumulus predictions that were correct and the recall for the cumulus cloud class will be the fraction of all cumulus cloud examples that were correctly predicted by the model. With seven cloud classes, we have seven precision and seven recall scores. We can analyze them individually for each class or aggregate them. There are three ways to do Microaverage calculates the precision and recall globally by counting the total true positives, false positives, and false negatives across all classes. It then computes the precision and recall using these aggregated values. Macro average computes the precision and recall for each class independently and takes the mean across all classes. Each class contributes equally to the final result regardless of its size. Weighted average calculates the precision and recall for each class independently and takes the weighted mean across all classes. The weight applied is proportional to the number of samples in each class. Larger classes have a greater impact on the final result. In Pytorch, we specify the average type when defining a metric. For example, for recall, we pass average as none to get seven recall scores, one for each class. Or we can set it to micro macro or weighted. But when do we use each of them if our dataset is highly imbalanced? Microaverage is a good choice because it takes into account the class imbalance. Macro averaging treats all classes equally, regardless of their size. It can be a good choice if you care about performance on smaller classes, even if those classes have fewer data points. Weighted averaging is a good choice when class imbalance is a concern and you consider errors in larger classes as more important. We start the evaluation by importing and defining precision and recall metrics. We will use macro averages for demonstration. Next, we iterate over test examples with no gradient calculation. For each test batch, we get model outputs, take the most likely class and pass it to the metric functions along with the labels. Finally, we compute the metrics and print the results. We got a recall higher than precision, meaning the model is better at correctly identifying true positives than avoiding false positives. Note that using larger images, more convolutional layers, and a classifier with more than one linear layer could improve both metrics. Sometimes it is informative to analyze the metrics per class to compare how the model predicts specific classes. We repeat the evaluation loop with the metric defined with average none. This time we only compute the recall. We get seven scores, one per class. But which score corresponds to which class? To learn this, we can use our dataset's classtoidx attribute, which maps class names to indices. We can use a dictionary comprehension to map each class name k to its recall score by indexing the list of all scores called recall with the vclass index from the classtoidx method. This will be a tensor of length one, so we call dot item on it to turn it into a scalar. Looking at the results, A recall of 1.0 indicates that all examples of Clear sky have been classified correctly, while high cumuliform clouds were harder to classify and have the lowest recall score. We've learned how to handle tabular and image data. Let's now discuss sequential data. Sequential data is ordered in time or space where the order of the data points is important and can contain temporal or spatial dependencies between them. Time series data recorded over time, like stock prices, weather or daily sales, is sequential. So is text in which the order of words in a sentence determines its meaning. Another example is audio waves, where the order of data points is crucial to the sound we produced when the audio file is played. In this chapter we will tackle the problem of predicting electricity consumption based on past patterns. We will use a subset of the electricity consumption dataset from UC Irvine Machine Learning Repository. It contains electricity consumption in kilowatts or kw for a certain user recorded every 15 minutes for four years. In many machine learning applications, one randomly splits the data into training and testing sets. However, with sequential data there are better approaches. If we split the data randomly, we risk creating a look ahead bias where the model has information about the future when making forecasts. In practice, we won't have information about the future when making predictions, so our test set should reflect this reality. To avoid the lookahead bias, we should split the data by time. We will train on the first three years of data and test on the fourth year. To fit the training data to the model, we need to chunk it first to create sequences that the model can use as training examples. First we need to select the sequence length, which is the number of data points in one training example. Let's make each forecast based on the previous 24 hours. Because data is at 15 minute intervals, we need to use 24 times 4, which is 96 data points. In each example, the data point right after the input sequence will be the target. To predict, let's implement a python function to create sequences. It takes the dataframe and the sequence length as inputs. We start with initializing two empty X's for inputs and Y's for targets. Next, we iterate over the dataframe. The loop only goes up to lenvf sequencelength, ensuring that for every iteration there are always sequence length data points available in the dataframe for creating the sequence and a subsequent data point to serve as the target for each considered data point, we define inputs x as the electricity consumption values starting from this point plus the next sequence length points and the target Y as the subsequent electricity consumption value. The one passed to the ILOC method stands for the second DataFrame column which stores the electricity consumption data. Finally, we append the inputs and the targets to pre initialized lists and after the loop return them as NUMPY arrays. Let's use our function to create sequences from the training data. This gives us almost 35,000 training examples. To convert them to a TORCH dataset, we will use the tensordataset function. We pass it two arguments, the inputs and the targets. Each argument is the NUMPY array converted to a tensor with torch from numpy and parsed to a float. The tensor dataset behaves just like all other TORCH datasets and it can be passed to a dataloader in the same way. Everything we have learned here can also be applied to other sequential data. For example, large language models are trained to predict the next word in a sentence, a problem similar to predicting the next amount of electricity used for speech recognition, which means transcribing an audio recording of someone speaking to text. One would typically use the same sequence processing model architectures we will learn about soon. It's time to discuss recurrent neural networks. So far we built feedforward neural networks where data is passed in one direction from inputs through all the layers to the outputs. Recurrent neural networks, or RNNs are similar but also have connections pointing back at each time step. A recurrent neuron receives some input X multiplied with the weights and pass through an activation. Out come two values, the main output Y and the hidden state H that is fed back to the same neuron. In Pytorch, a recurrent neuron is available as NN rnn. We can represent the same neuron once per time step. A visualization known as unrolling a neuron through time. At a given time step, the neuron represented as a gray circle Receives input data x0 and the previous hidden state h0 and produces output y0 and a hidden state h1. At the next time step, it takes the next value x1 as input and its last hidden state h1, and so it continues until the end of the input sequence. Since at the first time step there is no previous hidden state, H0 is typically set to zero. Notice that the output at each time step depends on all the previous inputs. This allows recurrent neural networks to maintain memory through time, which allows them to handle sequential data well we can also stack multiple layers of recurrent cells on top of each other to get a deep recurrent neural network. In this case, each input will pass through multiple neurons one after another, just like in dense and convolutional networks we have discussed before. Depending on the lengths of input and output sequences, we distinguish four different architecture types. Let's look at them one by one. In a sequence to sequence architecture, we pass the sequence as input and make use of the output produced at every time step. For example, a real time speech recognition model could receive audio at each time step and output the corresponding text in a sequence to vector architecture. We pass a sequence as input but ignore all the outputs but the last one. In other words, we let the model process the entire input sequence before it produces the output. We can use this architecture to classify text as one of multiple topics. It's a good idea to let the model read the whole text before it decides what it's about. We will also use the sequence to vector architecture for electricity consumption prediction. One can also build a vector to sequence architecture where we pass a single input and replace all other inputs with zeros, but make use of all the outputs from each time step. This architecture can be used for text generation. Given a single vector representing a specific topic, style or sentiment, a model can generate a sequence of words or sentences. Finally, in an encoder decoder architecture, we pass the input sequences and only then start the output sequence. This is different from sequence to sequence in which outputs are generated while the inputs are still being received. A canonical use case is machine translation. One cannot translate word by word. Rather, the entire input must be processed before output generation can start. Let's build a sequence to vector RNN. In Pytorch, we define a model class with the INIT method as usual. Inside it, we assign the NN RNN layer to self rnn, passing it an input size of one. Since we only have one feature, the electricity consumption, an arbitrarily chosen hidden size of 32 and two layers and we set batch first to true. Since our data will have the batch size as its first dimension, we also define a linear layer mapping from the hidden size of 32 to the output of 1. In the forward method, we initialize the first hidden state to zeros using torch 0 and assign it to h 0. Its shape is the number of layers 2 by input size, which we extract from x as x size 0 by hidden state size 32. Next, we pass the input x and the first hidden state through the RNN layer. Then we select only the last output by indexing the middle dimension with minus 1, pass the result through the linear layer and return. Let's discuss recurrent architectures more powerful than a plain rnn. Because RNN neurons pass the hidden state from one time step to the next, they can be set to maintain some sort of memory. That's why they're often called RNN memory cells or just cells for short. However, this memory is very short term. By the time a long sentence is processed, the hidden state doesn't have much information about its beginning. Imagine trying to translate a sentence between languages. As soon as we have read it, we then remember how it started. To solve this short term memory problem, two more powerful types of cells have been proposed. The long short term memory or LSTM cell and the gated recurrent unit or GRU cell. Before we look at LSTM and GRU cells, let's visualize the plane RNN cell at each time. Step T It takes two inputs, the current input data X and the previous hidden state H. It multiplies these inputs with the weights, applies activation and outputs two things. The current outputs Y and the next hidden state. The lSTM cell has three inputs and outputs. Next to the input data X, there are two hidden states. H represents the short term memory and C the long term memory. At each time step, H and X are passed through some linear layers called gate controllers which determine what is important enough to keep in the long term memory. The gate controllers first erase some parts of the long term memory in the forget gate. Then they analyze X and H and store their most important parts in the long term memory in the input gate. This long term memory C is one of the outputs of the cell. At the same time, another gate called the output gate determines what the current output Y should be. The short term memory output H is the same as Y. Building an LSTM network in Pytorch is very similar to the plain RNN we have already seen. In the INIT method, we only need to use the NN LSTM layer instead of NN rnn. The arguments that the layer takes as inputs are the same. In the forward method, we add the long term hidden state C and initialize both H and C with 0s. Then we pass H and C as a tuple to the LSTM layer. Finally, we take the last output, pass it through the linear layer and return just like before. The GRU cell is a simplified version of the LSTM cell. It merges the long term and short term memories into a single hidden state. It also doesn't use an output gate. The entire hidden state is returned at each time step. Building a GRU network in Pytorch is almost identical to the plain rnn. All we need to do is replace the NN RNN with NN GRU when defining the layer in the INIT method and then call the new GRU layer in the forward method. So which type of recurrent network should we use? The PlanARNN, LSTM or GRU? There's no single answer, but consider the Although PlanRNNs have revolutionized modeling of sequential data and are important to understand, they are not used much these days because of the short term memory problem, our choice will likely be between LSTM and GRU. GRU's advantage is that it's less complex than LSTM, which means less computation. Other than that, the relative performance of GRU and LSTM varies per use case, so it's often a good idea to try both and compare the results. We will learn how to evaluate these models soon. Welcome back. Let's train and evaluate our rnn. Up to now we have been solving classification tasks using cross entropy losses. Forecasting of electricity consumption is a regression task for which we will use a different loss mean squared error. Here is how it's calculated. The difference between the predicted value and the target is the error. We then square it and finally average over the batch of examples. Squaring the errors plays two roles. First, it ensures positive and negative errors don't cancel out and second, it penalizes large errors more than small ones. Mean squared error loss is available in Pytorch as NN mseloss. Before we take a look at the model training and evaluation, we need to discuss two useful expanding and squeezing tensors. Let's tackle expanding first, all recurrent layers, RNNs, LSTMs and GRUs expect input in the shape, batch size, sequence length, number of features. But as we loop over the data loader, we can see that we got the shape batch size of 32 by the sequence length of 96. Since we are dealing with only one feature, the electricity consumption, the last dimension is dropped. We can add it or expand the tensor by calling view on the sequence and passing the desired shape. Conversely, as we evaluate the model, we will need to revert the expansion we have applied to the model inputs which can be achieved through squeezing. Let's see why that's the case and how to do it. As we iterate through test data batches, we get labels in shape, batch size, model Outputs, however, are of shape batch size by one our number of features. We will be passing the labels and the model outputs to the loss function and each pytorch loss requires its inputs to be of the same shape. To achieve that, we can apply the squeeze method to the model outputs. This will reshape them to match the label's shape. The training loop is similar to what we have already seen. We instantiate the model and define the loss and the optimizer. Then we iterate over epochs and training data batches. For each batch, we reshape the input sequence. As we have just discussed, the rest of the training loop is the same as before. Let's look at the evaluation loop. We start by setting up the mean squared error metric from torch metrics. Then we iterate through the test data batches without computing the gradients. Next, we reshape the model inputs just like during training, pass them to the model and squeeze the outputs. Finally, we update the metric. After the loop, we can print the final metric value by calling compute on it just like we did before. Here's our LSTM's test Min scored error. Again, let's see how it compares to a GRU network. It seems that for our electricity consumption dataset, with the task defined as predicting the next value based on the previous 24 hours of data, both models perform similarly, with GRU achieving even a slightly lower error. In this case, GRU might be preferred as it achieves the same or better results, but while requiring less processing power. Let's learn to build multi input models. Multi input models, or models that accept more than one source of data, have many applications. First, we might want the model to use multiple information sources such as two images of the same car, to predict its model. Second, multimodal models can work on different input types such as image and text, to answer a question about the image. Next, in metric learning, the model learns whether two inputs represent the same object. Think about an automated passport control where the system compares our passport photo with a picture it takes of us. Finally, in self supervised learning, the model learns data representation by learning that two augmented versions of the same input represent the same object. Multi input models are everywhere throughout the chapter. We will be using the OmniGlod dataset, a collection of images of 964 different handwritten characters from 30 different alphabets. Let's use the OmniGlod dataset to build a two input model to classify handwritten characters. The first input will be the image of the character such as this Latin letter K. The Second input will be the Alphabet that it comes from, expressed as a one hot vector. Both inputs will be processed separately. Then we concatenate the representations. Finally, a classification layer predicts one of the 964 classes. We need two elements to build such a a custom dataset and an appropriate model architecture. Let's start with the custom OmniGLOT dataset. We set it up as a class based on TORCH dataset. In the INIT method, we store, transform and samples provided when instantiating the dataset as class attributes. Samples are tuples of image file path, Alphabet as a one hot vector and target label as the character class index. In the exercises, samples will be provided for personal projects. We would need to create them from data file paths. Next, we need to implement the LEN method that returns the number of samples. Finally, the GetItem method returns one sample based on the index it receives as input for the given index. We retrieve the sample and load the image using Image openfrompill. The convert method with the argument L makes sure that the image is read as grayscale. Then we transform the image and return a triplet, the transformed image, the Alphabet vector and the target label. Before we proceed to building the model, we need to understand tensor concatenation. Torch CAT concatenates tensors along a specified dimension. We pass it the tensors and the dimension. For 2D tensors, zero stands for horizontal and one stands for vertical concatenation. It's time to define our two input model. We start with defining a sub network or layer to process our first input. The image. It should look familiar. A convolution maxpool alloy activation flattened to a linear layer of shape 128 in the end. Next we define a layer to process our second input, the Alphabet vector. Its input size is 30, the number of alphabets. And we map it to an arbitrarily chosen output size of 8. Then a classifier would accept input of size 128. 8 image and Alphabet outputs concatenated and produce the output of size 964, the number of classes in the forward method. We pass each input through its corresponding layer. Then we concatenate the outputs with Torch cat. Finally, we pass the result through the classifier layer and return the training loop looks like all the ones we have seen so far. The only difference is that now the training data consists of three the image, the Alphabet vector and the labels. And we pass the images and alphabets to the model. Welcome back. In this video we'll look at multi output models. Just like multi input models, multi Output architectures are everywhere. Their simplest use case is for multitask learning, where we want to predict two things from the same input, such as a car's make and model from its picture. In multi label classification problem, the input can belong to multiple classes simultaneously. For instance, an image can depict both a beach and people. For each of these labels, a separate output from the model is needed. Finally, in very deep models built of blocks of layers, it is a common practice to add extra outputs predicting the same targets after each block. These additional outputs ensure that the early parts of the model are learning features useful for the task at hand, while also serving as a form of regularization to boost the robustness of the network. Lets use the OmniGLOT dataset again to build a model to predict both the character and the Alphabet it comes from based on the image. First, we will pass the image through some layers to obtain its embedding. Then we add two independent classifiers on top, one for each output. The good news is that we have already done much of the work needed. We can reuse the OmniGLOT dataset we built before with just one small difference in the samples. We pass it. When the Alphabet was an input to the model, we represented it as a one hot vector. Now that it is an output, all we need is the integer representing the class label. Just like with the other output, the character. This will be a number between 0 and 29. Since we have 30 alphabets in the dataset, let's look at the model's architecture. We start with defining a sub network for processing the image identical to the one we used before. Then we define two classifier layers, one for each output, with the output shape corresponding to the number of alphabets 30 and characters 964, respectively. In the forward method, we first pass the image through its dedicated sub network and then feed the result separately to each of the two classifiers. Finally, we return the two outputs. Let's examine the training loop. The beginning should look familiar, except for the fact that now the model produces two outputs instead of one. Having produced these outputs, we calculate the loss for each of them separately using the appropriate target labels. Next, we need to define the total loss for the model to optimize. Here we just sum the two partial losses together, indicating that the accuracy of predicting the Alphabet and the character is equally important. If that is not the case, we can weigh the partial losses with some weights to reflect the relative importance. We will explore this idea later in the next video. Finally, we run backpropagation and the optimization step. As always, welcome Back in this final video of the course, we will discuss loss weighting and evaluation of multi output models. Let's dive in. Let's start with the evaluation of a multi output model. It's very similar to what we have done before, however with two different outputs. We need to set up two accuracy metrics, one for Alphabet classification and one for character classification. We iterate over the test data loader and get the model's predictions as usual. Finally, we update the accuracy metrics and after the loop we can calculate their final values. The accuracy is higher for alphabets than for characters, which is not surprising. Predicting the Alphabet is an easier task with just 30 classes to choose from. For characters, there are 964 possible labels. The difference in accuracy scores is not very large, however, 31 versus 24%. This is because learning to recognize the alphabets helped the model recognize individual characters. There is a combined positive effect from solving these two tasks at once. Let's now take a look at the training loop for our last model, predicting characters and alphabets. Because the model solves two classification tasks at the same time, we have two losses, one for alphabets and another one for characters. However, since the optimizer can only handle one objective, we had to combine the two losses somehow. We chose to define the final loss as the sum of the two partial losses. By doing so, we are telling the model that recognizing characters and recognizing alphabets are equally important to us. If that is not the case, we can combine the two losses differently. Let's say that the correct classification of characters is twice as important for us as as the classification of alphabets. To pass this information to the model, we can multiply the character loss by two to force the model to optimize it more. Another approach is to assign weights to both losses that sum up to 1. This is equivalent from the optimization perspective, but arguably easier to read for humans, especially with more than two loss components. There's just one caveat. When assigning loss weights, we must be aware of the magnitudes of the loss values. If the losses are not on the same scale, one loss could dominate the other, causing the model to effectively ignore the smaller loss. Consider a scenario where we are building a model to predict house prices and use MSE loss. If we also want to use the same model to provide a quality assessment of the house categorized as low, medium or high, we would use cross entropy loss. Cross entropy is typically in the single digit range, while MSE can reach tens of thousands. Combining these two would result in the model ignoring the quality assessment task. Completely. A solution is to scale each loss by dividing it by the maximum value in the batch. This brings them to the same range, allowing us to weight them if desired and add together Congratulations on completing the course. In Chapter one we discussed object oriented programming and how it's used to construct Pytorch datasets and models. You also learned about different optimizers and how to combat the problems of vanishing and exploding gradients using weight initialization, activation functions and batch normalization. In Chapter two, you learned to handle images in Pytorch to train and evaluate image classifying convolutional neural networks. You also augmented the image data to improve the classification results. In Chapter three, you tackled sequential data. You learned how to process it and how to construct a Pytorch dataset from it. You also got familiar with popular recurrent architectures including LSTM and GRU models which you have trained and evaluated. Finally, in chapter four, you learned to build models with multiple inputs and multiple outputs and apply loss weighting to multi output models to put more focus on one of the tasks. You can now build a wide range of deep learning models to solve various problems. But the journey doesn't end here. Here are a couple of next steps you could take on your journey. First, the Transformers model. This type of architecture was first developed for natural language processing, but today it finds applications in other areas such as computer vision. Transformers also stand behind large language models like ChatGPT. Second, self supervised learning. It's a training method in which the model creates labels from unlabeled data. It's increasingly popular in many domains. You can learn more about various data science topics through other datacamp courses. Once again, congratulations.

Sam.