Daniel Bourke sings through PyTorch's training loop: set model to train mode, forward pass, compute loss, zero gradients, backprop, step optimizer. Then switch to eval mode and run inference. The song makes the repetitive cycle stick in your head.
Chapters
Oh, excuse me. How convenient, and on microphone. And apologies for the train wreck of a room. If I'd known I'd have visitors, I probably would have cleaned up a little. But you see, I'm coding some PyTorch over here, specifically learning about PyTorch optimization loops, like a PyTorch training loop and a PyTorch testing loop, or an eval loop. And you know what? There's a fair few steps in there to remember. So I wrote a little jingle to help me remember, and you possibly remember to, and that way we can never forget. So let's train! For an epoch in a range, call model.train, do the forward pass, calculate the loss, optimize a zero grad, loss backwards, optimize a step, step, step. Let's test! Call model.eval with torch inference mode, do the forward pass, Calculate the loss, print out what's happening. Let's do it again, again, again, for another epoch. Call model.train, do the forward pass, calculate the loss. Optimizer zero grad, loss backwards. Optimizer step, step, step. Let's test. Call model.eval with torch inference mode. Do the forward pass, calculate the loss, print out out what's happening. Let's do it again, again, again. For another epoch, go model.train, do the forward pass, calculate the loss, optimize a zero grad loss backward, optimize a step, step, step. Let's test, go model.eval with torch inference mode, do the forward pass, calculate the loss, Print out what's happening. Keep going if you want, but don't forget to save, save, save. But of course, due to the Pythonic nature of PyTorch, you don't necessarily have to learn that jingle off by heart, as fun as it is. You can functionalize the training loop and the test loop, so you can save them and call them again. Genn, genn. Anyway, back to it. Hey, Torchy, play that music. Are you ready? Let's train! Tomorrow, our train to the forward pass, calculate the loss, optimize a zero grad loss backward, optimize a step, step, step. Test time!
