In this instructional video, the speaker explains the bias‑variance trade‑off, a fundamental concept in supervised machine learning that accounts for why models may perform well on training data but poorly on unseen data. The talk begins by framing the problem: the total prediction error can be decomposed into three components—bias squared, variance, and irreducible (noise) error. Bias is described as the error from an overly simple model that underfits the data (e.g., fitting a straight line to a curved relationship), leading to high training and test error. Variance is the error from a model that is too sensitive to the training set, overfitting noise (e.g., deep decision trees, high‑degree polynomials, complex neural nets on small data), resulting in low training error but high test error. The speaker illustrates how model complexity inversely affects bias and variance, producing a U‑shaped total‑error curve across the complexity axis. Practical remedies are provided: to reduce high bias, increase model complexity, add features, reduce regularization, or switch to non‑linear models; to reduce high variance, gather more data, simplify the model, apply L1/L2 regularization, use cross‑validation, prune trees, or employ dropout in neural networks. Finally, the video links the bias‑variance trade‑off to broader machine‑learning practices such as regularization, cross‑validation, model selection, and hyper‑parameter tuning, emphasizing that mastering this trade‑off underpins a solid understanding of these techniques.
Chapters
A very good question which you have asked, like why does your machine learning model perform amazing on training data but sometimes terrible on testing data, or sometimes it every time performs bad? So the honest answer is in the one of the most important concept of machine learning, which is known as bias-variance trade-off, which we covered in the second last video, right? So if you truly understand this, you can understand machine learning in the simplest way possible. So let's first understand what is the problem. So what happens is in the machine learning, which is specifically a supervised learning We want to learn a function y f. So y f means what we are, what we are trying to do is we are trying to know the true function. We only have our training data, we have a model, and we have a hope. And model every time makes a mistake. So that particular error or that particular mistake which is made by the model can be broken into 3 things. So total error can be broken into 3 things. Total error is equal to bias squared plus variance plus irreducible error. So let us understand each and everything clearly. What is bias? In simple term, bias is basically that error which is caused by oversimplifying the model. So at this stage, it means in case of bias, your model is trying to answer one question, that is my model too simple to capture the real pattern? For example, you are trying to fit a straight line in the curved data, or your model is underfitting. So high bias means too simple, missing patterns, high training error, high testing error. Example, there are certain models with a high bias. What happens to them is in the linear regression model for the complex collinear data, this might happen. If you are trying to make a shallow decision tree, then high bias can happen. Bias means underfitting. So what is variance? Variance is basically the error that is caused by the model being too sensitive to the training data, and it tries to answer, does my model changes drastically with the small data change? For example, you are trying to fit a complex curve that perfectly matches every training point, but it fails on new data. What happens when there is a high variance? High variance means you are overfitting the complex model. You are getting low training error, you are getting high testing error, and it is giving you overfitting speech. There are multiple models with a high variance. For example, when you are trying to make very deep decision tree High variance is a high probability to get it. High degree polynomial regression, if you are trying to make, then yes. Complex neural network with a small data, that can also get you high variance. Third is the trade-off. What is the trade-off? What is the key idea behind trade-off is when the model complexity increases, the bias decreases and the variance increases. Understand it again. The model complexity increases, the bias decreases and the variance increases. When the model complexity decreases, what happens is the bias increases and the variance decreases, but you cannot minimize both at the same time. So the goal is to find that sweet spot where we can give minimum test error, and this balance is only known as bias-variance trade-off. This graph which is on my screen now, so x-axis is equal to model complexity, y-axis is the error. So what we are trying to draw is we are trying to draw the bias curve decreasing and the variance curve increasing. So what is happening is in total we are getting curve U-shaped. So left side is underfitting, which is high variance, high bias, and right side is overfitting, which is high variance, and the middle is the optimal model. Now, how to reduce high bias? If your model is underfitting, let's say, so you have to increase the model complexity, you have to add features in it, you have to reduce the regularization, and you also have to use non-linear models. If you want to reduce high variance, then you are going to follow these 5 to 6 things. If your model is overfitting, in simple terms, you have to collect more data, you have to reduce the model complexity, you have to add the regularization L1, L2 norm, and then you have to also— you can use cross-validation and then pruning for trees and dropout for neural networks. Remember these 6 things which I told you related to high variance. Simple term, bias means underfitting, variance means overfitting. Total error is bias squared plus variance plus the noise, which is reducible error. And the goal is to balance both of them. So if you understand this concept deeply, you can understand 4 concepts: regularization, cross-validation, model selection, parameter tuning. So everything connects back to the bias and variance. So if this made bias and variance finally clear, then you are good to go for the other models as well.
