Multiple linear regression with vectors: how to set up weights W and bias b, derive gradient descent updates for each parameter, and when to use the normal equation instead. Feature scaling and learning rate matter for getting it right.
So you've learned about gradient descent, about multiple linear regression, and also vectorization. Let's put it all together to implement gradient descent for multiple linear regression with vectorization. This would be cool. Let's quickly review what multiple linear regression looks like using our previous notation. Let's see how you can write it more succinctly. Using vector notation, we have parameters w1 to wn as well as b. But instead of thinking of w1 to wn as separate numbers, I.e. separate parameters, let's instead collect all of the w's into a vector W so that now W is a vector of length n. So we're just going to think of the parameters of this model as a vector W as well as B, where B is still a number, same as before. Whereas before we had defined multiple linear regression like this. Now, using vector notation, we can write the model as F, sub wb of x equals the vector W dot product with the vector x plus B. And remember that this dot here means dot product. Our cost function can be defined as J of W1 through WN comma B. But instead of just thinking of J as a function of these n different parameters wj as well as B, we're going to write J as a function of parameter vector W and the number B. So this W1 through WN is replaced by this vector w, and J now takes its input a vector W and the number B and returns a number. Here's what gradient descent looks like. We're going to repeatedly update each parameter wj to be wj minus alpha times the derivative of the cos j, where j has parameters w 1 through wn and b. And once again, we just write this as j of vector W and number B. Let's see what this will look like when you implement gradient descent. And in particular, let's take a look at the derivative term. We'll see that gradient descent becomes just a little bit different with multiple features compared to just one feature. Here's what we had when we had gradient descent with one feature, we had an update rule for W and a separate update rule for B. So hopefully these look familiar to you. And this term here is the derivative of the cost function J with respect to the parameter w. And similarly, we have an update rule for parameter B. With univariate regression, we had only one feature. We call that feature xi without any subscript. Now here's the new notation for when we have n features where n is 2 or more. We get this update rule for gradient descent. Update W1 to be W1 minus alpha times this expression here. And this formula is actually the derivative of the cos J with respect to W1. The formula for the derivative of j with respect to W1 on the right looks very similar to the case of one feature on the left. The error term still takes a prediction f minus the target y one difference is that W and x are now vectors. And just as W on the left has now become W1 here on the right, xi here on the left is now instead xi subscript 1 here on the right. And this is just for j 1. For multiple linear regression, we have j ranging from 1 through n. And so we'll update the parameters w 1, w 2 all the way up to wn and then as before, we'll update B. And if you implement this, you get gradient descent for multiple regression.
So that's it for gradient descent for multiple regression. Before moving on from this video, I want to make a quick aside or a quick side note on an alternative way for finding W and b for linear regression. And this method is called the normal equation. Whereas it turns out gradient descent is a great method for minimizing the cost function j to find W and B. There is one other algorithm that works only for linear regression and pretty much none of the other algorithms you see in this specialization for solving for W and B. And this other method does not need an iterative gradient descent algorithm called the normal equation method. It turns out to be possible to use an advanced linear algebra library to.
Just solve for W and B all.
In one go without iterations. Some disadvantages of the normal equation method are. First, unlike gradient descent, this does not generalize to other learning algorithms, such as the logistic regression algorithm that you learn.
About next week, or the neural networks.
Or other algorithms you see later in this specialization. The norm equation method is also quite slow. If the number of features n is large. Almost no machine learning practitioners should implement the normal equation method themselves. But if you're using a mature machine learning library and call linear regression, there is a chance that on the back.
End it will be using this to solve for W and B.
So if you're ever in a job interview and hear the term normal equation.
That'S what this refers to.
Don't worry about the details of how.
The normal equation works.
Just be aware that some machine learning libraries may use this complicated method in the backend to solve for W and B. But for most learning algorithms, including how you implement linear regression yourself, gradient descent.
Is often a better way to get the job done.
In the optional lab that follows this.
Video you see how to define a multiple regression model in code and also how to calculate the prediction F of x. You also see how to calculate the cost and implement gradient descent for a multiple linear regression model. This will be using Python's NUMPY library. So if any of the code looks very new that's okay. But you should feel free also to take a look at the previous optional Lab that introduces NumPy and vectorization for refresher of NUMPY functions and and how to implement those in code.
So that's it.
You now know multiple linear regression.
This is probably the single most widely used learning algorithm in the world today.
But there's more. With just a few tricks such as picking and scaling features appropriately and also choosing the learning rate alpha appropriately, you'll be able to make this work much better. So just a few more videos to go for this week. Let's go on to the next video to see those little tricks that will help you make multiple video linear regression work much better.
