Vectorization (C1W2L11)

Welcome back. Vectorization is basically the art of getting rid of explicit for loops in your code in the deep learning era, certainly in deep learning in practice, you often find yourself training on relatively large data sets because that's when deep learning algorithms tend to shine. And so it's important that your code run quickly because otherwise if you're training on a big data set, your code might take a long time to run and you just find yourself waiting a very long time to get a result. So, so in the deep learning era, I think the ability to perform vectorization has become a key skill. Let's start with an example. So what is vectorization? In logistic regression, you needed to compute Z equals w transpose x plus B, where W was this your column vector and X is also this vector. And maybe they're very large vectors. If you have a lot of features so wide, W and X were both these RN or RNX dimensional vectors. So to compute W transpose X, if you had a non vectorized implementation, you would do something like z equals 0 and then for I in range of nx, so for I equals 1 to nx, z plus equals wi, you know, times xi, and then maybe you do z plus equals b @ the end. So that's a non vectorized implementation and you find that that's going to be really slow. In contrast, a vectorized implementation would just compute W transpose x directly. In Python or in numpy, the the command you use for that is z equals np.w, x. So this computes W transpose x and you can also just add B to that directly and you find that this is much faster. Let's actually illustrate this with a little demo. So here's my jupyter notebook in which I'm going to write some Python code. So first let me import the numpy library. So import this np. And so for example, I can create A as in array as follows. 2, 3, 4. Let's say print A. Now having written this chunk of code, if I hit shift enter, then it executes the code. So it created the array A and it prints it out. Now let's do the vectorization demo. I'm gonna import the time library. So I'm gonna use that in order to time how long different operations take. I'm gonna create an array. A equals random dot rand. Let's create a million dimensional array, right? With random variable values. B equals p.random.rand. another million dimensional array. And now tick equals time dot time. So let's measure the current time c equals np.ab. let's see. Toc equals time dot time. And let's print. This is the vectorized version, right? So vectorized version. And so let's print out, um, let's see the last time. So that's TOC minus tick times 1000. So that we can express this in milliseconds. So Ms. Is milliseconds. Okay, so I'm gonna hit shift enter. Okay, so that code took about 3 milliseconds or this time 1.5. Okay, maybe about 1.5 or 3.5 milliseconds that time. So, you know, it varies a little bit as I run it, but it looks like maybe on average it's taking like 1.5 milliseconds, maybe 2 milliseconds as I, as I run this. All right, let's keep adding to this block of code. So let's now implement a non vectorized version. Let's say C equals 0, then tick equals time time. And now let's implement a for loop for I in range of 1 million. Right. Hope I got the number of zeros right. C plus equals a I times bi. And then toc equals time dot time. And finally print, you know, with an explicit for loop. The time it takes is this 1000 times tock minus tick plus. Oops, miss. Just to denote that we're doing this in milliseconds. Okay. Oh, and actually, let's do one more thing. Let's just, you know, print out the value of C we computed to make sure that it's the same value in both cases. Okay, so I'm gonna hit shift enter to run this and check that out. So in both cases, the vectorized version and the non vectorized version computed the same value. This is, you know, 250-286-989. So on the vectorized version took 1.5 milliseconds. The explicit for loop, the non vectorized version took about 400 over almost 500 milliseconds. So the non vectorized version took, you know, something like 300 times longer than the vectorized version. Right. And so with this example, you see that if only you remember to vectorize your code, your code actually runs, you know, over 300 times faster. Yeah, well, let's just run it again. Right, here's run it again. Yeah, vectorized version 1.5 milliseconds. And the for loop took 481 milliseconds. So again, about 300 times slower to do the explicit volume. So 300x slowdown is the difference between your code taking maybe one minute to run versus it taking say five hours to run. And when you are implementing deep learning algorithms, you can really get a result back faster and iterate much faster if you vectorize your code. So some of you might have heard that a lot of scalable deep learning implementations are done on a GPU on a graphics processor unit. But all the demos I did just now in the jupyter notebook were actually on a CPU. And it turns out that both GPU and CPUs have parallelization instructions. They're sometimes called SIMD instructions. This stands for a single instruction multiple data. But what this basically means is that if you use built in functions such as this NP function or other functions that don't require you explicitly implementing a for loop, it enables, it enables Python numpy to take much better advantage of parallelism to do your computations much faster. And this is true both for computations on CPUs and computations on GPUs. It's just that GPUs are remarkably good at these SIMD calculations, but CPUs are actually also not too bad at them, maybe just not as good as GPUs. So you've seen how vectorization can significantly speed up your code. The rule of thumb to remember is whenever possible, avoid using explicit for loops. Let's go on to the next video to see some more examples of vectorization and also start to vectorize logistic.