This short tutorial explains why you should choose the right numerical scale when sampling hyperparameters and gives concrete examples and Python implementations. It contrasts cases where uniform sampling is reasonable (e.g., integer counts like number of layers or hidden units) with cases that need log-scale sampling (e.g., learning rate alpha sampled between 1e-4 and 1). It also shows how to sample beta for exponentially weighted averages by sampling 1 - beta on a log scale (e.g., set R ~ Uniform(-3,-1), 1 - beta = 10^R, beta = 1 - 10^R).
Chapters
In the last video, you saw how sampling at random over the range of hyperparameters can allow you to search over the space of hyperparameters more efficiently. But it turns out that sampling at random doesn't mean sampling uniformly at random over the range of valid values. Instead, it's important to pick the appropriate scale on which to explore the hyperparameters. In this video, I want to show you how to do that. Let's say that you're trying to choose the number of hidden units nl for a given layer L. And let's say that you think a good range of values is somewhere from 50 to 100. In that case, if you look at the number line from 50 to 100, maybe picking some number of values at random within this number line is a pretty reasonable way to search for this particular hyperparameter. Or if you're trying to decide on the number of layers in your neural network, we're calling that capital L. Maybe you think the total number of layers should be somewhere between 2 to 4, then sampling uniformly at random among 2, 3, and 4 might be reasonable. Or even using a grid search where you explicitly evaluate the values 2, 3, and 4 might be reasonable. So these were a couple examples where sampling at uniformly at random over the range you're contemplating might be a reasonable thing to do. But this is not true for all hyperparameters. Let's look at another example. Say you're searching for the hyperparameter alpha, the learning rate. And let's say that you suspect 0.0001 might be on the low end, or maybe it could be as high as 1. Now, if you draw the number line from 0.0001 to 1 and sample values at random uniformly at random over this number line, or you know, about 90% of the values you sample will be between 0.1 and 1. So you're using 90% of the resources to search between 0.1 and 1, and only 10% of the resources to search between zero point zero zero zero one and 0.1. So that doesn't seem right. Instead, it seems more reasonable to search for hyperparameters on a log scale, where instead of using a linear scale, you'd have 0.0001 here and then 0.001, 0.01, 0.1 and then 1, and you instead sample uniformly at random on this type of logarithmic scale. Now you have more resources dedicated to searching between 0.0001 and 0 and 0.001 and between 0.001 and 0.01 and so on. So in Python, the way you implement this is let r equals negative 4 times np random dot rand. And then a randomly chosen value of alpha would be alpha equals 10 to the power of R. So after this first line, R will be a random number between minus 4 and and 0. And so alpha here would be between 10 to the minus 4 and 10 to the 0. So 10 to the minus 4 is this left thing, this 10 to the minus 4 and 1 is 10 to the 0. In a more general case, if you're trying to sample between 10 to the A to 10 to the B on the log scale, and in this example, this is 10 to the A, and you can figure out what A is by taking the log base 10 of 0.0001, which is gonna tell you A is negative 4. Um, and this value on the right, this is 10 to the B. And you can figure out what B is by taking log base 10 of 1, which tells you B is equal to 0. Um, so what you do is then sample R uniformly at random between A and B. So in this case, R would be between -4 and 0, and you can set alpha or your randomly sample hyperparameter value as 10 to the R. Okay, so just to recap, to sample on the log scale, you take the low value, take logs to figure out what is A, take the high value, take a log to figure out what is B. So you're now trying to sample from 10 to the A to 10 to the B on a log scale. So you set R uniformly at random between A and B. And then you set the hyperparameter to be 10 to the R. So that's how you implement sampling on this logarithmic scale. Finally, one other tricky case is sampling the hyperparameter beta used for computing exponentially weighted averages. So let's say you suspect that beta should be somewhere between 0.9 to 0.999. You know, maybe this is the range of values you want to search over. So remember that when computing exponentially weighted averages, using 0.9 is like averaging over last the last 10 values, kinda like taking the average of 10 days temperature, whereas using 0.999, it's like averaging over the last 1000 values. So similar to what we saw on the last slide, if you want to search between 0.9 and 0.999, you know it doesn't make sense to sample on a linear scale, right? Uniformly at random between 0.9 and 0.99. 9. So a better way to think about this is that we want to explore the range of values of 1 minus beta, which is going to now range from 0.1 to this 10 to the minus 1. This is 10 to the minus 3. Notice on the previous slide, we had the small value on the left and the large value on the right. But here we have a reverse with the large value on the left and the small value on the right. So what you do is you sample R uniformly at random from negative 3 to negative 1, and you set 1 minus beta equals 10 to the R. And so beta equals 1 minus 10 to the R. And this becomes a your randomly sampled value of your hyperparameter chosen on the appropriate scale. And hopefully this makes sense in that this way, you spend as much resources exploring the range 0.9 to 0.99 as you would exploring 0.99 to 0.999. So if you want a slightly more formal mathematical justification for why we're doing this right, why. Why is it such a bad idea to sample on a linear scale? It is that when Beta is close to one, the sensitivity of the results you get changes even with very small changes to Beta. So if beta goes from 0.9 to 0.9005, you know, it's no big deal. This is hardly any change in your results. But if beta goes from 0.999 to 0.99995, this will have a huge impact on exactly what your algorithm is doing. Right. In both of these cases, it's averaging over roughly 10 values. But here it's gone from an exponentially weighted average over about 1,000, the last 1,000 examples, to now the last 2,000 examples. And it's because that formula, we had one over one minus beta. This is very sensitive to small changes in Beta when Beta is close to one. So what this whole sampling process does is it causes you to sample more densely in the regime of when Beta is close to close to one, or alternatively, when one minus Beta is close to zero, so that you can be more efficient in terms of how you distribute the samples to explore the space of possible outcomes more efficiently. So I hope this helps you select the right scale on which to sample the hyperparameters. Um, in case you don't end up making the right scaling decision on some hyperparameter choice. You know, don't worry too much about it. Even you sample on a uniform scale, but where some other scale would have been superior, you might still get okay results, especially if you use a coarse defined search so that in later iterations you focus in more on the most useful range of hyperparameter values to sample. Hope this helps you with your hyperparameter search in the next video. I also want to share with you some thoughts on how to organize your hyperparameter search process that I hope will make your workflow a bit more efficient.
