Learn how R-squared and F-tests measure regression fit, why evaluating models on the same data causes overfitting, and how AIC and stepAIC select simpler models. The tutorial shows how to use predict() to apply a fitted model to new data.
Chapters
There are several goodness of fit measures used to judge a model's fit. One is the so called coefficient of determination or the multiple R squared. The value of multiple R squared provides the proportion of the dependent variable's variance that is explained by the regression model. Hence, if R squared equals zero, none of the variation is explained. An r squared equal to one corresponds to a model that explains 100% of the dependent variable's variation. In general, I want my R squared to be as high as possible, but values above 0.9 are rarely reached. The F test is a test for the overall fit of the model. It tests whether or not R squared is equal to zero. That is to say, at least one regressor or a set of regressors has significant explanatory power. In our model, the p value of the F test is smaller than 0.05. Hence the hypothesis of an r squared of 0 is rejected. The variables included in the model explain some variation of the margin in year two. So far I have considered only in sample goodness of fit measures. That is to say, the model is evaluated on the same data that it was fitted on. This bears the risk of overfitting. Overfitting occurs when not only the relation between the variables shown in blue is modeled but also the relation between the errors shown in red. Then the model performs great when predicting on the data set it has been fitted on, but the prediction results on new data are poor. The linear model at first glance looks like it does not fit well, but for predictions it will be superior to the more complicated more model shown in red. There are several ways to avoid overfitting. One is to keep your model lean. Some measures for the goodness of fit, for example the AIC penalize every additional explanatory variable so that you can control for overfitting while developing a model. When comparing two models, the AIC minimizing model is preferred. In R. You can find the AIC value using the AIC from the STATS package. Note that here, since I am not comparing models to each other, I cannot draw any conclusions from an AIC of 33950.45. Automatic model selection can be done using step AIC from the MAAS package. More on that in the next chapter on logistic regression. Our methods to avoid overfitting like out of sample model validation or cross validation are also explained in depth in the chapter on logistic regression. Let's turn to prediction. So far I have used explanatory variables in year one in order to explain variation in the margin of year 2. Now I will use explanatory variables in year 2 in order to predict the margin of year 3. Therefore, I will make use of the new dataset CLVData 2. Now prediction is fairly easy. I just hand the model multiple LM2 and our new data set CLV data two over to the predict function. If I store the predictions in a vector such as spread margin, I can use them for further analysis, like calculating a mean, for example. Perfect, you made it through the whole chapter. Check out what you've learned and don't forget to practice.
