Learn how to split data into training and test sets, then evaluate classification models using confusion matrices. The tutorial walks through a 14-observation credit-risk example, calculating accuracy (78.57%), sensitivity, and specificity to measure model performance.
Chapters
We have seen several techniques for pre processing the data. When the data is fully pre processed, you can go ahead and start your analysis. You can run the model on the entire data set and use the same data set for evaluating the result. But this will most likely lead to a result that is too optimistic. One alternative is to split the data into two pieces. The first part of the data, the so called training set, can be used for building the model and the second part of the data, the test set, can be used to test the results. One common way of doing this is to use 2/3 of the data for a training set and 1/3 of the data for the test set. Of course, there can be a lot of variation and the performance estimates depending which 2/3 of the data you select for the training set. One way to reduce this variation is by using cross validation for the 2/3 training set and a 1/1 test set. Example. A cross validation variant would look like this. The data would be split in three equal parts and each time two of these parts would act as a training set and one part would act as a test set. Of course, we could use as many parts as we want, but we would have to run the model many times. If using many parts, this may become computationally heavy. In this course, we will just use one training set and one test set containing 2/3 versus 1/3 of the data. Imagine we have just run a model and now we apply the model to our test set to see how good the results are. Evaluating the model for credit risk means comparing the observed outcomes of default versus non default stored in the loan status variable of the test set with the predicted outcomes according to the model. If we are dealing with a large number of predictions, a popular method for summarizing the result uses something called a confusion matrix. Here we use just 14 values to demonstrate the concept. A confusion matrix is a contingency table of correct and incorrect classifications. Correct classifications are on the diagonal of the confusion matrix. We see, for example that 8 non defaulters were correctly classified as non default and 3 defaulters were correctly classified as defaulters. However, we see that two non defaulters were wrongly classified as defaulters and one defaulter was wrongly classified as as a non defaulter. The items on the diagonals are also called the true positives and true negatives. The off diagonals are called the false positives versus false negatives. Several measures can be derived from the confusion matrix. We will discuss the classification accuracy, sensitivity and the specificity. The classification accuracy is a percentage of correctly classified incentives which is equal to 78.57 in this example. The sensitivity is a percentage of bad customers that are classified correctly, or 75% in this example, the specificity is a percentage of good customers that are classified correctly, or 80% in this example. Let's practice splitting the data and constructing confusion matrices.
