In this instructional video, the presenter explains how a classification (decision) tree learns from labeled data. The tutorial begins by defining key tree components—root, internal nodes, leaves, and how they represent questions or predictions. It then introduces the concept of information gain, showing the formula for the gain obtained when a node with n samples is split into left and right subsets, and discusses impurity measures such as the Gini index and entropy. The speaker describes the recursive growth process for unconstrained trees, where splits are chosen to maximize information gain, and explains the stopping condition when gain is zero. The effect of imposing a maximum depth constraint is also covered, noting that nodes at the depth limit become leaves regardless of gain. Finally, a hands‑on example using a 2‑dimensional breast‑cancer dataset demonstrates setting the criterion to Gini, fitting a DecisionTreeClassifier, predicting test labels, and achieving roughly 92% accuracy. The video ends by encouraging viewers to practice the material.
Chapters
Welcome back. In this video, you'll examine how a classification tree learns from data. Let's first start by defining some terms. A decision tree is a data structure consisting of a hierarchy of individual units called nodes. A node is a point that involves either a question or a prediction. The root is the node at which the decision tree starts growing. It has no parent node and involves a question that gives rise to two children nodes through two branches. An internal node is a node that has a parent. It also involves a question that gives rise to two children nodes. Finally, a node that has no children is called a leaf. A leaf has one parent node and involves no questions. It's where a prediction is made. Recall that when a classification tree is trained on a labeled data set, the tree learns patterns from the features in such a way to produce the purest leaves. In other words, the tree is trained in such a way so that in each leaf one class label is predominant. In the tree diagram shown here, consider the case where an instance traverses the tree to reach the leaf on the left. In this leaf there are 257 instances classified as benign and 7 instances classified as malignant. As a result, the tree's prediction for this instance would be benign. In order to understand how a classification tree produces the purest leafs possible, let's first define the concept of information gain. The nodes of a classification tree are grown recursively. In other words, the obtaination of an internal node or leaf depends on the state of its predecessors. To produce the purest leafs possible at each node, a tree asks a question involving one feature F and the split point sp. But how does it know which feature and which split point to pick? It does so by maximizing information gain. The tree considers that every node contains information and aims at maximizing the information gain obtained after each split. Consider the case where a node with n samples is split into a left node with n left samples and a right node with n right samples. The information gained for set split is given by the formula shown here. A question that you may have in mind is what criterion is used to measure the impurity of a node? Well, there are many criteria you can use, among which are the Gini index and entropy. Now that you know what is information gain, let's describe how a classification tree learns. When an unconstrained tree is trained, the nodes are grown recursively. In other words, a node exists based on the state of its predecessors. At a non leaf node, the data is split based on feature F and split point sp. In such a way to maximize information gain. If the information gain obtained by splitting a node is nil, the node is declared as a leaf. Keep in mind that these rules are for unconstrained trees. If you constrain the maximum depth of a tree to two, for example, all nodes having a depth of two will be declared as leaves, even if the information obtained by splitting such nodes is not nil. Revisiting the 2D breast cancer dataset from the previous lesson, you can now set the information criterion of DT to the Gini index by setting the criterion parameter to Gini, as shown on the last line here. Now fit DT to the training set and predict the test set labels. Then determine DT's test set accuracy, which evaluates to about 92%. Now it's your turn to practice.
