If all rows belong to the same class, the node is considered a leaf node.Construct a node using the feature with the maximum Information gain.Split the data into subsets using the feature having maximum information gain.For feature selection, the information gain finally comes into role.The feature selection is done using top to down greedy approach that is iterating from top to bottom and at each, it selects a feature. Firstly, we will select the best features to create a node that will split properly.In the beginning, we saw how the methods of creating the decision tree, in our case we use the ID3 method as we are dealing with a binary classification problem. Now, You might be wondering how exactly is the decision tree constructed, and what’s the procedure. When a person meets both the conditions, he/she is said to be fit, else if it does not meets any one of the condition, then the person is said to be unfit. In the second condition having a weight above 45 is true we consider the person as fit else if the condition is false we consider it as an unfit body. If the first condition is false the splitting is stopped. In the figure, the root node is split into two branches where if the condition height greater than four is true then we further split the node as per the second condition where the weight has to be considered above 45. The root node and the intermediate node represent the features of the dataset and the leaf node represents the labels. In the table, we can see how the labels are classified according to height and weight. And the labels are whether the person has a fit or unfit body type. Now consider an instance where we have a dataset consisting of 2 features height and weight. Let’s take a look at the working of the decision tree quickly. In our case, we’ll be diving into the ID3 method that includes entropy and information gain techniques. The methods that are used to build the decision tree are basically the CART(Classification and Regression Trees) and ID3(Iterative Dichotomiser 3). The components of a decision tree consist of a node that represents a feature(attribute), The branch represents a rule or condition and each leaf represents an outcome that can be into a categorical or continuous value form. A supervised learning method is used for both classification and regression problems. Decision tree at a glanceĪ tree-like structure that flows from top to bottom like a flowchart. If you haven’t yet seen what is a decision tree, that really cool you are at the right address, you’ll take away something memorable from here. A basic understanding of the decision tree is required to understand these two terminologies. We will try to understand entropy and information gain in a very simple and uncomplicated way. Entropy and information gain belong to the same categories of terminologies. Some moments occur when we notice that we have seen this term somewhere and we don’t remember what it exactly means.

Decision tree entropy