Information gain measures exactly how much a candidate split reduces uncertainty (entropy) — it's the metric a decision tree actually compares across every candidate feature and threshold to decide where to split next.
Formula
\(\text{Entropy(parent)}\) is the impurity before splitting. The sum is the weighted average entropy of the resulting child nodes, weighted by how many samples \(n_k\) landed in each child out of the total \(n\). Information gain is simply the drop in entropy the split achieved.
Full Worked Calculation
Continuing the running example from Decision Tree Classification: root has 10 samples (6 Yes, 4 No), split on "Sunny?" into Sunny (4 samples: 1 Yes, 3 No) and Not-Sunny (6 samples: 5 Yes, 1 No).
| Quantity | Value |
|---|---|
| Entropy(parent) | 0.971 |
| Entropy(Sunny) | 0.811 |
| Entropy(Not Sunny) | 0.650 |
| Weighted child entropy | \(\tfrac{4}{10}(0.811)+\tfrac{6}{10}(0.650) = 0.714\) |
| Information Gain | \(0.971 - 0.714 = \mathbf{0.257}\) |
def entropy(labels):
import numpy as np
from collections import Counter
counts = Counter(labels)
n = len(labels)
probs = [c/n for c in counts.values()]
return -sum(p * np.log2(p) for p in probs)
def information_gain(parent, children):
n = len(parent)
weighted_child_entropy = sum((len(c)/n) * entropy(c) for c in children)
return entropy(parent) - weighted_child_entropy
parent = [1,1,1,1,1,1,0,0,0,0]
sunny = [1,0,0,0]
not_sunny = [1,1,1,1,1,0]
print(information_gain(parent, [sunny, not_sunny])) # 0.2565How the Tree Uses This — Comparing Multiple Candidates
In a real dataset, the tree computes information gain for every possible feature and threshold, then picks whichever produces the highest IG:
| Candidate Split | Information Gain |
|---|---|
| Outlook = Sunny? | 0.257 |
| Humidity > 70%? | 0.150 (hypothetical) |
| Wind = Strong? | 0.048 (hypothetical) |
Here, "Outlook = Sunny?" wins and becomes the root split, because it reduces uncertainty the most out of every option considered.
Practical Use Cases
- The core decision-making mechanism inside every entropy-based decision tree split
- Feature selection — features that never produce meaningful information gain across any split are effectively uninformative for this target
Common Mistakes
- Forgetting to weight child entropy by group size — an impure but tiny child group shouldn't count as heavily as a large one.
- Assuming a feature with high information gain at the root will also be useful deeper in the tree — relevance can change once the data has already been split by other features.
Interview Relevance
Q: "How does a decision tree choose which feature to split on at each node?" It computes information gain (or Gini reduction) for every candidate feature and threshold at that node, and selects whichever split yields the highest value — repeating this process independently at every subsequent node.
Practice Question
A split produces two children with entropy 0.4 and 0.9, weighted 70% and 30% of the parent's samples respectively. If the parent's entropy was 0.85, compute the information gain.
