Decision Tree Classification

# CHAPTER 8

Decision Tree Classification #

1. Introduction #

2. Learning Objectives #

• Explain how a Decision Tree splits data based on conditions.

• Understand the concepts of Gini Impurity and Entropy.

• Understand why Trees do not require Feature Scaling.

• Train a DecisionTreeClassifier using scikit-learn.

• Visualize the logical flowchart of a trained tree.

• Identify the extreme Overfitting risks of unconstrained trees.

3. How a Classification Tree Works #

1. 1. The Root Node: *Does the email contain the word "Winner"?*

• Left Branch (No): 60 emails go left (mostly Safe).

• Right Branch (Yes): 40 emails go right (mostly Spam).

1. 2. Internal Nodes: The algorithm asks another question on the right branch: *Is the Sender Domain unknown?*

1. 3. Leaf Nodes (The Prediction): Eventually, the splitting stops. If 10 emails end up in a final bucket, and 9 are Spam and 1 is Safe, the Leaf Node becomes a "Spam" predicting node. Any future email that lands in this bucket is predicted as Spam!

4. Gini Impurity & Information Gain #

• A bucket with 50 Spam and 50 Safe has High Impurity.

• A bucket with 100 Spam and 0 Safe has Zero Impurity (Pure).

5. Mini Project: Student Performance Classifier #

import numpy as np
from sklearn.tree import DecisionTreeClassifier

# 1. Provide the Data
# Features: [Hours Studied, Attendance %]
X_train = np.array([
    [2, 50],  # Fail
    [3, 60],  # Fail
    [8, 90],  # Pass
    [7, 85]   # Pass
])
y_train = np.array([0, 0, 1, 1])

# 2. Initialize the Model
# max_depth limits how many questions the tree can ask. Crucial for preventing overfitting!
tree_model = DecisionTreeClassifier(max_depth=3, random_state=42)

# 3. Train the Model
tree_model.fit(X_train, y_train)

# 4. Make a Prediction
# Student studies 6 hours, 80% attendance
X_test = np.array([[6, 80]])
prediction = tree_model.predict(X_test)
print(f"Predicted Class: {prediction[0]}") # Output: 1 (Pass)

6. The Magic of Trees: No Scaling Required! #

7. Visualizing the Tree #

from sklearn.tree import plot_tree
import matplotlib.pyplot as plt

plt.figure(figsize=(10,6))
# This draws the tree!
plot_tree(tree_model, feature_names=["Hours", "Attendance"], class_names=["Fail", "Pass"], filled=True)
plt.title("Decision Tree Flowchart")
plt.show()

*If you run this code, a beautiful visual flowchart will appear showing the exact logical splits the tree created.*

8. The Danger: Extreme Overfitting #

9. Common Mistakes #

• Forgetting max_depth: Always constrain your tree using hyperparameters like max_depth=5 or min_samples_split=10. This forces the tree to generalize rather than memorize.

10. Best Practices #

• Use for Interpretability: In industries like banking or medicine, if you deny a loan or diagnose a disease, you must be able to explain *why* to regulators. Neural Networks cannot explain their logic. Decision Trees provide a clear, printable flowchart of exactly why the decision was made.

11. Exercises #

1. 1. Why does a Decision Tree Classifier not require the use of a StandardScaler or MinMaxScaler?

1. 2. What is the primary method for preventing a Decision Tree from overfitting the training data?

12. MCQ Quiz with Answers #

13. Interview Questions #

• Q: Explain the mechanism by which an unconstrained Decision Tree overfits the training data.

• Q: Contrast the interpretability of a Decision Tree model versus a K-Nearest Neighbors model.

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #