Model Evaluation Techniques

# CHAPTER 25

Model Evaluation Techniques #

1. Chapter Introduction #

2. The Confusion Matrix #

A Confusion Matrix breaks down exactly *how* your model was right, and *how* it was wrong. It is a 2x2 grid.

• True Positives (TP): Model predicted Spam (1), and it WAS Spam. (Good!)

• True Negatives (TN): Model predicted Safe (0), and it WAS Safe. (Good!)

• False Positives (FP): Model predicted Spam (1), but it was actually Safe. (Bad - A false alarm).

• False Negatives (FN): Model predicted Safe (0), but it was actually Spam. (Bad - A missed threat).

from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

# Assuming you have y_test and predictions from Chapter 24
cm = confusion_matrix(y_test, predictions)

# Visualize it cleanly using Seaborn
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.title("Confusion Matrix")
plt.show()

3. Precision vs. Recall #

Depending on your business problem, you must optimize for either Precision or Recall.

1. Precision (Quality of Alarms): *Formula: TP / (TP + FP)* Out of all the emails the model flagged as Spam, how many were *actually* Spam? *Use Case:* When False Positives are terrible. (e.g., You don't want a legitimate email from your boss going to the Spam folder).

2. Recall (Catching the Threats): *Formula: TP / (TP + FN)* Out of all the actual Spam emails that existed, how many did the model *catch*? *Use Case:* When False Negatives are terrible. (e.g., Cancer detection. It is better to falsely alarm a patient than to miss a real tumor).

from sklearn.metrics import precision_score, recall_score, classification_report

print(f"Precision: {precision_score(y_test, predictions):.2f}")
print(f"Recall: {recall_score(y_test, predictions):.2f}")

# The ultimate function that prints everything at once:
print(classification_report(y_test, predictions))

4. Cross-Validation #

A Train-Test split relies on a random slice of data. What if the Test set happens to contain only the easiest data points by pure luck? Your score will be artificially high.

K-Fold Cross-Validation solves this. It chops the data into 5 pieces (folds). It trains on 4, tests on 1. Then it rotates, doing this 5 times until every piece of data has been used as the Test set once. The final score is the average of all 5 tests.

from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=3)

# Run 5-Fold Cross Validation on the entire X and y datasets
# cv=5 means 5 folds
scores = cross_val_score(model, X, y, cv=5)

print("Scores for each fold:", scores)
print(f"True Average Accuracy: {scores.mean():.2f}")

5. Mini Project: Cancer Detection Evaluator #

Let's evaluate a fake model predicting Malignant (1) vs Benign (0) tumors.

from sklearn.metrics import classification_report, confusion_matrix
import pandas as pd

# 10 Patients. 1 = Malignant (Cancer), 0 = Benign (Safe)
y_actual = [1, 0, 0, 1, 1, 0, 0, 1, 0, 0]

# The model's guesses
y_pred =   [1, 0, 0, 0, 1, 1, 0, 1, 0, 0] 

# Evaluate
print("--- CANCER PREDICTION REPORT ---")
print(classification_report(y_actual, y_pred))

# Look closely at the Confusion Matrix
cm = confusion_matrix(y_actual, y_pred)
print("\nConfusion Matrix:")
print(f"True Negatives (Safe correctly predicted): {cm[0][0]}")
print(f"False Positives (Safe patient told they have cancer): {cm[0][1]}")
print(f"False Negatives (Cancer patient told they are safe): {cm[1][0]}")
print(f"True Positives (Cancer correctly detected): {cm[1][1]}")

*Business Decision: The False Negative is deadly. We must adjust the algorithm to prioritize Recall over Precision.*

6. Common Mistakes #

• Relying solely on Accuracy for Imbalanced Data: If a dataset is 99% Class A and 1% Class B, accuracy is a useless metric. You must look at the Confusion Matrix.

• Confusing Precision and Recall: Precision asks "When you yelled wolf, was there actually a wolf?" Recall asks "Of all the wolves that were there, how many did you yell at?"

7. MCQs #

8. Interview Questions #

• Q: You are building a model to detect fraudulent credit card transactions. Fraud is extremely rare (0.1% of transactions). Why is Accuracy a terrible metric here? What metric would you use instead?

• Q: Explain K-Fold Cross-Validation. Why is it more robust than a single Train-Test split?

9. Summary #

10. Next Chapter Recommendation #