Hyperparameter Tuning and Cross Validation

# CHAPTER 17

Hyperparameter Tuning and Cross Validation #

1. Introduction #

2. Learning Objectives #

• Explain the concept of K-Fold Cross Validation.

• Differentiate between Parameters and Hyperparameters.

• Define a Hyperparameter Grid for classification.

• Implement GridSearchCV to automate model tuning.

• Extract the "Best Model" from the search results.

3. The Flaw of the Train/Test Split #

4. K-Fold Cross Validation #

1. 1. We train on Folds 1, 2, 3, 4. We test on Fold 5. Record the F1-Score.

1. 2. We train on Folds 1, 2, 3, 5. We test on Fold 4. Record the F1-Score.

1. 3. We repeat this until every fold has been used as the Test Set exactly once.

1. 4. We take the Average of the 5 scores.

*(Note: For classification, we use StratifiedKFold, which ensures that the ratio of Spam/Safe emails is exactly the same in every single fold).*

5. What is a Hyperparameter? #

• Parameters: The numbers the model calculates on its own during training (like the slope $m$ in Logistic Regression). You cannot change these.

• Hyperparameters: The "knobs and dials" you set *before* training begins (like max_depth in a Tree, or C in an SVM).

6. Mini Project: Automating the Search (GridSearchCV) #

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

# Assume X_train and y_train are preloaded datasets

# 1. Initialize a blank model
base_model = RandomForestClassifier(random_state=42)

# 2. Define the "Grid" of Hyperparameters to test
# The dictionary keys MUST match the exact spelling in scikit-learn
param_grid = {
    'n_estimators': [50, 100, 200],   # Test 3 different forest sizes
    'max_depth': [None, 5, 10],       # Test 3 different depth limits
    'min_samples_split': [2, 5]       # Test 2 splitting rules
}
# Total Combinations = 3 x 3 x 2 = 18 combinations.

# 3. Setup GridSearchCV
# cv=5 means use 5-Fold Stratified Cross Validation.
# Total models trained = 18 combinations * 5 folds = 90 models!
# scoring='f1' tells it to optimize for the F1-Score, not generic Accuracy!
grid_search = GridSearchCV(estimator=base_model, param_grid=param_grid, cv=5, scoring='f1', n_jobs=-1)

# 4. Start the automated search!
print("Searching for optimal hyperparameters. This may take a moment...")
grid_search.fit(X_train, y_train)

# 5. Extract the results
print(f"The best hyperparameter combination is: {grid_search.best_params_}")

# You can now grab the "Ultimate" perfectly tuned model directly!
best_model = grid_search.best_estimator_

7. RandomizedSearchCV (The Faster Alternative) #

8. Common Mistakes #

• Data Leakage in CV: If you scale your entire dataset (StandardScaler.fit_transform(X)) or apply SMOTE *before* putting it into GridSearchCV, information leaks across the folds. You must pass a Pipeline into GridSearchCV to ensure scaling and SMOTE happen independently inside each fold! (We build this in Chapter 18).

• Over-tuning: Testing n_estimators from 1 to 1000 in increments of 1 is a waste of computing power. Trees don't care about the difference between 101 and 102 trees. Jump by large logical chunks.

9. Best Practices #

• Use n_jobs=-1: In the GridSearchCV function, setting n_jobs=-1 tells scikit-learn to use 100% of your computer's CPU cores to train the 90 models in parallel. It will speed up the search dramatically!

10. Exercises #

1. 1. If your param_grid contains 4 options for C and 3 options for gamma, and you set cv=5, exactly how many models will GridSearchCV train?

1. 2. Explain why a Cross-Validated score is more trustworthy than a single Train/Test split score.

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain the mechanics of a 5-Fold Stratified Cross Validation process step-by-step.

• Q: If a Grid Search is taking 48 hours to run, what specific Scikit-Learn alternative class would you use to drastically reduce compute time while maintaining accuracy?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #