Hyperparameter Tuning and Cross Validation

# CHAPTER 18

Hyperparameter Tuning and Cross Validation #

1. Introduction #

2. Learning Objectives #

• Explain the concept of K-Fold Cross Validation.

• Differentiate between Parameters and Hyperparameters.

• Define a Hyperparameter Grid.

• Implement GridSearchCV to automate model tuning.

• Extract the "Best Model" from the search results.

3. The Flaw of the Train/Test Split #

4. K-Fold Cross Validation #

1. 1. We train on Folds 1, 2, 3, 4. We test on Fold 5. Record the score.

1. 2. We train on Folds 1, 2, 3, 5. We test on Fold 4. Record the score.

1. 3. We repeat this until every fold has been used as the Test Set exactly once.

1. 4. We take the Average of the 5 scores.

5. What is a Hyperparameter? #

• Parameters: The numbers the model calculates on its own during training (like the slope $m$ and intercept $b$). You cannot change these.

• Hyperparameters: The "knobs and dials" you set *before* training begins (like max_depth in a Tree, or alpha in Ridge Regression).

6. Mini Project: Automating the Search (GridSearchCV) #

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV

# Assume X_train and y_train are preloaded datasets

# 1. Initialize a blank model
base_model = RandomForestRegressor(random_state=42)

# 2. Define the "Grid" of Hyperparameters to test
# The dictionary keys MUST match the exact spelling in scikit-learn
param_grid = {
    'n_estimators': [50, 100, 200],   # Test 3 different forest sizes
    'max_depth': [None, 5, 10],       # Test 3 different depth limits
    'min_samples_split': [2, 5]       # Test 2 splitting rules
}
# Total Combinations = 3 x 3 x 2 = 18 combinations.

# 3. Setup GridSearchCV
# cv=5 means use 5-Fold Cross Validation for EVERY combination.
# Total models trained = 18 combinations * 5 folds = 90 models!
grid_search = GridSearchCV(estimator=base_model, param_grid=param_grid, cv=5, scoring='neg_mean_squared_error', n_jobs=-1)

# 4. Start the automated search!
print("Searching for optimal hyperparameters. This may take a moment...")
grid_search.fit(X_train, y_train)

# 5. Extract the results
print(f"The best hyperparameter combination is: {grid_search.best_params_}")

# You can now grab the "Ultimate" perfectly tuned model directly!
best_model = grid_search.best_estimator_

7. RandomizedSearchCV (The Faster Alternative) #

8. Common Mistakes #

• Data Leakage in CV: If you scale your entire dataset (StandardScaler.fit_transform(X)) *before* putting it into GridSearchCV, information leaks across the folds. You must pass a Pipeline (containing the scaler and the model) into GridSearchCV to ensure scaling happens independently inside each fold!

• Over-tuning: Testing n_estimators from 1 to 1000 in increments of 1 is a waste of computing power. Trees don't care about the difference between 101 and 102 trees. Jump by large logical chunks.

9. Best Practices #

• Use n_jobs=-1: In the GridSearchCV function, setting n_jobs=-1 tells scikit-learn to use 100% of your computer's CPU cores to train the 90 models in parallel. It will speed up the search dramatically!

10. Exercises #

1. 1. If your param_grid contains 4 options for alpha and 5 options for l1_ratio, and you set cv=3, exactly how many models will GridSearchCV train?

1. 2. Explain why a Cross-Validated score is more trustworthy than a single Train/Test split score.

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain the mechanics of a 5-Fold Cross Validation process step-by-step.

• Q: If a Grid Search is taking 48 hours to run, what specific scikit-learn alternative class would you use to drastically reduce compute time while maintaining accuracy?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #