Ridge Regression and Lasso Regression

# CHAPTER 12

Ridge Regression and Lasso Regression #

1. Introduction #

2. Learning Objectives #

• Explain the concept of Regularization.

• Understand the Alpha ($\alpha$) penalty parameter.

• Implement Ridge Regression (L2 Regularization).

• Implement Lasso Regression (L1 Regularization).

• Use Lasso for automatic Feature Selection.

3. The Problem: Exploding Coefficients #

4. Ridge Regression (L2 Regularization) #

• How it works: It forces the algorithm to shrink all coefficients closer to zero. It will never force them to *exactly* zero, but it makes them very small.

• When to use it: When you have a dataset with many features, and you believe *all* of them contribute slightly to the prediction. It prevents any single feature from dominating the model.

from sklearn.linear_model import Ridge
import numpy as np

# Assume X_train and y_train are preloaded and SCALED

# alpha is the penalty strength. 
# alpha=0 is standard Linear Regression. alpha=100 is a massive penalty.
ridge_model = Ridge(alpha=1.0)
ridge_model.fit(X_train, y_train)

print(f"Ridge Coefficients: {ridge_model.coef_}")

5. Lasso Regression (L1 Regularization) #

• How it works: Unlike Ridge, Lasso has a unique mathematical property: it forces the coefficients of useless features to become exactly 0.0.

• When to use it: When you have a dataset with 500 features, and you suspect 400 of them are useless noise. Lasso will automatically delete the useless features by turning their weights to 0! It is an algorithm that performs its own Feature Selection.

from sklearn.linear_model import Lasso

# alpha is the penalty strength. Higher alpha = more coefficients turned to 0.
lasso_model = Lasso(alpha=0.1)
lasso_model.fit(X_train, y_train)

print(f"Lasso Coefficients: {lasso_model.coef_}")
# Output might look like: [4.5, 0.0, 0.0, 1.2, 0.0] -> It deleted 3 features!

6. The Critical Importance of Scaling #

7. Tuning the Alpha ($\alpha$) Parameter #

• If alpha is too low (e.g., 0.0001): The penalty is too weak. The model acts like standard Linear Regression and overfits.

• If alpha is too high (e.g., 1000): The penalty is too strong. The model shrinks all coefficients to near zero and completely Underfits.

8. Common Mistakes #

• Using Lasso on highly correlated features: If you have two features that are identical (e.g., Size in SqFt and Size in SqMeters), Lasso will randomly pick one and force the other to zero. Ridge will divide the weight equally between them.

• Forgetting to scale: As mentioned, running Ridge or Lasso on raw, unscaled CSV data renders the regularization completely mathematically invalid.

9. Best Practices #

• Default to Ridge: If you don't know which to choose, start with Ridge Regression. It generally provides the most robust improvement over standard Linear Regression for preventing overfitting.

10. Exercises #

1. 1. Which regression algorithm (Ridge or Lasso) is capable of reducing a feature's coefficient to exactly 0.0?

1. 2. If you increase the alpha parameter in a Ridge model from 1.0 to 50.0, what happens to the size of the model's coefficients?

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain the mathematical difference between the penalty applied in Ridge Regression (L2) versus Lasso Regression (L1).

• Q: Why is Feature Scaling (Standardization) an absolute mandatory prerequisite before training a Ridge or Lasso model?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #