Data Preprocessing for Machine Learning

# CHAPTER 22

Data Preprocessing for Machine Learning #

1. Chapter Introduction #

2. The Train-Test Split #

If you teach a student the answers to a specific test, and then give them that exact same test, an A+ doesn't mean they are smart—it means they memorized it.

In ML, we must hide a portion of our data (usually 20%) to test the model later.

from sklearn.model_selection import train_test_split
import pandas as pd

# Assume 'df' is our clean DataFrame
df = pd.DataFrame({
    'Age': [25, 30, 35, 40, 45],
    'Salary': [50k, 60k, 70k, 80k, 90k],
    'Bought_Product': [0, 1, 0, 1, 1]
})

# 1. Separate Features (X) and Target (y)
X = df[['Age', 'Salary']]
y = df['Bought_Product']

# 2. Split the data (80% Train, 20% Test)
# random_state ensures we get the same split every time we run the code
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Training rows: {len(X_train)} | Testing rows: {len(X_test)}")

3. Encoding Categorical Variables #

Algorithms only understand numbers. If you have a column named City with values "NY", "LA", and "SF", you must convert them to numbers.

One-Hot Encoding (pd.get_dummies) This creates a new binary (1 or 0) column for every unique category.

data = pd.DataFrame({'City': ['NY', 'LA', 'NY', 'SF']})

# Pandas converts 'City' into 3 separate columns: City_NY, City_LA, City_SF
encoded_data = pd.get_dummies(data, columns=['City'])

print(encoded_data)
# Output:
#    City_LA  City_NY  City_SF
# 0        0        1        0
# 1        1        0        0
# 2        0        1        0
# 3        0        0        1

4. Feature Scaling (Standardization) #

Imagine predicting house prices using Bedrooms (range 1-5) and Square_Footage (range 1000-5000). The algorithm might think Square_Footage is 1000x more important just because the numbers are bigger.

We must Scale the features so they are on a similar playing field (usually a mean of 0 and standard deviation of 1).

from sklearn.preprocessing import StandardScaler

# 1. Initialize Scaler
scaler = StandardScaler()

# 2. Fit the scaler on the TRAINING data, and transform it
X_train_scaled = scaler.fit_transform(X_train)

# 3. Transform the TEST data (DO NOT fit it again!)
X_test_scaled = scaler.transform(X_test)

5. Mini Project: The ML Preprocessing Pipeline #

Let's prep a messy dataset for Machine Learning.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

raw = pd.DataFrame({
    'Age': [22, 38, None, 55],
    'Gender': ['M', 'F', 'F', 'M'],
    'Income': [45000, 80000, 60000, 120000],
    'Approved': [0, 1, 1, 1] # Target
})

# 1. Clean NaNs (Scikit-Learn hates NaNs)
raw['Age'] = raw['Age'].fillna(raw['Age'].mean())

# 2. Encode Categories
encoded = pd.get_dummies(raw, columns=['Gender'], drop_first=True) # drop_first prevents multicollinearity

# 3. Define X and y
X = encoded.drop(columns=['Approved'])
y = encoded['Approved']

# 4. Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 5. Scale
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print("Data is 100% ready for model.fit()!")

6. Common Mistakes #

• Data Leakage: Fitting the StandardScaler on the *entire* dataset before splitting it. The scaler will learn the average of the test data, leaking information about the "hidden" test set to the model. Always split first, then fit_transform the training data, and only transform the test data.

• Forgetting random_state: If you don't set this, train_test_split shuffles the data differently every time you run the code, making it impossible to tell if your model is actually getting better.

7. MCQs #

8. Interview Questions #

• Q: Explain the concept of Data Leakage. How does fitting a Scaler before doing a Train-Test split cause it?

• Q: Why do algorithms require One-Hot Encoding for categorical variables?

9. Summary #

10. Next Chapter Recommendation #