Machine Learning Workflows in Jupyter

# CHAPTER 14

Machine Learning Workflows in Jupyter #

1. Chapter Introduction #

2. The Machine Learning Lifecycle #

A standard ML workflow in a notebook follows these discrete steps, usually separated by Markdown headers:

1. 1. Load Data: Import Pandas and read the CSV.

1. 2. Preprocess: Clean NaNs, encode text, scale numbers.

1. 3. Train/Test Split: Hide 20% of the data to test the model later.

1. 4. Train (Fit) Model: Give the algorithm the data to learn from.

1. 5. Evaluate: Check how accurate the predictions are on the hidden test data.

3. Step 1 & 2: Load and Preprocess #

*(Note: If you don't have scikit-learn, run !pip install scikit-learn)*

Cell 1:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Simulating a dataset of houses (Square Footage vs Price)
data = pd.DataFrame({
    'sqft': [1000, 1200, 1500, 1800, 2000, 2200, 2500, 3000],
    'price': [150000, 180000, 220000, 270000, 290000, 310000, 360000, 420000]
})

# X is the features (input). Must be a 2D structure (DataFrame).
X = data[['sqft']]

# y is the target (output). Usually a 1D Series.
y = data['price']

4. Step 3: Train / Test Split #

We must split the data. Why? If you teach a student the answers to a specific test, and then give them that exact same test, an A+ doesn't mean they are smart—it means they memorized it. We must train on one dataset and test on unseen data.

Cell 2:

# Split data: 80% for training, 20% for testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Training rows: {len(X_train)}")
print(f"Testing rows: {len(X_test)}")

5. Step 4: Training the Model #

Because Jupyter keeps data in memory, we can train the model in one cell, and it remains in the Kernel ready for predictions in the next cell.

Cell 3:

# 1. Initialize the algorithm
model = LinearRegression()

# 2. Train the model using the training data
# This is where the "learning" happens
model.fit(X_train, y_train)

print("Model training complete!")

6. Step 5: Evaluation and Prediction #

Now we ask the model to predict the prices for the hidden X_test data, and compare it to the real answers (y_test).

Cell 4:

# Make predictions on the unseen test data
predictions = model.predict(X_test)

# Calculate the error (How far off were we on average in dollars?)
error = mean_absolute_error(y_test, predictions)
print(f"Average Error: ${error:,.2f}")

# Predict a brand new house not in the dataset
new_house = pd.DataFrame({'sqft': [2100]})
predicted_price = model.predict(new_house)[0]
print(f"Predicted price for 2100 sqft: ${predicted_price:,.2f}")

7. Why Jupyter Shines Here #

Imagine if training the model took 2 hours instead of 1 second. If this was a traditional Python .py script, and you realized you wanted to change the print statement in the last line, you would have to re-run the entire script, waiting 2 hours again just to see a text change.

In Jupyter, the model is already trained and sitting in the Kernel's memory. You just edit Cell 4 and hit Shift+Enter. It updates instantly. This iterative nature is why Jupyter is synonymous with Machine Learning.

8. Common Mistakes #

• Data Leakage in Cells: Running the train_test_split cell, then running a cleaning cell *below* it, then jumping back up. Your Kernel state is messed up. Always structure ML notebooks strictly chronologically: Import -> Clean -> Split -> Train -> Predict.

• Forgetting random_state: If you don't set random_state=42 (or any integer) in train_test_split, your data will split differently every time you run the cell. Your accuracy will jump around wildly, making it impossible to tell if your model is actually improving.

9. MCQs #

10. Interview Questions #

• Q: Explain the purpose of a Train/Test split in Machine Learning.

• Q: Walk me through the structural flow of a standard Machine Learning Jupyter Notebook from top to bottom.

11. Summary #

12. Next Chapter Recommendation #