NumPy, Pandas, and Data Preparation

# CHAPTER 4

NumPy, Pandas, and Data Preparation #

1. Introduction #

2. Learning Objectives #

• Create and manipulate multidimensional NumPy arrays.

• Load datasets (CSVs) into Pandas DataFrames.

• Explore dataset structure using .info() and .value_counts().

• Filter rows based on specific conditions.

• Handle missing data (NaN) without crashing your models.

3. NumPy Basics #

import numpy as np

# 1D Array (Vector) - e.g., a single column of ages
ages = np.array([25, 30, 35, 40])

# 2D Array (Matrix) - e.g., A dataset of patients
# Columns: [Age, Heart_Rate]
patient_data = np.array([
    [45, 80],
    [50, 85],
    [35, 70]
])

print(f"Matrix Shape: {patient_data.shape}") 
# Output: (3, 2) -> 3 rows (patients), 2 columns (features)

4. Pandas Basics and DataFrames #

import pandas as pd

# Creating a DataFrame manually
data = {
    "Age": [25, 45, 30],
    "Income": [50000, 80000, 60000],
    "Defaulted": [0, 1, 0] # 1=Defaulted on loan, 0=Paid
}

df = pd.DataFrame(data)
print(df)

5. Reading CSV Files and Exploration #

# Load a CSV file (Assuming 'bank_data.csv' exists)
# df = pd.read_csv("bank_data.csv")

# View the first 5 rows
print(df.head())

# View dataset information (Column names, data types)
print(df.info())

# CRITICAL FOR CLASSIFICATION: Check class balance!
# How many people Defaulted vs Paid?
print(df['Defaulted'].value_counts())
# Output might be: 
# 0    850
# 1    150

6. Data Filtering #

# Get a single column
incomes = df['Income']

# Filter rows: Find all customers older than 40
older_customers = df[df['Age'] > 40]

# Multiple conditions: Older than 40 AND Defaulted
high_risk = df[(df['Age'] > 40) & (df['Defaulted'] == 1)]

7. Handling Missing Data #

# Check how many missing values are in each column
print(df.isnull().sum())

# Strategy 1: Drop any row containing NaN (Quick, but loses data)
df_clean = df.dropna()

# Strategy 2: Imputation (Fill missing values with the column's mean)
mean_income = df['Income'].mean()
df['Income'].fillna(mean_income, inplace=True)

8. Mini Project: Prepare Dataset for ML #

import pandas as pd

# Mock Data
df = pd.DataFrame({
    "Age": [22, 45, 30, 55, 28],
    "Credit_Score": [600, 750, 680, 800, 590],
    "Approved": [0, 1, 1, 1, 0] # 1=Approved for credit, 0=Denied
})

# 1. Isolate the Features (X)
# We drop the Target column, keeping ONLY the inputs
X = df.drop("Approved", axis=1)

# 2. Isolate the Target Label (y)
# This is the single column the algorithm must learn to predict
y = df["Approved"]

print("Features (X) shape:", X.shape) # Output: (5, 2)
print("Target (y) shape:", y.shape)   # Output: (5,)

*Your data is now mathematically separated and ready to be fed into a Classification algorithm!*

9. Common Mistakes #

• Confusing loc and iloc: In Pandas, if you want to select a row by its exact integer position in the list (e.g., the 5th row), use df.iloc[5]. If you use df.loc[5], it searches for a row whose literal name/index label is "5".

• Forgetting axis=1: When dropping a column (df.drop('Approved')), Pandas defaults to looking for a *row* named 'Approved' and will throw an error. You must specify axis=1 to tell it to drop a column.

10. Best Practices #

• Always check value_counts(): In Classification, knowing if your target variable is balanced is critical. If your dataset has 990 "Not Fraud" rows and only 10 "Fraud" rows, standard algorithms will fail. (We dedicate Chapter 14 entirely to fixing this issue!).

11. Exercises #

1. 1. If you load a DataFrame and df.shape returns (5000, 12), what does that tell you about the dataset?

1. 2. Write the Pandas code to find out how many rows contain missing values (NaN) in the entire DataFrame.

12. MCQ Quiz with Answers #

13. Interview Questions #

• Q: Explain why feeding NaN (missing values) directly into a standard machine learning model causes it to crash, and describe two strategies to handle them using Pandas.

• Q: What is the purpose of separating a dataset into X and y matrices?

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #