Performance Optimization for Large Datasets

# CHAPTER 18

Performance Optimization for Large Datasets #

1. Chapter Introduction #

2. The Problem: Memory Limits #

Pandas loads data entirely into your computer's RAM. If you have 8GB of RAM, and you try to load a 10GB CSV file using pd.read_csv(), your kernel will crash. Furthermore, Pandas usually requires 2x to 3x the size of the dataset in RAM to perform operations (like sorting or merging).

3. Optimization 1: Data Type Downcasting #

By default, Pandas assigns the largest possible memory bucket to numbers (int64 and float64). If your column represents "Age" (0 to 120), using a 64-bit integer is a massive waste of memory. An 8-bit integer (int8) can store numbers up to 127 and uses 8x less memory.

import pandas as pd
import numpy as np

# Create a sample large dataset
df = pd.DataFrame({
    'id': np.random.randint(1, 1000000, 1000000), # Large numbers
    'age': np.random.randint(1, 100, 1000000),    # Small numbers (max 100)
    'status': np.random.choice(['Active', 'Pending', 'Closed'], 1000000) # Text
})

print("=== MEMORY BEFORE OPTIMIZATION ===")
df.info(memory_usage='deep')
# You will see memory usage is around 68 MB.

# Downcast Integers
df['age'] = pd.to_numeric(df['age'], downcast='integer')
# Pandas automatically chooses int8 because max value is < 127

# Convert low-cardinality text to Categorical
# Instead of storing the word "Active" 300,000 times, it stores it once in a dictionary 
# and replaces the column with tiny integers (0, 1, 2)
df[&#039;status'] = df['status'].astype('category')

print("\n=== MEMORY AFTER OPTIMIZATION ===")
df.info(memory_usage=&#039;deep')
# Memory drops significantly (e.g., from 68 MB to 9 MB!)

4. Optimization 2: Reading Data in Chunks #

If the file is simply too big to fit in RAM, you must read it in chunks. You load 100,000 rows, clean them, append the results to a new file, and repeat.

# 1. Prepare an empty output file with headers
# (Pseudo-code for the process)

# 2. Create a chunk iterator
chunk_size = 100000
chunk_iterator = pd.read_csv('massive_file.csv', chunksize=chunk_size)

for i, chunk in enumerate(chunk_iterator):
    print(f"Processing chunk {i}...")
    
    # --- Clean the chunk ---
    # Drop rows missing critical IDs
    clean_chunk = chunk.dropna(subset=['customer_id'])
    
    # Standardize strings
    clean_chunk['name'] = clean_chunk['name'].str.upper()
    
    # --- Append the clean chunk to the new file ---
    # mode='a' means append. header=False prevents writing the header 100 times.
    if i == 0:
        clean_chunk.to_csv('cleaned_massive_file.csv', mode='w', index=False)
    else:
        clean_chunk.to_csv('cleaned_massive_file.csv', mode='a', index=False, header=False)

print("Batch processing complete.")

5. Optimization 3: Only Load What You Need #

If a CSV has 50 columns, but you only need 3 columns for your analysis, do not load the whole file and then drop the columns. Load only the 3 columns.

# Load ONLY the 'id', 'date', and 'amount' columns
# This immediately saves massive amounts of RAM
cols_to_keep = ['id', 'date', 'amount']
df_lean = pd.read_csv('data.csv', usecols=cols_to_keep)

6. Optimization 4: Avoid Loops (Vectorization) #

Python for loops are incredibly slow. Pandas is built on C code. Always use vectorized Pandas functions instead of iterating through rows.

import time

# Dataset with 1 million rows
df = pd.DataFrame({'price': np.random.rand(1000000) * 100})

# SLOW WAY (Iterating with a for loop)
start = time.time()
discounted_slow = []
for index, row in df.iterrows():
    discounted_slow.append(row['price'] * 0.9)
print(f"Loop time: {time.time() - start:.2f} seconds") # Might take 50+ seconds

# FAST WAY (Vectorization)
start = time.time()
df['discounted_fast'] = df['price'] * 0.9
print(f"Vectorized time: {time.time() - start:.5f} seconds") # Takes 0.005 seconds

7. Common Mistakes #

• Using apply() for everything: Beginners discover .apply(lambda x: ...) and use it for everything. apply() is essentially a hidden for loop. It is slow. Use built-in vectorized .str or .dt accessors whenever possible.

• Not using the category data type: If you have 10 million rows of US States, storing them as strings is a massive waste of RAM. Converting to .astype('category') provides the fastest and easiest memory reduction.

8. MCQs #

9. Interview Questions #

• Q: You have a 20GB dataset but only 8GB of RAM. How do you calculate the total revenue across all rows?

• Q: Explain 3 ways to optimize memory usage in a Pandas DataFrame.

10. Summary #

11. Next Chapter Recommendation #