Data Types and Data Formatting

# CHAPTER 5

Data Types and Data Formatting #

1. Chapter Introduction #

2. Understanding Pandas Data Types (dtypes) #

Pandas uses specific terms for data types:

• object or O: String/Text (or mixed types)

• int64: Integer (whole numbers)

• float64: Floating point (decimals, also used when integers have missing values)

• bool: Boolean (True/False)

• datetime64: Dates and times

• category: Categorical data (limited unique values, like "Small", "Medium", "Large")

import pandas as pd
import numpy as np

# Sample data with bad types
df = pd.DataFrame({
    'id': ['1', '2', '3'],                   # Should be int
    'price': ['$1,200.50', 'Free', '$300'],  # Should be float
    'is_active': ['Y', 'N', 'Yes'],          # Should be bool
    'category': ['A', 'B', 'A']              # Should be category
})

print("=== INITIAL DTYPES ===")
print(df.dtypes)
# Everything is 'object' (string) because of the formatting!

3. Converting to Numeric Data #

The most common task is removing currency symbols and commas so Pandas can convert text to numbers.

# 1. Clean the string (remove $ and commas)
# Using regex to replace everything that isn't a digit, minus, or period
df['price_clean'] = df['price'].str.replace(r'[^\d.-]', '', regex=True)

# 2. Convert to numeric
# errors='coerce' forces invalid parsing (like "Free") to become NaN
df['price_numeric'] = pd.to_numeric(df['price_clean'], errors='coerce')

print("\n=== AFTER NUMERIC CONVERSION ===")
print(df[['price', 'price_clean', 'price_numeric']])

# Converting simple strings to integers
df['id'] = df['id'].astype(int)

4. Handling Boolean Values #

Databases and users represent True/False in many ways (1/0, Y/N, Yes/No, T/F).

# Standardizing boolean indicators
bool_mapping = {
    'Y': True,
    'Yes': True,
    'N': False,
    'No': False,
    '1': True,
    '0': False
}

# Use map() to apply the dictionary
df['is_active_bool'] = df['is_active'].map(bool_mapping)

print("\n=== AFTER BOOLEAN CONVERSION ===")
print(df[['is_active', 'is_active_bool']])

5. Categorical Data #

If a string column has a few repeated values (e.g., Status, Country, Grade), converting it to category saves memory and speeds up operations.

# Check memory usage
print("\nMemory before category:", df['category'].memory_usage(deep=True))

# Convert to category
df['category'] = df['category'].astype('category')

print("Memory after category: ", df['category'].memory_usage(deep=True))
# The larger the dataset, the bigger the memory savings!

6. The astype() Function vs Helper Functions #

• Use astype() when the data is perfectly clean but wrong type (e.g., converting integer 1 to float 1.0).

• Use pd.to_numeric(), pd.to_datetime() when the data needs smart parsing or you need to handle errors (using errors='coerce').

# Astype will FAIL if it hits bad data:
# df['price'].astype(float) -> ValueError: could not convert string to float: '$1,200.50'

# to_numeric with coerce handles it gracefully:
# pd.to_numeric('bad_data', errors='coerce') -> NaN

7. Common Mistakes #

• Using astype(int) when there are missing values (NaN): In Pandas, NaN is technically a float. You cannot convert a column with NaNs to int64. You must either fill the NaNs first, leave it as float64, or use the newer nullable integer type astype('Int64').

• Forgetting regex=True in str.replace(): df['price'].str.replace('$', '') only replaces exact matches in newer Pandas versions unless you specify it's a literal string or a regex pattern.

8. MCQs #

9. Interview Questions #

• Q: How do you handle a column that contains both numbers as integers and numbers as text strings (e.g., 50 and "50")?

• Q: Why would you use pd.to_numeric(errors='coerce') instead of astype(float)?

10. Summary #

11. Next Chapter Recommendation #