Messy Data: Missing Values & Categories

The unglamorous 80%

Ask practitioners where their time goes and the answer is unanimous: not modeling — cleaning. Real tables have blanks, "N/A" strings, cities spelled three ways, and income in mixed currencies. Models eat numbers, so every non-number must be converted and every gap filled deliberately. This chapter is that craft.

Missing values: fill thoughtfully

• Numbers: median imputation is the safe default (robust to outliers, unlike means). Deleting rows is only okay when gaps are few and random — otherwise you bias the dataset toward whoever fills forms completely.
• Categories: a "Missing" category of its own often outperforms fancy imputation — missingness itself carries signal (people who skip "income" differ from those who answer).
• Never: magic constants like 99999, which the model reads as a real, enormous value.

Categories to numbers

Models multiply; words don't multiply. One-hot encoding (a column per city, 0/1) works for a handful of values. Hundreds of cities? One-hot explodes dimensionality — switch to ordinal (only if order is real), frequency, or target encoding. And scale numeric ranges for distance-based models, or income will drown out visit counts.

import pandas as pd
df = df.fillna({"age": df["age"].median()})
df = pd.get_dummies(df, columns=["city"], drop_first=True)

The golden rule: fit on train, apply everywhere

Fit imputers, encoders, and scalers on train only, then transform validation/test with those frozen parameters. Fitting on full data leaks test information into training — the quietest, commonest form of cheating in ML.

Remember this

• Median for numbers, explicit "Missing" for categories, one-hot for few, encodings for many.
• Fit preprocessing on train only. Clean data beats fancy models, every time.