Train/Test Split, Done Right

Exams need unseen questions

A student who memorizes past papers aces those papers and fails anything new. Models do exactly this — so we lock away part of the data as the test set, never touched during training or tuning, and judge exactly once at the end. That single number is the only honest estimate of real-world performance.

from sklearn.model_selection import train_test_split
Xtr, Xte, ytr, yte = train_test_split(
    X, y, test_size=0.2, random_state=7, stratify=y
)

Standard recipes: 80/20, or 70/15/15 with a validation slice for tuning. Fix random_state so results reproduce exactly — "works on my machine" includes randomness.

Two refinements beginners miss

• Stratify. With 5% churners, a random split might put nearly none in test. stratify=y preserves class ratios on both sides — rare classes must be gradeable everywhere.
• Time data splits by time. Predicting next month from past months? Random shuffling lets the model train on the future. Always split chronologically: past trains, future tests.

Leakage: the 99% that lies

A churn model with a cancel_date feature scores 99% — because at prediction time that column doesn't exist yet. Duplicates straddling train/test, preprocessing fit on full data, future info in features: all the same crime. The defense is one question, asked relentlessly: "Would I know this value when actually predicting?"

Remember this

• Train learns, validation tunes, test judges once. Stratify classes, split time by time.
• Leakage turns metrics into fiction; the prediction-time question catches it.