Generalization & Validation
Bias–variance, the curse of dimensionality, double descent, train/validation/test discipline, cross-validation schemes and the four faces of data leakage.
14 questions
JuniorTheoryVery commonWhat is k-fold cross-validation, and when do you need the stratified variant?
What is k-fold cross-validation, and when do you need the stratified variant?
The data is cut into k equal folds; the model is refit k times, each run holding one fold out for scoring, and the k scores are averaged. Every row is scored once, so the estimate is steadier than one split. Stratified folds keep class shares equal.
Common mistakes
- ✗Averaging training-fold scores instead of the held-out fold scores
- ✗Using plain k-fold on a rare-class problem and getting folds without positives
- ✗Believing cross-validation removes the need for a final untouched test set
Follow-up questions
- →How does the choice of k trade estimate variance against compute cost?
- →Why does plain k-fold break when several rows belong to the same user?
JuniorTheoryVery commonWhat are the train, validation and test sets for, and in which order do you split and fit?
What are the train, validation and test sets for, and in which order do you split and fit?
Train fits model parameters, validation compares model variants and picks hyperparameters, test gives one unbiased estimate at the end. Split first, before any fitting; shares are near 60/20/20. Every transform is fit on train alone.
Common mistakes
- ✗Tuning hyperparameters on the test set and reporting that score: the best of several test scores is inflated, so tune on validation and use test once.
- ✗Fitting scalers or encoders on the full table before splitting: their statistics then include validation and test rows, which leaks those rows into training.
Follow-up questions
- →When would you replace a fixed validation split with k-fold cross-validation?
- →How do the split shares change when you have ten million labelled rows?
JuniorTheoryCommonWhat is the bias-variance trade-off, and where do a depth-2 and a depth-30 tree sit on it?
What is the bias-variance trade-off, and where do a depth-2 and a depth-30 tree sit on it?
Bias is error from a model too rigid to fit the signal; variance is how much the fit swings with the training sample drawn. A depth-2 tree is high-bias and underfits; a depth-30 tree is high-variance and memorizes noise.
Common mistakes
- ✗Swapping the two terms: bias is the error of a model too rigid for the signal, as in the shallow tree; variance is how much the deep tree changes between training samples.
- ✗Assuming a deeper tree always generalizes better: past some depth it memorizes noise, so its test error rises again while its training error drops to zero.
Follow-up questions
- →How does the ensembling method
baggingcut variance without moving bias much? - →Which term does an L2 penalty (ridge regularization) move, and in which direction?
JuniorTheoryCommonWhat does this program print?
What does this program print?
(4,)
Open full question →Common mistakes
- ✗Expecting
(), one averaged score:cross_val_scorereturns one score per fold, socv=4gives an array of 4 values; the mean is taken with.mean(). - ✗Expecting
(150,), a score per row: the held-out rows of a fold are scored together as one accuracy, so the array has one value per fold, not per sample.
Follow-up questions
- →How would you report these four scores in one line, and why does their spread matter?
- →Why are the class shares the same in every fold when
cv=4is passed for a classifier?
JuniorTheoryCommonWhich of these preprocessing steps leaks test-set information into training?
Which of these preprocessing steps leaks test-set information into training?
Fitting StandardScaler on every row of X and only then calling train_test_split
Common mistakes
- ✗Treating
transformon the test set as a leak: the statistics come fromX_trainalone, so nothing about the test rows reaches training; onlyfiton test rows leaks. - ✗Blaming a fixed
random_state: it only makes the split repeatable; the leak is a mean or standard deviation computed over rows that later land in the test set.
Follow-up questions
- →Why does a leaked scaler make the test score look better than it will be on new data?
- →Which other transformers leak in the same way when fitted before the split?
JuniorTheoryCommonWhat does this program print?
What does this program print?
6 2 2
Open full question →Common mistakes
- ✗Expecting
2 6 6:test_size=0.25is the share of rows that go to the test set, so 2 of the 8 rows are test rows and 6 stay for training. - ✗Expecting
yto stay whole:train_test_splitsplits every array it gets by the same rows, soy_testholds the 2 answers of the 2 test rows.
Follow-up questions
- →How many test rows would the same call produce for 10 rows, and why?
- →Why does a fixed
random_statematter when you compare two versions of a model?
MiddleTheoryCommonWhy do kNN and k-means degrade as dimension grows, and what does that mean for embeddings?
Why do kNN and k-means degrade as dimension grows, and what does that mean for embeddings?
Volume grows exponentially with dimension, so a sample turns sparse and distances concentrate: nearest and farthest neighbour become nearly equidistant, and a neighbourhood is no longer local. Embeddings survive because data sits on a low-dimensional manifold.
Common mistakes
- ✗Reading the curse as a compute problem rather than a geometry problem
- ✗Assuming a 768-dimensional embedding is as sparse as 768 raw one-hot columns
- ✗Expecting more features to keep helping a distance-based model indefinitely
Follow-up questions
- →Why does cosine similarity often behave better than Euclidean distance on embeddings?
- →How do approximate nearest-neighbour indexes such as
HNSWchange the picture?
MiddleTheoryCommonGroup, stratified and nested CV — when does plain k-fold overestimate model quality?
Group, stratified and nested CV — when does plain k-fold overestimate model quality?
Plain k-fold assumes rows are independent. When one user or near-duplicate spans folds, the model recognises the entity rather than the pattern, so grouped folds are needed. Stratified folds hold class shares; nested CV keeps tuning in the inner loop.
Common mistakes
- ✗Shuffling rows that share an entity and calling the result independent folds
- ✗Reporting a tuned cross-validation score without an outer loop around the search
- ✗Leaving near-duplicate rows in the data and trusting the averaged fold score
Follow-up questions
- →How do you build folds when data is grouped by user and also ordered in time?
- →What does the gap between an inner and an outer nested-CV score tell you?
MiddleDebuggingCommonCV says 0.92 and production says 0.71 — which leakage channels explain the gap?
CV says 0.92 and production says 0.71 — which leakage channels explain the gap?
Four channels: target leakage from a field filled only after the outcome, contamination from duplicate rows spanning folds, group leakage from one user on both sides, and temporal leakage from shuffling time-ordered data. Fixing one alone drops the score.
Open full question →Common mistakes
- ✗Blaming drift for a gap that a post-outcome feature fully explains
- ✗Assuming a shuffled split is safe on data that is ordered in time
- ✗Trusting an averaged fold score while duplicates span the folds
Follow-up questions
- →How would you rank the four channels by the damage each one usually does?
- →What audit would catch a post-outcome feature before any model is trained?
MiddleDebuggingCommonA scaler, imputer and selector are fit before the split — what exactly leaks, and how?
A scaler, imputer and selector are fit before the split — what exactly leaks, and how?
Fitting them on the whole table pushes held-out rows into the training statistics: the scaler carries held-out means and variances, the imputer fills from them, and selection reads held-out labels, which leaks worst. A pipeline refits each step per training fold.
Open full question →Common mistakes
- ✗Calling unsupervised transforms leak-free because they never read the target
- ✗Scaling once up front for convenience and splitting the array afterwards
- ✗Selecting features on the full table and then cross-validating the survivors
Follow-up questions
- →Why does target encoding need an inner split even inside a correct pipeline?
- →How do you apply the same discipline to a resampling step such as
SMOTE?
MiddleTheoryCommonWhy touch the test set once, and what does validation overfitting look like after 200 runs?
Why touch the test set once, and what does validation overfitting look like after 200 runs?
Every decision taken on a score leaks that set into the model, so a set reused for selection stops being unbiased. After 200 runs the best validation score is the maximum of 200 noisy draws, inflated by selection; the tell is a widening validation-to-test gap.
Common mistakes
- ✗Treating the maximum over many runs as an unbiased estimate of quality
- ✗Rescoring the test set between experiments to monitor progress
- ✗Assuming cross-validation alone removes hyperparameter selection bias
Follow-up questions
- →How does nested cross-validation restore an honest estimate under heavy tuning?
- →What changes if the search is a Bayesian optimizer rather than a random search?
SeniorTheoryCommonA model that holds up on random 80/20 splits fails on next month — build an honest protocol
A model that holds up on random 80/20 splits fails on next month — build an honest protocol
A random split scores interpolation between nearby days; deployment is extrapolation into an unseen period. Use rolling-origin splits that train on the past and score a strictly later window with a gap for label delay, then run adversarial validation.
Common mistakes
- ✗Reading a random-split score as an estimate of next month's performance
- ✗Adding a raw timestamp feature and expecting the model to extrapolate the trend
- ✗Blaming variance and re-running seeds when the real cause is distribution shift
Follow-up questions
- →How do you set the gap width when label delay and drift speed disagree?
- →What follow-up action does a high adversarial-validation AUC justify?
MiddleTheoryOccasionalDecompose expected test error into bias, variance and noise — which change moves which term?
Decompose expected test error into bias, variance and noise — which change moves which term?
Expected squared error at a point splits into squared bias, variance and irreducible noise. Capacity and boosting cut bias; averaging, regularization and more rows cut variance. Noise is a floor set by the data, moved only by better features or labels.
Common mistakes
- ✗Believing a better model or longer search can reduce irreducible noise
- ✗Swapping the levers — thinking capacity cuts variance and bagging cuts bias
- ✗Reading the three terms as the three data splits rather than error sources
Follow-up questions
- →How would you estimate the noise floor empirically from repeated labels?
- →Where does the double-descent curve sit relative to this decomposition?
SeniorDesignOccasionalYou own a card-fraud model. A transaction is labelled fraudulent only when the cardholder disputes it, and disputes arrive up to 60 days after the transaction, so the most recent two months of data carry labels that are still maturing — an unreported fraud currently looks exactly like an honest payment. Fraud rings also adapt within weeks, so the patterns that dominate this quarter were largely absent last quarter. The base rate is roughly 0.2% positives, one cardholder generates many transactions, and the model is retrained and redeployed every two weeks. Design the validation scheme you would use to decide whether a candidate model is fit to ship — the split, the fold construction, what you hold back and what you report — and be explicit about what this scheme costs you in data, in freshness and in the honesty of the estimate.
You own a card-fraud model. A transaction is labelled fraudulent only when the cardholder disputes it, and disputes arrive up to 60 days after the transaction, so the most recent two months of data carry labels that are still maturing — an unreported fraud currently looks exactly like an honest payment. Fraud rings also adapt within weeks, so the patterns that dominate this quarter were largely absent last quarter. The base rate is roughly 0.2% positives, one cardholder generates many transactions, and the model is retrained and redeployed every two weeks. Design the validation scheme you would use to decide whether a candidate model is fit to ship — the split, the fold construction, what you hold back and what you report — and be explicit about what this scheme costs you in data, in freshness and in the honesty of the estimate.
Split forward in time with a 60-day embargo before the validation window, so no immature label is scored. Roll that origin across several windows and fold by cardholder. Report precision at a fixed review capacity; the cost is a stale estimate and unused fresh data.
Common mistakes
- ✗Validating on the freshest window whose labels have not finished maturing
- ✗Treating a not-yet-disputed transaction as a confirmed honest label
- ✗Judging a fast-drifting model on one window instead of several rolling ones
Follow-up questions
- →How would you monitor once deployed, given the score only settles 60 days later?
- →What would adversarial validation add about the shift between train and deployment?
