Cross-Validation Lab

Cross-Validation Trial Stability Lab

How do we trust a model's performance if it was evaluated on the same data it was trained on? K-Fold Cross-Validation prevents this by rotating the test set.

Press ▶ Animate to watch each fold rotate through the test position one-by-one.

How K-Fold Works

The dataset is split into K equal folds. In each of K rounds, one fold becomes the test set and the remaining K−1 folds train the model.

For K=5, 100 patients: — Fold 1 (pts 1-20) → Test — Folds 2-5 (pts 21-100) → Train Round 2: — Fold 2 (pts 21-40) → Test — Folds 1,3-5 → Train ...repeat K times CV Score = mean(Accuracy across K rounds)

Stability & Standard Deviation

A well-generalising model produces consistent accuracy across all K folds. High variance between fold scores signals:

σ(CV scores) is low → stable model σ(CV scores) is high → model is sensitive to which patients are in training data → overfitting or class imbalance

The error bars on the accuracy chart show fold-to-fold variability.

⚖️ Ethical Reality

Stratification Matters: Standard CV may put all diabetic patients from a minority ethnic group into one fold. Stratified K-Fold preserves class proportions in each fold — essential for equitable model evaluation.

① CV Configuration

Number of Folds (K)5

2 = holdout · 10 = LOOCV-like

Model Complexity3

higher = more overfit = higher σ

② CV Formula

CV Score = (1/K) Σᵢ Acc(Mᵢ)
σ = √[(1/K) Σᵢ (Accᵢ − CV)²]
— ± —

Mᵢ = model trained without fold i
Acc(Mᵢ) = accuracy on fold i test set
Active fold: —

③ Summary Metrics

Mean CV Accuracy

—

Best Fold

—

Worst Fold

—

Std Dev (σ)

—

95% CI

④ Fold Animation — highlighted fold = current test set · glowing = active rotation

⑤ Per-Fold Accuracy — train (dim) vs test (bright) · error = overfitting gap