Bootstrap Resampling for Model Evaluation

1. The intuition

In many AI projects, collecting a much larger test set is expensive. Bootstrap gives us a practical way to estimate uncertainty from the test set we already have.

2. Why sampling with replacement?

Sampling with replacement lets one example appear multiple times and another not appear at all. This mimics the idea that the observed test set is one sample from a broader population.

original: [1, 2, 3, 4, 5]
sample:   [2, 2, 4, 5, 5]

The bootstrap distribution of scores shows how sensitive the metric is to the composition of the test set.

3. Model comparison

For paired tasks like MT, summarization, or classification on the same examples, bootstrap is often more useful when applied to the difference between systems on each resampled set.

4. AI/NLP example

For English—Norwegian domain MT, I would bootstrap over sentence pairs. Each resampled test set produces BLEU, chrF, and maybe COMET for both systems. If most resamples show LoRA beating the baseline, the evidence is stronger than a single metric table.

5. Limitations

Bootstrap does not fix a bad test set. If the test set is biased, too small, duplicated, or missing the hard domain cases, bootstrap estimates uncertainty around that flawed sample.

Takeaway

Bootstrap is one of the most practical bridges between statistics and AI evaluation: it makes uncertainty visible without requiring a new dataset.

References and learning path

This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.

• An Introduction to Statistical Learning / ISLP
• Think Stats, 3rd edition
• All of Statistics — Larry Wasserman
• Seeing Theory
• 看见统计
• StatQuest
• Stanford CS229 materials
• scikit-learn: model evaluation