· Xiaojing Yang · Statistics · 3 min read
中文Confidence Intervals: Why One Score Is Not Enough
A model score without uncertainty is easy to read but easy to overtrust.
Core idea
A confidence interval turns a lonely score into a range of plausible values.
1. The problem with one number
Model evaluation often ends with a table of scores. That table is useful, but it can quietly hide sampling uncertainty. If the test set had been slightly different, the score might have moved.
A finite sample
A rule for measuring output
One result
How much the score may vary
A plausible range
2. What an interval does
A confidence interval is not decorative. It changes the claim from “the model scored 87.4” to “under this evaluation process, plausible values are roughly around this range.”
score = 87.4
95% CI = [86.1, 88.6]That range is the beginning of responsible interpretation.
3. Comparing models
If Model A scores 87.4 and Model B scores 87.1, the difference may not be meaningful if their uncertainty ranges heavily overlap or if paired differences are unstable.
Leaderboard habit
Pick the larger number.
Research habit
Ask whether the difference is stable, meaningful, and visible under the right metric.
4. AI/NLP example
In LLM evaluation, a benchmark accuracy can vary because prompts, sampled examples, decoding settings, and judge behavior vary. Reporting a confidence interval makes the result less shiny but more credible.
5. Writing pattern
| Weak claim | Better claim |
|---|---|
| Model A is better than Model B. | Model A scored higher on this test set, but the interval suggests the difference is small. |
| The system achieves 91%. | The system achieved 91% on this sample; uncertainty should be checked before generalizing. |
| The improvement is obvious. | The observed improvement is larger than seed variation and supported by bootstrap intervals. |
Takeaway
One score is an observation. A confidence interval helps turn that observation into evidence.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.