· Xiaojing Yang · Statistics · 2 min read
中文Effect Size: Statistical Significance Is Not Enough
A statistically significant result can still be too small to matter in research or deployment.
Core idea
Statistical significance asks whether an effect is detectable; effect size asks whether it matters.
1. Why this topic belongs early
AI papers often focus on whether a model beats another model. But a tiny gain can become statistically significant when the test set is large. The next question is practical and scientific: is the effect big enough to care about?
How far apart are the scores?
Could noise explain it?
How large is the effect?
What did we pay?
Is it worth saying?
2. Example
Model A: 87.20 accuracy
Model B: 87.35 accuracy
difference: +0.15This might be statistically detectable on a huge benchmark. But if the model is twice as expensive, slower, less interpretable, or worse on rare safety-critical cases, the headline improvement is not enough.
3. AI/NLP effect sizes
| Context | Effect-size question |
|---|---|
| Classification | How many more examples are corrected? |
| MT | Is the BLEU/COMET gain visible in human error analysis? |
| RAG | Does the improvement reduce unsupported answers? |
| Bias evaluation | Is the disparity practically meaningful? |
| Systems | Is the quality gain worth the cost and latency? |
Metric improvement
A number moved.
Research contribution
The movement changed what the system can reliably do.
4. How I would write it
Instead of “our method significantly improves performance,” I prefer:
The improvement is statistically detectable but small; its practical value depends on whether the reduced terminology errors matter for the target domain.
Takeaway
Effect size is what keeps statistical testing connected to research meaning.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.