· Xiaojing Yang · Statistics · 3 min read
中文Power Analysis: How Many Experiments Do You Need?
Power analysis connects sample size, effect size, and the chance of detecting a real improvement.
Core idea
Power analysis asks whether the experiment is large enough to detect the effect we care about.
1. Why power matters
A non-significant result can mean there is no effect. It can also mean the experiment was too small to detect the effect. Power analysis helps separate those possibilities before we spend time and compute.
How large the real effect is
How much evidence we collect
How variable measurements are
How strict the test is
Chance of detecting the effect
2. AI experiments are expensive
In NLP and LLM evaluation, more examples may mean more human annotation, more API cost, more inference time, or more expert review. Power analysis is a planning tool: it helps decide whether a proposed experiment is worth running.
Too small
The experiment may miss effects that matter.
Large enough
The experiment has a reasonable chance to detect the target effect.
3. Practical framing
Instead of asking “how many examples do I need?” ask:
| Question | Example |
|---|---|
| What effect would matter? | +1 COMET or 10 fewer severe errors |
| How noisy is the metric? | seed variation or item-level variance |
| What error rate is acceptable? | false positives and false negatives |
| What budget exists? | annotation and inference constraints |
4. AI/NLP example
If a RAG evaluation has only 50 questions, it may be too weak to detect a small but useful improvement in groundedness. If the target effect is reducing unsupported answers from 20% to 10%, the needed sample size depends on the desired confidence and power.
Takeaway
Power analysis moves evaluation from “we tested what we had” to “we designed an experiment capable of answering the question.”
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.