· Xiaojing Yang · Statistics · 3 min read
ChineseRandom Variables, Expectation, and Variance
A practical introduction to random variables, expectation, and variance for AI experiments.
Core idea
A random variable is how uncertainty becomes something we can calculate with.
1. The idea
In AI research, many quantities are not fixed facts. Accuracy on a test set, BLEU on a translation benchmark, annotation agreement, loss after fine-tuning, and latency under real traffic all vary with sampling and experimental conditions. A random variable is a disciplined way to represent this variation.
One possible thing that can happen
A value assigned to the outcome
How often values occur
The long-run center
The spread around the center
2. Expectation is a long-run center
Expectation is not a promise about the next observation. It is the average we would expect after many repetitions under the same data-generating process.
E[X] = sum over values: value × probabilityFor a classifier, if we sample many test sets from the same population, the average score across those test sets estimates the model’s expected performance under that population.
3. Variance tells us how unstable the number is
Two models can have the same mean score but very different stability. This matters in NLP because small test sets, rare phenomena, domain-specific terminology, or low-resource settings can make scores jump around.
Low variance
Scores are tightly clustered. A single score is more representative.
High variance
Scores move a lot across samples or seeds. One score is risky to overinterpret.
4. AI/NLP example
Imagine evaluating a machine translation model on 500 sentence pairs. If the test set contains many short generic sentences, BLEU may look stable. If it contains rare petroleum terms, long legal sentences, or noisy alignments, the observed score may depend strongly on which examples appear.
The statistical habit is: treat the score as one draw from a process, not as the whole truth about the model.
5. Common mistakes
| Mistake | Better habit |
|---|---|
| Reporting one number without context | Report the score and the evaluation setup |
| Ignoring random seeds | Track seed variation when training is unstable |
| Treating small differences as decisive | Ask whether the difference is large relative to variance |
| Forgetting the population | Name what real task the dataset is supposed to represent |
Takeaway
Expectation helps me talk about the center of an uncertain quantity. Variance helps me talk about how much trust to place in one observed result.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.