· Xiaojing Yang · Statistics · 3 min read
中文Probability Distributions for Machine Learning
How distributions become assumptions about data, labels, errors, and model behavior.
Core idea
A distribution is a compact story about what values are possible and how likely they are.
1. Why distributions matter
Machine learning is full of distributional assumptions, even when we do not name them. Classification assumes labels come from some process. Regression assumes errors have structure. Language modeling learns a distribution over tokens. Retrieval systems face query distributions that shift over time.
What examples look like
How outcomes are generated
How predictions miss
What values are plausible
How evaluation results vary
2. Discrete and continuous distributions
Discrete distributions describe countable outcomes, such as token IDs, class labels, or whether a retrieval result is relevant. Continuous distributions describe quantities such as embedding dimensions, latency, loss values, or annotation time.
| Distribution | Useful mental model | AI example |
|---|---|---|
| Bernoulli | yes/no event | correct vs incorrect prediction |
| Binomial | count of successes | number of correct answers in N cases |
| Categorical | one of many labels | sentiment class or next token |
| Normal | noisy measurement around a center | repeated metric estimates |
| Long-tailed | many rare events | words, domains, user queries |
3. The hidden danger: real data is often not neat
Text data is rarely cleanly normal. Token frequencies are long-tailed. Domains are uneven. Annotation errors are not always independent. A benchmark may overrepresent easy cases and underrepresent rare but important failures.
That is why distributional thinking is not only mathematical. It is also diagnostic: What kinds of examples are common? What kinds are rare? What changes between training and deployment?
Formula view
A distribution gives probabilities or densities.
Research view
A distribution tells us what the experiment is actually sampling from.
4. AI/NLP example
In domain machine translation, general web text and petroleum regulatory documents come from different distributions. The model may have learned common language patterns, but the test domain may contain uncommon technical phrases. A model can look strong on the broad distribution and still fail on the tail that matters.
Takeaway
Distributions help me avoid treating data as a neutral pile of examples. They make me ask: what world does this dataset represent, and what world will the model face later?
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.