· Xiaojing Yang · Statistics · 2 min read
中文Regularization: Statistics Behind L1 and L2
Regularization controls model complexity by making some parameter values less plausible.
Core idea
Regularization is a way of saying: prefer simpler explanations unless the data strongly argues otherwise.
1. The overfitting problem
A flexible model can explain training data too well. It may learn signal, but it may also memorize noise. Regularization adds a penalty that makes overly complex parameter settings costly.
Fit the observed data
Discourage complexity
Loss + penalty
Prefer stable values
Reduce overfitting risk
2. L2 and L1 intuition
L2 regularization discourages large weights smoothly. L1 regularization can push some weights exactly toward zero, which makes it useful for feature selection in simpler models.
| Method | Penalty | Intuition |
|---|---|---|
| L2 / Ridge | squared weights | shrink weights smoothly |
| L1 / Lasso | absolute weights | encourage sparse solutions |
Without regularization
The model may chase every detail in the training set.
With regularization
The model must justify complexity with enough evidence.
3. Deep learning connection
In neural networks, weight decay is closely related to L2-style regularization. Dropout, early stopping, and data augmentation also act as regularizing strategies, although through different mechanisms.
4. AI/NLP example
For fine-tuning language models on small domain data, full fine-tuning can overfit quickly. Parameter-efficient methods like LoRA are not identical to classical regularization, but they share a practical goal: adapt the model while limiting how much can change.
Takeaway
Regularization is not a trick. It is a statistical preference for models that generalize rather than merely remember.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.