· Xiaojing Yang · Machine Learning · 2 min read
中文Cross-Validation for Model Evaluation
Why one split is fragile, how K-fold works, and when cross-validation can mislead in NLP.
Core idea
Cross-validation estimates performance by rotating which part of the data plays the validation role.
1. Why one split is fragile
One train/validation split can be unlucky. Maybe the validation set is unusually easy, unusually hard, or missing important subgroups. Cross-validation reduces dependence on one split.
Each fold is one block
Hold out one fold each time
Collect K validation scores
Estimate performance and variation
2. Common variants
| Variant | Use when |
|---|---|
| KFold | Regression or balanced general data |
| StratifiedKFold | Classification with class imbalance |
| GroupKFold | Examples share users, documents, speakers, or sources |
| TimeSeriesSplit | Future must not leak into past |
| Nested CV | Hyperparameter tuning and performance estimation both matter |
3. sklearn example
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=cv, scoring="f1_macro")
print(scores.mean(), scores.std())4. When CV can mislead in NLP
Cross-validation assumes that the split structure matches the real generalization problem. In NLP, random folds can leak near-duplicate documents, same templates, same speakers, same topics, or same source corpora.
Good NLP CV
Use group-aware, document-aware, time-aware, or domain-aware folds.
Bad NLP CV
Randomly split sentence-level examples when documents or sources overlap.
Takeaway
Cross-validation is not just a function call. It is a way to ask whether the evaluation result survives different views of the data.
Interview pattern
When this appears in an interview, I would answer in four layers:
- give the short definition;
- explain the intuition;
- name the common failure mode;
- connect it to a real evaluation or deployment decision.