· Xiaojing Yang · NLP and LLMs · 2 min read
中文NLP Evaluation
Why NLP evaluation needs metrics, uncertainty, human judgment, and task-specific error analysis.
Core idea
NLP evaluation is hard because language quality is multidimensional.
1. The problem
Language outputs can be fluent but wrong, correct but awkward, faithful but incomplete, or useful but not literal. A single metric rarely captures the whole task.
what success means
representative cases
measurement rule
stability of score
what failed and why
2. Metric families
| Task | Metrics |
|---|---|
| Classification | accuracy, F1, macro-F1 |
| NER | entity-level precision/recall/F1 |
| MT | BLEU, chrF, COMET, terminology accuracy |
| Summarization | ROUGE, factuality checks, human evaluation |
| Retrieval | recall@k, MRR, nDCG |
3. Hugging Face Evaluate practice
import evaluate
f1 = evaluate.load("f1")
result = f1.compute(predictions=preds, references=labels, average="macro")4. My research connection
For MT, I would not rely on BLEU alone. chrF helps with morphology, COMET connects better to human judgments, terminology metrics check domain terms, and human error analysis reveals severity.
Metric score
A compact measurement.
Research evidence
Metric + uncertainty + examples + error categories.
Takeaway
Good NLP evaluation is a chain of evidence, not one number.
Interview pattern
My interview answer would usually be:
- define the concept in one sentence;
- explain the data flow;
- name the main failure mode;
- connect it to evaluation, multilinguality, or fine-tuning.
References
- Hugging Face Course
- Hugging Face Transformers documentation
- Hugging Face tokenizer summary
- Hugging Face fine-tuning guide
- Hugging Face PEFT
- The Illustrated Transformer
- Speech and Language Processing, Jurafsky & Martin
- Stanford CS224N readings
- Attention Is All You Need
- COMET: A Neural Framework for MT Evaluation