· Xiaojing Yang · NLP and LLMs · 2 min read
中文LLM Evaluation and Failure Modes
A practical map of LLM evaluation risks: hallucination, prompt sensitivity, bias, contamination, and brittle benchmarks.
Core idea
LLM evaluation must test behavior, not just benchmark scores.
1. Why LLM evaluation is different
LLMs are general-purpose systems. They can follow instructions, generate fluent text, retrieve-looking answers, and explain themselves even when wrong. Evaluation must therefore check behavior under realistic conditions.
what the user needs
instructions and contexts
possibly stochastic
metric, human, or model judge
patterns and severity
2. Common failure modes
| Failure mode | What to check |
|---|---|
| Hallucination | unsupported claims |
| Prompt sensitivity | unstable answers across wording |
| Bias | systematic disparities |
| Contamination | benchmark examples seen in training |
| Verbosity trap | fluent but low-value output |
| Long-context failure | ignores or misuses evidence |
3. Evaluation practice
# Pseudocode
for prompt in prompt_suite:
output = model.generate(prompt)
score = judge(output, reference_or_rubric)
log_failure_mode(output, score)4. My research connection
This connects to RAG, long documents, multilingual evaluation, and responsible AI. For multilingual LLMs, a model may look strong in English while failing on lower-resource languages or domain-specific technical documents.
Benchmark view
How high is the score?
Research view
Which behaviors are reliable, which fail, and under what conditions?
Takeaway
LLM evaluation is not a scoreboard. It is behavioral science for model systems.
Interview pattern
My interview answer would usually be:
- define the concept in one sentence;
- explain the data flow;
- name the main failure mode;
- connect it to evaluation, multilinguality, or fine-tuning.
References
- Hugging Face Course
- Hugging Face Transformers documentation
- Hugging Face tokenizer summary
- Hugging Face fine-tuning guide
- Hugging Face PEFT
- The Illustrated Transformer
- Speech and Language Processing, Jurafsky & Martin
- Stanford CS224N readings
- Attention Is All You Need
- COMET: A Neural Framework for MT Evaluation