Tag: Tokenization

Tokenization is the first modeling decision in NLP: it decides what units a model can see, how rare terms are represented, and how multilingual systems handle domain language.

Tokenization 是 NLP 里的第一个建模决定：它决定模型能看见什么单位、稀有术语如何表示，以及多语系统如何处理领域语言。