· Xiaojing Yang · NLP and LLMs
Tokenization and Subwords
Tokenization is the first modeling decision in NLP: it decides what units a model can see, how rare terms are represented, and how multilingual systems handle domain language.
Tokenization is the first modeling decision in NLP: it decides what units a model can see, how rare terms are represented, and how multilingual systems handle domain language.