Tokenization and Subwords

1. Why I care about tokenization

When I first learned NLP, tokenization looked almost boring: split text, convert tokens to ids, feed the ids into a model. But the more I work with multilingual and domain-specific text, the less innocent tokenization looks.

For English—Norwegian petroleum-domain machine translation, tokenization can affect:

• whether rare technical terms stay recognizable;
• whether Norwegian compounds are split into useful pieces or awkward fragments;
• whether a sentence becomes much longer after tokenization;
• whether multilingual vocabulary sharing helps or hurts low-resource/domain language;
• whether evaluation errors come from the model, the data, or the tokenizer.

So my working definition is:

Tokenization is the interface between human language and model computation.

If the interface is poor, the model starts the task already disadvantaged.

2. From raw text to model input

A Transformer model does not read text directly. It reads integer ids. The tokenizer performs the conversion:

raw text
  → tokens / subwords
  → token ids
  → embeddings
  → attention layers

For example, a sentence like:

Norwegian petroleum terminology matters.

might become a sequence of subword tokens, then a sequence of ids. The model never sees the original sentence as we see it. It sees a sequence of vocabulary items.

The key point: every downstream representation depends on this first segmentation.

3. Why not just use words?

Word-level tokenization feels intuitive, but it creates a huge vocabulary. Every inflected form, rare technical term, misspelling, name, compound, or domain-specific expression may need its own entry.

Character-level tokenization avoids unknown words, but sequences become long and the model has to learn meaning from very small units.

Subword tokenization is the compromise:

This is why most Transformer models use subwords.

4. BPE, WordPiece, and SentencePiece

The three names that appear again and again are BPE, WordPiece, and SentencePiece.

I do not need to memorize every implementation detail for an interview. What matters is understanding the trade-off:

Subword tokenizers reduce vocabulary size by representing rare words as combinations of common pieces.

That trade-off is powerful, but it is not free.

5. A concrete subword example

Here is the kind of table I would use when debugging a multilingual/domain model:

These are illustrative examples, not outputs from one fixed tokenizer. In real work, I would inspect the exact tokenizer used by the model.

6. Hugging Face demo

The fastest way to make tokenization visible is to inspect tokens directly.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")

examples = [
    "wellbore integrity",
    "decommissioning of petroleum installations",
    "Petroleumstilsynet published new guidelines.",
]

for text in examples:
    tokens = tokenizer.tokenize(text)
    ids = tokenizer.convert_tokens_to_ids(tokens)
    print(text)
    print(tokens)
    print(ids)
    print("num_tokens:", len(tokens))
    print()

What I would look for:

• Does a domain term become many small fragments?
• Does the Norwegian example become much longer than the English one?
• Are important terms represented consistently?
• Does tokenization create a truncation risk for long technical documents?

This tiny diagnostic is often more useful than staring at a final BLEU or COMET score.

7. Why tokenization matters for multilingual models

Multilingual tokenizers try to share a vocabulary across many languages. This can help transfer: related scripts, loanwords, names, and technical terms may share pieces.

But sharing is not equal coverage.

For a high-resource language, a tokenizer may contain many meaningful pieces. For a lower-resource or domain-heavy setting, the same tokenizer may break important words into longer, less meaningful sequences.

That creates an evaluation question:

Is the model worse because it lacks knowledge, or because the tokenizer gives it a poor representation of the input?

8. Why tokenization matters for domain MT

In domain machine translation, terminology is not decoration. If a model mistranslates a technical term, the output can become unusable even if the sentence is fluent.

For English—Norwegian petroleum MT, I would inspect tokenization before and after domain adaptation:

This connects directly to LoRA and PEFT. If the tokenizer fragments rare domain terms badly, an adapter may still improve translation style or terminology, but it is adapting on top of a limited input representation.

9. Common failure modes

The practical habit is simple: when a multilingual or domain model fails, inspect tokenization early.

10. Interview answer

If an interviewer asks “What is tokenization in NLP?”, I would answer:

Tokenization converts raw text into the units a model can process, usually tokens or subwords mapped to integer ids. Modern Transformer models often use subword tokenization because it balances vocabulary size and rare-word coverage. The trade-off is that important words, especially in low-resource or domain-specific settings, may be fragmented in ways that affect sequence length, representation quality, and evaluation.

If they ask “Why do subwords matter?”, I would add:

Subwords let the model represent unseen or rare words using smaller learned pieces. This is useful for morphology and multilingual transfer, but I would always inspect whether important domain terms are split into meaningful pieces.

11. How I would use this in a project

For a real project, I would add a small tokenizer audit before training:

def tokenizer_audit(tokenizer, terms):
    rows = []
    for term in terms:
        tokens = tokenizer.tokenize(term)
        rows.append({
            "term": term,
            "tokens": tokens,
            "num_tokens": len(tokens),
        })
    return rows

Then I would inspect domain terms such as:

terms = [
    "wellbore integrity",
    "blowout preventer",
    "decommissioning",
    "petroleumstilsynet",
    "subsea installation",
]

This is not glamorous, but it is exactly the kind of small diagnostic that makes an NLP project more credible.

Takeaway

Tokenization is where language becomes model input. For multilingual and domain-specific NLP, it can shape everything that follows: sequence length, representation quality, term preservation, fine-tuning behavior, and evaluation.

Before I ask “does the model understand the term?”, I want to ask:

How did the tokenizer show the term to the model?

References and further reading

• Hugging Face Course
• Hugging Face tokenizer summary
• Hugging Face Transformers documentation
• Speech and Language Processing, Jurafsky & Martin
• Stanford CS224N readings
• The Illustrated Transformer
• Neural Machine Translation of Rare Words with Subword Units
• SentencePiece: A simple and language independent subword tokenizer and detokenizer