Tokenization Deep Dive
An LLM does not see letters or words. It sees numbers. The first step in processing text is tokenization – breaking text into small pieces called tokens, then mapping each token to an integer ID.
Tokenization is the process of converting raw text into a sequence of tokens (subwords, words, or characters) that the model can understand.
Why Not Just Words?
If we only used whole words, the vocabulary would be huge (millions of words). Rare words would be out‑of‑vocabulary. Subword tokenization solves this: common words are whole tokens, rare words are split into smaller pieces.
Example: Byte‑Pair Encoding (BPE) – Used by GPT
BPE merges the most frequent pairs of characters or subwords iteratively.
Original: "lower"
Tokens: ["low", "er"]Why Tokenization Matters
- Vocabulary size is fixed (e.g., 50,000 tokens).
- Handles unknown words gracefully (splits into known subwords).
- Affects model performance – poor tokenization can break reasoning (e.g., "ChatGPT" vs "Chat" "GPT").
Visualizing Tokenization
Try OpenAI’s tokenizer online: `platform.openai.com/tokenizer`
Example: "I love AI!" might become ["I", " love", " AI", "!"] – 4 tokens.
Two Minute Drill
- Tokenization converts text into integer IDs.
- Subword tokenization (BPE) balances vocabulary size and coverage.
- Rare words are split into common subwords.
- Tokenization affects model understanding and cost (API pricing per token).
Need more clarification?
Drop us an email at career@quipoinfotech.com