WTF is Tokens?
The chunks of text (words, word pieces or characters) a model actually reads and writes, and what you pay for.
Also called: token, tokenization
TL;DR
Models don't see letters or words; they see tokens. "Unbelievable" might be three tokens. Context limits, prices and speed are all counted in tokens, so they are the unit of everything in LLM engineering. A rough rule for English: one token is about three-quarters of a word.
How it works
A tokenizer (commonly byte-pair encoding or a variant) splits text into sub-word units from a fixed vocabulary, each mapped to an integer ID the model embeds. Input and output tokens are often priced differently, and output tokens dominate latency because they are generated one at a time. Tokenization quirks explain classic failures such as miscounting letters in a word, and why non-English text and code can cost more tokens per character.
Origin
- Coined / introduced by
- NLP practice
- Story
- Sub-word tokenization became standard with transformer language models.