Definition
A token is the basic unit of text that a language model reads, processes, and generates. Formally, a tokenizer is a function T that maps a raw string s into a sequence of discrete symbols drawn from a fixed vocabulary V:
T(s) = (t₁, t₂, …, tₙ), where each tᵢ ∈ V
Each token tᵢ is mapped to an integer index, which the model looks up in an embedding table to obtain a vector representation. The model never sees characters or words directly—it sees sequences of these integer IDs. Modern large language models typically use subword tokenization algorithms such as Byte-Pair Encoding (BPE), introduced for NLP by Sennrich et al. [1], or the related WordPiece and Unigram approaches. These algorithms build a vocabulary by iteratively merging the most frequent pairs of adjacent symbols, starting from individual bytes or characters.
A token can be a whole word (“dog”), a piece of a word (“token” + “ization”), a single character, whitespace, or even punctuation—whatever unit the tokenizer’s vocabulary happens to contain. English text averages roughly 4 characters per token; the exact ratio depends on the tokenizer and the language.
TL;DR
Imagine trying to teach someone a language using flashcards, but you cannot make a flashcard for every possible sentence—there are infinitely many. So instead, you make flashcards for common word pieces: “un”, “believe”, “able”, “ing”, “the”, “s”. Any sentence can be built by stringing enough of these cards together.
That is a tokenizer. It chops your text into a limited set of reusable chunks—tokens—so the model only ever has to deal with a fixed, manageable vocabulary (often 30,000 to 200,000 entries) instead of an infinite space of possible words and sentences.
Why does this matter to you?
- The model sometimes struggles to count letters within a word—it is not looking at letters, it is looking at token chunks.
- It is what you are billed for. API pricing for language models is quoted per token, not per word or per character and can become expensive very quick.
- It is what “context window” measures. When a model advertises a 200,000-token context window, that is the total number of these chunks—input plus output—it can keep track of in one conversation.
In short, before a model can think about your text, it has to chop it into tokens. Everything downstream—cost, memory, even little things like letter-counting mistakes—flows from that first chopping step.
Why tokens instead of words or characters?
Language models could, in principle, operate directly on characters or on whole words. In practice, subword tokenization is the standard because it balances several competing constraints
1. Fixed, manageable vocabulary size
A word-level vocabulary would need an entry for every word in every language the model supports, plus every misspelling, name, and neologism—effectively unbounded.
2. Shorter sequences than character-level models
A character-level tokenizer avoids the “unknown word” problem entirely, but it produces very long sequences—every character is its own token.
3. Graceful handling of rare and unseen words
Because the vocabulary is built from frequent substrings rather than whole words, a tokenizer can represent words it has never seen by decomposing them into familiar fragments. This is what lets a model handle typos, made-up words, and technical jargon without breaking.
4. Efficient use of the context window
Since cost and context-window limits are measured in tokens, a tokenizer that represents common text efficiently—few tokens per sentence—lets more actual content fit into a fixed budget.
Summary
A token is the atomic unit a language model actually operates on—not a character, not a word, but a subword chunk chosen by a tokenizer to balance vocabulary size, sequence length, and coverage of rare or unseen text.
References
[1] Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural Machine Translation of Rare Words with Subword Units.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016. arXiv:1508.07909