Tokenizer
Also known as: tokeniser, LLM tokenizer, text tokenizer
A tokenizer is the part of an AI language model system that breaks text into tokens, small chunks such as whole words, parts of words, or punctuation, and converts them into numbers the model can process.
Every AI price, context limit, and speed figure is measured in tokens. While tokenizers are invisible to most users, they explain some odd AI behaviour, such as miscounting letters in a word, and why some languages cost more to process than English.
Tokenizer, explained at three levels
At home
Imagine building every word out of Lego pieces. Common words like 'the' are a single piece, while rare words snap together from smaller ones, so 'unbelievable' might be 'un,' 'believ,' and 'able.' A tokenizer is the tool that breaks your text into these pieces, called tokens, and gives each one a number. The AI never sees letters, only that string of numbered pieces.
At work
A tokenizer is the component that splits text into tokens, the word pieces an AI language model actually reads and writes, and converts them into numbers. It matters to businesses because every AI price, context limit, and speed figure is counted in tokens. The same document can produce different token counts with different providers, which changes what it costs to process.
At home
Most tokenizers use byte pair encoding. Before the model is trained, the tool scans huge amounts of text and repeatedly merges the most common pairs of characters into new pieces, so frequent words end up as one token and rare words split into several. In English, a token averages about three-quarters of a word. Because the model sees chunks instead of letters, questions like 'how many r's are in strawberry?' have tripped up AI models.
At work
Each model family ships its own tokenizer and vocabulary, typically from about 32,000 to over 200,000 tokens; OpenAI's o200k encoding, for example, has about 200,000. Bigger vocabularies cover more languages and code patterns in fewer tokens. Since the vocabulary is built from training text, languages and formats that were rare in it, such as non-English scripts, numbers, and some programming languages, split into more tokens per word, using more context and costing more per request.
At home
A tokenizer's vocabulary reflects the text it was built from, which is mostly English. Languages that were rare in that text get chopped into many small pieces, so the same sentence in Inuktitut or Punjabi can take several times as many tokens as in English. More tokens means more cost, slower replies, and less room in the context window. In my opinion, that means speakers of those languages often pay more for the same AI help.
At work
Tokenizer design affects arithmetic, since numbers may split inconsistently into digit groups, as well as multilingual fairness, code efficiency, and odd 'glitch tokens,' rare vocabulary entries the model barely saw in training that trigger strange outputs. Changing a tokenizer means retraining the model, so it is fixed for a model's life. For cost estimates, count tokens with each provider's own tools, such as OpenAI's tiktoken library or Anthropic's token-counting API, rather than assuming word counts.
Real-world examples
At home
Sophie noticed an AI chatbot used up its limit faster when she wrote in French than in English. Her nephew explained that the tokenizer split French words into more pieces.
At work
Marcus is a developer at an Ottawa translation service. He measured token counts per language with each AI provider's tokenizer, which changed which vendor was cheapest for their multilingual documents.
Common confusions
- A token is the same as a word. — Tokens are often word pieces. Short common words may be one token, while long or rare words split into several.
- All AI models count tokens the same way. — Each model family has its own tokenizer, so the same text can cost different amounts across providers.
For analysts
When budgeting AI costs, I'd measure real token counts for your actual documents and languages. Rough word-based estimates can be far off, especially for multilingual or code-heavy content.
History & origin
Early language systems split text into whole words, which struggled with rare words and typos. Philip Gage described byte pair encoding (BPE) as a data compression method in 1994. Rico Sennrich, Barry Haddow, and Alexandra Birch at the University of Edinburgh adapted BPE for neural machine translation in 2016, splitting rare words into common pieces. Google's WordPiece, used in BERT in 2018, and SentencePiece by Taku Kudo and John Richardson in 2018 offered alternatives. OpenAI's GPT-2 in 2019 used byte-level BPE so any text could be represented. Later models increased vocabulary sizes to handle more languages and code efficiently. In my view, the tokenizer is the hidden layer behind many 'why is AI so dumb about this?' moments.