Building on Tokenization, this note covers what tokens mean specifically in the context of using and reasoning about LLMs — where token counts directly affect cost, speed, and how much text a model can actually process at once.
Tokens ≠ Words ≠ Characters
| Text | Words | Approximate Tokens (English, GPT-style BPE) |
|---|---|---|
| "Hello world" | 2 | 2 |
| "unhappiness" | 1 | 2–3 (subword pieces) |
| "antidisestablishmentarianism" | 1 | 6+ (many subword pieces) |
A common rough rule of thumb for English text: roughly 1.3 tokens per word on average — though this varies significantly by language, domain (code, technical jargon), and the specific tokenizer used.
Why Token Counts Matter Practically
- Cost: most commercial LLM APIs charge per token, for both input and generated output.
- Context window budget: every model has a maximum number of tokens it can process at once (covered fully in Context Window) — token count determines whether a given piece of text fits.
- Latency: generation happens one token at a time (autoregressively), so more output tokens directly means more generation steps and more time.
Code — Counting Tokens Before Sending a Request
import tiktoken
encoding = tiktoken.encoding_for_model("gpt-4")
text = "The quick brown fox jumps over the lazy dog."
tokens = encoding.encode(text)
print(len(tokens)) # count of tokens, not words or characters
print(tokens[:5]) # the actual token IDs
print(encoding.decode(tokens[:5])) # decode back to see what those specific tokens representCommon Mistakes
- Estimating cost or context usage by word count or character count instead of actual token count — the two can diverge significantly, especially for non-English text, code, or unusual vocabulary.
- Assuming every model uses the same tokenizer — different model families (and even different versions of the same family) frequently use different tokenizers, so a piece of text can tokenize into a meaningfully different number of tokens depending on which model you're targeting.
Interview Relevance
Q: "Why might the same English sentence use noticeably more tokens when translated into another language, even with the same word count?" Subword tokenizers (BPE) are typically trained predominantly on English-heavy corpora, so common English words often map to a single token, while words in less-represented languages more frequently get split into multiple smaller sub-word pieces, since the tokenizer's vocabulary contains fewer whole-word entries for that language.
Practice Question
A piece of Python code contains many multi-character variable names and symbols. Would you expect this to tokenize more or less efficiently (fewer tokens per character) than typical English prose, and why?
