Tokens
Breaking a Sentence into Lego Bricks
5 min read
Breaking a Sentence into Lego Bricks
Imagine cutting up a newspaper article with scissors — not by word, but by syllables and common word chunks.
You'd have pieces like 'un', 'believ', 'able' for 'unbelievable'. AI models read and write using similar pieces called tokens. 'ChatGPT is amazing' might be 5 tokens. Tokens are how AI models measure text — it affects how much they can read at once and how much processing it costs.
In Plain English
A token is a chunk of text — roughly 3/4 of a word on average. AI models don't read letter by letter or word by word — they process tokens. Tokens are how AI providers measure usage and charge for API calls.
The Technical Picture
Tokens are the atomic units of text in LLMs, generated by a tokeniser (e.g., Byte-Pair Encoding). The model operates entirely on token IDs from its vocabulary. Token count affects memory, computation cost, and context window utilisation. Approximately 1000 tokens ≈ 750 English words.
Real-World Examples
- OpenAI charges per token used in API calls
- Claude's context window is measured in tokens (200K tokens ≈ 150,000 words)
- A typical tweet is about 15–20 tokens
Tokens are how AI models read and measure text — roughly 3/4 of a word each.