Generative AI & LLM Engineering · Chapter 2 · ~15 min · Interactive Visuals

Tokens: Watch Text Split into Pieces

LLMs do not read words or letters. They digest streams of subword chunks called tokens. Understanding how tokenization works reveals why models struggle with simple spelling, how whitespace changes everything, and why non-English prompts cost more.

See text sliced into subwords and IDs
Watch Byte-Pair Encoding (BPE) merge in action
Solve the 'strawberry' letter counting mystery
Understand the whitespace & capitalization traps

1Why not letters or whole words?

Why don't language models just read letters or full words?

Single characters (A-Z)

Too small. A single paragraph would take 1,000 steps to process. The model loses sight of the context and computes very slowly.

Whole dictionary words

Too big. English alone has millions of words, plus typos, slang, and names. The vocabulary table would explode in size.

Subwords (Tokens) ✔

The sweet spot (~50,000 to 100,000 pieces). Frequent words stay whole, while rare words and typos get chopped into smaller pieces.

Tokens are the universal currency of LLMs. You are billed in tokens, context limits are measured in tokens, and generation speed is rated in tokens per second.

Why don't LLMs use a pure whole-word dictionary?

2Interactive Token & Token ID Inspector

A model maps each token in its vocabulary to an integer ID. The visualizer below uses an approximate teaching tokenizer and synthetic demo IDs; real splits and IDs differ across models.

Try typing or pick an example:

Approximate teaching split (8 demo pieces):

TheDemo #464·catDemo #8415·satDemo #4332·onDemo #319·theDemo #1820·warmDemo #6845·matDemo #2618.Demo #13

Demo pieces

Characters

Chars / Token

3.5

Notice how the dot “·” indicates a leading space. Many tokenizers treat " cat" and "cat" differently; exact splits and IDs vary by tokenizer. The IDs above are teaching-only examples, not IDs from a particular model.

You can test any custom prompt or paste long documents into the dedicated AI Token Counter to check token counts for supported model families.

In the phrase 'the cat', does the space belong to 'cat' or stand alone?

3How tokenizers learn: Byte-Pair Encoding (BPE)

How does the tokenizer know where to slice words? Most models use an algorithm called Byte-Pair Encoding (BPE). It starts with raw letters and repeatedly merges the most frequent adjacent pairs. Step through the simulation below:

Step 0: Raw CharactersStep 1 of 4

Before training merges, everything starts as individual letters plus special space markers.

the·cat·sat·the

Because common prefixes and suffixes like "ing", "pre", and "tion" appear thousands of times, BPE naturally turns them into reusable subword blocks!

What does Byte-Pair Encoding (BPE) do during its training phase?

4Token Traps: Spaces, Caps & The Strawberry Mystery

Because models think in token chunks rather than letters, subtle quirks arise that trip up users and developers alike.

Why can't ChatGPT count letters in "strawberry"?

Toggle views to see why counting letters is surprisingly hard for models.

The LLM receives two opaque numbers from its tokenizer vocabulary:

Token #21884

"straw"

Token #14198

"berry"

The internal transformer never inspects individual letters! Unless prompted to spell it out letter-by-letter first, it has to hallucinate from general statistical memories of the word.

The Whitespace and Capitalization Trap:

Without space

"apple"

Demo ID #28072

With space / alternate case

"·apple"

Demo ID #12845

💡 Key takeaway: A leading space can change the tokenization and token IDs. Always keep prompt templates consistent with whitespace.

When you ask an LLM to count characters, reverse a string, or write rhyming poems, asking it to "First spell the word with dashes (s-t-r-a-w...)" forces it to emit single-letter tokens, dramatically improving accuracy!

Why does asking an LLM to 'spell out the word letter-by-letter first' help it count characters?

5The Token Tax: Code and Multilingual Costs

If you write in English, one token is roughly 4 characters (¾ of a word). But for languages with non-Latin scripts (like Hindi, Arabic, or Japanese), or heavily indented code, token efficiency drops.

Token counts vary by model, tokenizer, and exact text. Some tokenizers encode some non-English scripts less efficiently than English; the example counts below are illustrative, not universal measurements.

English7 tokens (1.0x (baseline))

"Hello, how are you today?"

Spanish9 tokens (illustrative)

"Hola, ¿cómo estás hoy?"

Japanese18 tokens (illustrative)

"こんにちは、今日はお元気ですか？"

Hindi23 tokens (illustrative)

"नमस्ते, आज आप कैसे हैं?"

In some model and language combinations, equivalent non-English prompts can use more tokens than English. Check the tokenizer and prices for the model you plan to use.

Cost calculation tip: When designing applications for global audiences, factor in that non-English prompts require 2x to 4x more tokens. Calculate your estimated budget with the LLM API Cost Calculator.

6Chapter challenge

Token Master Challenge

Answer all four questions correctly to lock in your mastery of tokens.

0 / 4 correct

1What is a token in the context of LLMs?
2Does an LLM treat ' apple' (with space) and 'apple' (without space) as the same token?
3Why do non-English languages often cost more to run through an LLM?
4Which of the following explains why LLMs struggle to count letters in 'strawberry'?

7Recap

Tokens are subwords: A sweet spot between single letters and whole dictionaries (~4 characters per token in English).
Numbers, not words: The model looks up tokens in a fixed vocabulary and only processes their numeric IDs.
Spaces count: " hello" and "hello" are completely distinct tokens.
BPE merges frequent pairs: Byte-Pair Encoding builds the vocabulary by merging frequent neighbor pairs.
The Token Tax: Code, symbols, and non-English scripts take significantly more tokens for the same information.

Solve the challenge above to complete the chapter, or