How LLM tokens really work (and why your word count is a bad estimate)

Ask most people how long an LLM prompt is and they count words. The model does not see words — it sees tokens, and the mapping between the two is not the obvious 1:1 you might expect. Understanding that mapping is the difference between a prompt that fits the model, a bill you can predict, and a request that silently truncates.

What a token actually is

Large language models work with a fixed vocabulary of token ids — integers, typically between 0 and a few hundred thousand. Text is turned into those ids by a tokenizer, which for modern GPT-family models is Byte-Pair Encoding (BPE). BPE learns the most common substrings across a training corpus and merges them into single tokens, so frequent chunks like " the" or "ing" become one id while rare words are spelled out byte by byte.

That has three consequences people trip over:

• Whitespace is part of the token. " the" and "the" are different tokens because the leading space is baked in. Trailing spaces in your prompt literally cost tokens.
• Punctuation is often its own token. A comma, a period and a colon each take an id, so a tidy bulleted list can cost more than a run-on sentence.
• Non-English text is expensive. Languages the tokenizer saw less of during training get split into more pieces. A sentence in English might be 12 tokens; the same idea in a less-represented script can be 25.

Why your word count lies

English prose averages around 1.3 tokens per word, but the variance is wide. Short common words are roughly 1:1. Long technical or compound words can be 3 or 4 tokens each. Code is the worst case: every brace, semicolon and quote mark is a token, so a JSON object can run two to three times its character count in tokens.

This is why pasting a 4,000-word article into a chat and assuming "it's fine" fails. At 1.3 tokens per word that is already 5,200 tokens; if the text is dense or technical, it can be 7,000+. The AI Token Counter runs the real BPE tables (o200kbase for GPT-4o, cl100k_base for GPT-4) in your browser so the number is exact for OpenAI models, not a guess from a word count.

Context windows are a budget, not a limit you hit

A model's context window — 128k for GPT-4o, 200k for Claude, 1M for Gemini — sounds enormous until you account for everything that shares it:

• The system prompt
• The full conversation history
• Retrieved documents you inject for RAG
• Tool-call scaffolding the API adds around each message
• The reply you have not generated yet

The reply is the part people forget. The output has to live in the same window, so a 128k context does not mean 128k of input — it means 128k of input plus output. If you fill the window, the model has no room to answer and will truncate or refuse.

The LLM Context Window Calculator does this arithmetic for you: split the request into system, context and user parts, add realistic chat overhead (about four tokens per message plus a priming token), reserve output tokens, and see whether the request fits before you send it.

A practical workflow

1. Count the raw input with the token counter so you know your starting point.
2. Reserve at least the maximum output length you want — 1,000 to 4,000 tokens for most answers.
3. Add chat overhead: roughly four tokens per message plus one priming token.
4. Compare the total against the model's window with the context calculator.
5. If you are over, the Context Window Packer helps you trim retrieved chunks to the largest set that still fits.

Token counts drive cost, not just fit

Every token you send is billed at the input rate and every token the model produces at the output rate. A request that is 30 percent padding and duplicated context is 30 percent more expensive per call, and across thousands of calls that compounds. Counting once, trimming once, and reusing the result is the cheapest optimisation in LLM engineering. The LLM API Cost Calculator turns those token counts into per-call and per-month spend across models, so you can see the tradeoff before you commit.

The one rule to take away

Never estimate a prompt in words. Count it in tokens, reserve room for the reply, and compare against the window. Everything else — cost, latency, whether the model answers at all — follows from those three numbers.