Generative AI & LLM Engineering · Chapter 3 · ~18 min · Interactive Lab
Temperature, Sampling & Context Windows
How do models turn raw numbers into creative text? Discover how Temperature, Top-P (Nucleus), and Top-K control randomness, how to budget context windows, and why facts get "lost in the middle."
- Master Temperature, Top-P and Top-K live
- Prevent loops with repetition penalties
- Budget context windows (prompts vs outputs)
- Avoid the 'lost in the middle' attention trap
1Inside the Logit-to-Probability engine
Before an LLM picks a token, its final neural layer outputs a raw numerical score for every single word in its vocabulary (all ~100,000 of them). These raw unnormalized scores are called logits.
To turn logits into clean percentages that add up to 100%, the model runs them through a mathematical formula called Softmax:
probability = exp(logit / temperature) / sum(exp(all_logits / temperature))What does the Softmax function do to the model's raw logits?
2The Sampling Lab: Temperature, Top-P & Top-K
How should the model pick from those percentages? You have three primary dials to steer generation:
Controls sharpness. At T=0, it greedily picks #1 every time. High T flattens the distribution so unlikely tokens can win.
Dynamic threshold. Accumulates probabilities until reaching P (e.g. 0.90) and truncates the rest of the tail.
Hard count limit. Discards everything beyond the top K most likely tokens, preventing completely nonsensical picks.
Common starting range; tune for the task.
Keeps the smallest pool whose cumulative prob exceeds P.
Cuts off everything below the top 6 candidates.
"coffee""tea""water""juice""soup""oil"Which sampling setting usually makes token selection least random for the same prompt and model?
3Stopping Infinite Loops (Frequency & Presence Penalties)
Have you ever seen an AI get stuck repeating: "In conclusion, furthermore, in conclusion, furthermore..."?
Because generating a word makes that word appear in its own history, language models can fall into positive feedback loops. LLM APIs provide two penalties to break this cycle:
One-time penalty if a word has appeared at all in the text so far. Encourages the model to introduce fresh topics and vocabulary.
Grows larger the more times a specific token is repeated. Directly penalizes verbatim phrase repetitions.
4Context Window Budgeting (Input vs Output)
A common mistake when calling LLM APIs is forgetting that the context window is a shared pool between your input prompt and the model's generated response.
If your model has a 32,000-token limit and your prompt is 30,000 tokens long, the model can generate at most 2,000 tokens before crashing into the wall. Test different configurations below:
max_tokens (completion tokens). If a response cuts off mid-sentence, it almost always means it hit your configured max completion token ceiling!Your prompt is 110,000 tokens in a 128,000-token context window model. What is the maximum length the answer can be?
5The 'Lost in the Middle' Trap
Even when a model technically supports 100K or 1M tokens, it doesn't pay equal attention to every position. Research demonstrates that LLMs exhibit a U-shaped attention curve:
Simulated attention weight across a long document prompt:
Dramatically lower recall! The model often skips or confuses facts buried in the middle of 20+ pages.
💡 Engineering Rule: Put critical instructions and formatting constraints at the very top or very bottom of the prompt — never buried halfway through your text.
6Chapter challenge
Sampling & Context Mastery
Answer all four questions correctly to complete Chapter 3.
1Which temperature is best suited for extracting JSON fields from an invoice?
2How does Top-P (Nucleus Sampling) work?
3What causes a model's response to abruptly truncate mid-sentence?
4According to the 'Lost in the Middle' phenomenon, where should critical instructions be placed in a long prompt?
7Recap
- Logits to Probabilities: Raw model outputs pass through Softmax to produce token odds.
- Temperature dial: T=0 for code & extraction; T=0.7 for chat; T=1.2+ for creative writing.
- Top-P & Top-K: Filter out the improbable tail of bizarre tokens.
- Context Budgeting: Input tokens + Output tokens must fit within the model's max window.
- Lost in the Middle: Keep your most critical facts and instructions at the start or end of long prompts.
Solve the challenge above to complete the chapter, or