Skip to content
Velloreth
Artificial Intelligence 10 min read

Understanding Context Windows: Why Bigger Isn't Always Better

A context window is how much text fits, not how much the model uses. How positional bias, context rot, and cost make effective context smaller than advertised.

By

A context window is a limit on how much text a model can accept. It says nothing about how well the model uses what you send. Those two numbers diverge more than the marketing suggests, and most prompt-quality problems that get blamed on the model are actually problems of what was put in front of it and where.

Treat the window as a budget under two constraints — attention and money — rather than a container you fill.

What the window physically is

The window is measured in tokens, not words. A token is roughly three-quarters of an English word on average; code, other languages, and unusual strings tokenize less efficiently, so the same character count can cost meaningfully more tokens depending on content.

Every request consumes the window with, in order:

ComponentTypically sits at
System prompt / instructionsBeginning
Conversation historyAccumulating in the middle
Retrieved documentsMiddle or end
Current questionEnd

Two properties follow. First, the limit is shared: your huge system prompt shrinks the room for the actual data. Second, attention is not uniform across positions — which is where the trouble starts.

There is also a cost dimension people underweight. Attention work grows with input length, input tokens are billed per request, and latency rises with prompt size. Modern serving tricks like prompt caching soften this for repeated prefixes, but a 100,000-token prompt is still a hundred-thousand-token decision every time you make it.

Tokenization and the size of what you send

Token counts surprise people in predictable directions:

ContentTypical token cost
English prose~4 characters per token
Source codeNoticeably denser — operators, indentation, identifiers split
JSONDense, and often half of it is punctuation and key names
Non-English textVaries widely; some languages use several tokens per character
Base64, hashes, logsWorst case — nearly random strings, poor compression

The practical failure is not hitting the wall; it is thinking you are under it. A quick estimate before a big prompt costs nothing:

# rough budget check — exact counts come from the model's tokenizer
def context_budget(system: str, history: str, docs: str, question: str,
                   limit: int) -> int:
    used = int(len(system + history + docs + question) / 4)  # ~4 chars/token
    remaining = limit - used
    if remaining < 0:
        raise ValueError(f"prompt exceeds window by {-used + limit} tokens")
    return remaining

remaining = context_budget(system, history, docs, question, limit=128_000)

Being off by a factor of two on tokenization is normal; being off by a factor of two on truncate-or-not is how a conversation silently loses its first thirty turns.

When the window gets squeezed

Overflow behavior deserves explicit attention because it fails quietly:

  • Hard rejection. The API refuses the request. Loud and easy to fix — you find out immediately.
  • Silent truncation. A client library drops the oldest messages to fit. The conversation continues; the model simply no longer knows what was said at the start.
  • Compaction. A harness summarizes old turns to reclaim space. Better than dropping, but the summary is lossy, and whatever was compressed out is gone from the model’s view without any signal that it left.

All three produce the same visible symptom — the model ignoring earlier facts or constraints — and none of them announce themselves in the response. If you maintain any kind of agent or chat harness, log the token count and the truncation/compaction events on every turn. “The model forgot” is almost always “we removed it and did not notice.”

Lost in the middle

The foundational finding, from Liu et al. (2023), is deceptively simple: place the one fact a model needs at the very start or very end of a long prompt and it finds it; bury the same fact in the middle and accuracy drops — a U-shaped curve across position, even for models explicitly marketed as long-context.

The shape is called primacy and recency bias. Transformers attend non-uniformly, and the practical consequence for prompt design is blunt: position is a feature you control.

  • Instructions that must never be forgotten: start and end.
  • The specific document answering this question: near the question.
  • Conversation history: assume the middle of it is approximately decorative once it grows long.

Extended-context variants do not automatically fix this. When a task fits inside both a standard and an extended model’s window, measured performance between them has been close to identical. A larger window buys capacity, not attention.

Context rot: quality decays before the limit

A second effect compounds positional bias. Chroma Research’s July 2025 study, “context rot,” tested 18 models across needle-retrieval, conversational-memory, and repeated-word tasks and found accuracy degrading as input length increased — often in non-uniform ways — well before any window was full.

Three findings from that work are directly useful:

  • Distractors hurt. Even a single irrelevant passage reduced performance against the needle-only baseline; four compounded the loss.
  • Structure can hurt. Models performed better on shuffled haystacks than on logically ordered ones — coherent narrative means more plausible-sounding material competing for attention.
  • Bigger windows did not prevent it. Rot is not a capacity problem you can buy your way out of.

Two later results reinforce the point. An EMNLP 2025 study found that even when models could perfectly retrieve the relevant evidence — retrieval failure ruled out — performance still dropped between roughly 14% and 85% as input length grew within claimed limits. And a 2026 study on lexical density identified a third axis: dense text, packed with distinct information, shrinks the usable window faster than sparse text of the same length. Filler is not neutral; it is load.

For agents, the effect shows up as premature termination: with very long accumulated contexts, models begin giving up or hedging toward confident wrong answers well before the window is exhausted. The context is technically there. The model stops being able to work with it.

Effective context, not advertised context

The gap between the two is your real design constraint. A 200,000-token window describes how much text the API will accept. Effective context — the length over which you can still expect reliable use of a fact you placed — is smaller, and shrinks further with distractors, dense material, and mid-context placement.

You can measure it for your own workload instead of trusting anyone’s number:

  1. Build 20 real questions from your data with known answers.
  2. Assemble contexts of increasing length: the answer plus controlled amounts of irrelevant-but-plausible material from the same corpus.
  3. Vary the answer’s position: start, middle, end.
  4. Record accuracy per (length, position) cell.

That grid is your prompt-design map. It tells you how much retrieved context to pass, where to put it, and how much noise costs you — on your data, with your model, which is the only measurement that transfers to production.

Long context versus retrieval, with actual numbers

The choice gets clearer when you do the token math instead of arguing in the abstract.

Suppose your knowledge base is 500 pages of documentation, roughly 150,000 words, which lands around 200,000 tokens before you add instructions or a question. That does not fit in one request on most models, so the “just send everything” option is already dead — and if a window that large existed, you would be paying for 200,000 input tokens on every single question.

Now shrink the corpus to a 30-page policy, about 12,000 words, roughly 16,000 tokens. That fits comfortably. Sending it whole is the right call: no index to build, no chunking to tune, no retrieval metric to miss, and no risk that the answer got split across a chunk boundary. The model sees everything, and for a document this size the position effects are manageable — especially if you keep instructions outside it and the question right after it.

The crossover is not one number, but three questions:

QuestionPoints toward long contextPoints toward retrieval
Does the whole corpus fit with room to spare?YesNo
Does answering require seeing documents together?Comparison, consistency checksUsually per-document facts
Does the corpus change often?StaticYes — index updates beat re-uploading
Must you cite or filter by source/date?Manual, weakNatural — metadata filters
How many calls?A fewThousands — cost compounds

There is a middle pattern worth knowing: map over documents, reduce to an answer. For a question spanning many files, retrieve or select each file, ask the model for a short structured extract from that one file, then run the final reasoning over the extracts. It is retrieval-shaped, but the expensive reasoning step sees a compact, ordered summary instead of a wall of text — sidestepping both the cost and the rot problem.

The mistake on both sides is symmetric. Fanning out long context when retrieval would be cheaper burns money for degraded quality; building a full retrieval stack when three documents fit in the prompt burns weeks for a problem you did not have. Measure the corpus against the window first — it takes one division.

Designing prompts for a budget

Practices that hold up across the research and in practice:

Front-load and back-load what matters. Put the goal and hard constraints at the top; restate the single most important rule and the current question at the bottom. The middle is where information goes to be forgotten.

Send less, but denser. Do not ship the whole document set “in case.” Retrieve the three passages that matter; drop near-duplicates; strip boilerplate. Every distractor has a measured cost.

Extract before you reason. When the task is long-context analysis, a cheap pattern wins twice: first ask the model to recite or list the relevant evidence, then run the actual reasoning on that short extract. It converts a long-context problem into a short one, and it makes the evidence inspectable.

Keep history trimmed. Summarize old turns, keep recent ones verbatim, and always re-inject the standing goal and constraints outside the summary so they cannot be summarized away.

Match window size to task shape, not to ambition:

TaskWindow choice
Single document Q&A, doc fits comfortablyLong context, no retrieval needed
Large, changing, or filterable corpusRetrieval with a modest context
Multi-turn agent workSmaller working set + external state
Codebase-scale questionsRetrieval + structure, not a full repo dump
Simple classification / extractionMinimal prompt, small window

Cache the stable prefix. If your system prompt and reference material are identical across calls, put them first and use prompt caching. It turns a recurring cost into a one-time one and improves time-to-first-token — which is the practical way to afford a longer prompt when you genuinely need one.

Bigger is only better when you have the room to spend

A larger window is real leverage for one class of problem: content that must be considered together, all at once, where splitting it changes the answer — comparing twenty contracts, reading a long novel for continuity, holding an entire code module in view.

It is a liability when it becomes a dumping ground. The pattern repeats across teams: the window grows, the prompt grows with it, and quality slides — then someone discovers that sending a third of the content in a better order produces better results than sending all of it.

The question was never “how much fits.” It is “how much can this model use for this task, at this position, surrounded by this much noise.” Answer it with a small grid of your own measurements, and the window stops being a spec you shop for and becomes a budget you manage.

Frequently asked questions

What is a context window?
It is the maximum number of tokens — words and word fragments — a model can accept in a single request, including the system prompt, conversation history, retrieved documents, and your question. Anything beyond the limit is rejected or truncated.
Do models use their full context window equally well?
No. Retrieval of relevant facts is consistently strongest near the beginning and end of the context and weakest in the middle — the lost-in-the-middle effect — and accuracy often declines with total length before any limit is reached.
How much context should I actually put in a prompt?
Only what the current step needs. Relevant, well-organized content beats exhaustive dumps: distractors measurably reduce performance, and each extra token adds cost and latency without adding value.
Is a long context window a replacement for RAG?
Sometimes, for a bounded corpus you read once — a few documents that fit comfortably in the window. For large, changing, or filterable collections, retrieval stays cheaper and more precise, and long context degrades as documents pile up.