The context window: a finite budget

Modern models can accept enormous amounts of text — hundreds of thousands of tokens, sometimes around a million. It's tempting to conclude: just put everything in and let the model sort it out. That's the single most common context-engineering mistake, and this lesson explains why it fails.

What the context window is. The context window is the maximum amount of text (measured in tokens — roughly, word-pieces) a model can consider in one go, covering both what you put in and what it generates. Even as windows have grown large, they remain finite — and, crucially, the model's ability to use that window well is even more limited than its size suggests.

The finite attention budget. Here's the key idea, and Anthropic explains the mechanism: a model has a limited attention budget it spends across all the tokens in its context. Under the hood, the model relates every token to every other token, so the number of relationships grows roughly with the square of the context length. As the context gets longer, that budget is stretched thinner — the model has more to attend to and less capacity per item. So a bigger context isn't free: every token you add is a claim on a finite, shared resource, and adding low-value tokens dilutes attention away from the high-value ones.

Why 'stuff everything in' backfires. Because of that finite budget, piling in everything you might need actively hurts. The irrelevant material doesn't just sit there harmlessly — it competes for attention, and it can degrade performance even when you're nowhere near the window's size limit (Module 3 covers the documented research on this: "context rot" and "lost in the middle"). The counterintuitive truth is that less, well-chosen context often beats more. This is why the goal from the last lesson — the smallest set of high-signal tokens — is stated the way it is.

A performance gradient, not a hard cliff. An honest nuance: this degradation is gradual, not a sudden failure at some threshold. Anthropic describes it as a performance gradient — as context grows, reliability tends to decline smoothly rather than falling off a cliff. So it's not that context past some line is useless; it's that every added token has a cost, and the returns diminish. You're always spending a budget, so spend it deliberately.

The practical stance. Treat the context window like a budget you're spending, not a bucket you're filling. Before adding anything, ask: is this high-signal enough to justify the attention it will consume? That question — asked about every document, example, and instruction — is the daily discipline of context engineering, and the rest of the course is about answering it well.

The mindset: the context window is finite, and the model's attention budget is more limited than the window's raw size — because the model relates every token to every other, longer context stretches that budget thinner. So "just stuff everything in" backfires: irrelevant tokens compete for attention and can degrade quality even below the size limit, which is why less, well-chosen context often beats more. The degradation is a smooth performance gradient, not a hard cliff, so the point isn't a magic line to stay under — it's that every token has a cost. Treat the window as a budget you spend deliberately, asking of everything you add: is this high-signal enough to justify the attention it consumes?