A context window is a budget, not a bucket

For a while I treated context as a bucket. Two hundred thousand tokens meant two hundred thousand tokens of room, so I put in everything I had. The results were not bad, but they never got better either.

Treating it as a budget changed that. Spending a slot means not spending it elsewhere, and the question turns from “what else can I add” into “what can I take out”.

The difference a budget makes

A bucket has no order and no priorities. A budget has both. Even for the same document, where it sits, whether it goes in summarised or truncated, is a choice — and every one of those choices moves cost and accuracy at the same time.

One rule did most of the work: when something new goes in, decide in the same breath what comes out to make room. If nothing could come out, the case for adding it was usually weak.

Being able to include something is not the same as needing to.

Working note, Feb 2026

The accounting I use

To run a budget you have to know what is left. Below is the smallest version I keep: every item carries a priority and a rough token count, and when the total overruns, the lowest priority goes first.

PARTS = [
    ("instructions", 1, 600),
    ("task", 1, 900),
    ("retrieved_docs", 3, 8000),
    ("history", 4, 4000),
]

def fit(parts, limit):
    kept, used = [], 0
    for name, prio, size in sorted(parts, key=lambda p: p[1]):
        if used + size > limit:
            continue          # dropped, and we know which
        kept.append(name)
        used += size
    return kept, used

Exact token counts turned out not to matter. What mattered was that the order of eviction lives in the code, so when the output went strange I could tell what had been cut.

Where this doesn't hold

It fits long conversations badly. Once a thread runs on, choosing what to drop becomes a judgement, and I could not write that judgement down as a rule.

So conversations get a rolling summary and one-shot tasks get the budget rule. Both attempts to unify them failed.