Skip to content

Fit a corpus to a context budget

You have a pile of documents and a model with a finite window. This is the four-step pattern: build documents, ask what fits, reduce, and place the result.

Reduction lives in anyinfer.context, an optional dependency-free subpackage. Your application decides what exists and what is safe to send; the library decides what fits. See context reduction for why that line is drawn there.

1. Build documents

from anyinfer import context

documents = [
    context.ContextDocument.of("src/auth/credentials.py", credentials_source),
    context.ContextDocument.of("src/auth/tokens.py", tokens_source),
    context.ContextDocument.of("README.md", readme_text, pinned=True),
]

of() computes the digest, detects the language from the path, and derives a structural extract — the signatures-and-imports view the tiered strategy falls back to. Pass extract="" to skip extraction, or language= to override detection.

pinned=True means the user explicitly chose this file: it sorts ahead of everything and is never chunked.

2. Ask what fits

Build the request you would send without the corpus, and ask what is left over:

import anyinfer as ai

messages = [ai.user("How does credential resolution work?")]
budget = client.budget(messages, target="anthropic:claude-sonnet-4-5")

max_tokens = budget.remaining_tokens
if max_tokens is None:
    # The window is unknown; the library will not guess one for you.
    max_tokens = 8_000

That if is deliberate. When a target's context window is unknown, remaining_tokens is None and the fallback is your explicit choice, made where you can see it.

3. Reduce

reduction = context.select(
    documents,
    query="how does credential resolution work?",
    max_tokens=max_tokens,
)

The default auto strategy sends everything when it fits and falls back to tiered when it does not. Name a strategy when you want a specific shape:

context.select(documents, query, max_tokens=max_tokens, strategy="ranked")   # whole files
context.select(documents, query, max_tokens=max_tokens, strategy="tiered")   # full coverage
context.select(documents, query, max_tokens=max_tokens, strategy="packed")   # chunk-level

4. Place the result and check what happened

messages.insert(0, ai.user(reduction.text))
result = client.generate(messages, target="anthropic:claude-sonnet-4-5")

The envelope goes in your message — the library never modifies the request. Then check what it cost you:

if not reduction.complete:
    log.info("context reduced: %s", reduction.summary())
    # ranked: 12 of 340 document(s); ~7900 of 8000 tokens; 328 omitted; limited by tokens

summary() is content-free: counts and ceilings, never paths or content. metadata() gives the full machine-readable record for a debug pane.

Observe reductions like any other event

class ContextWatcher:
    def on_event(self, event):
        if isinstance(event, ai.ContextReduced):
            metrics.gauge("context.omitted", event.omitted_count)

reduction = context.select(documents, query, max_tokens=max_tokens,
                           observer=ContextWatcher())

Reuse the ranking cache across turns

An interactive application ranks the same corpus on every turn. Build the statistics once:

cache = context.build_rank_cache(documents)

for turn in conversation:
    reduction = context.select(documents, turn.text, max_tokens=budget_for(turn),
                               rank_cache=cache)

Invalidation is yours: key the cache on a corpus hash and rebuild when the corpus changes. A stale cache produces undefined ranking, not an error.

Keep the prompt prefix stable

Documents render in path order by default, whatever their rank. Two turns that select the same documents produce byte-identical text, so provider prompt caches keep hitting. If you would rather have strongest-first ordering:

context.select(documents, query, max_tokens=max_tokens, render_order="rank")

When it will never fit

At some size no fidelity reduction is enough, and the answer is more requests rather than fewer tokens. That is distill — it reads everything and writes something shorter, reporting exactly how many calls that took.

See also