Capabilities

Provenance-tagged model metadata: every value knows whether it was cataloged, discovered, probed, or defaulted. The reasoning is in capabilities and provenance; token estimation and the budget calculator are explained in token estimation and context budgets.

anyinfer.ModelCapabilities dataclass

ModelCapabilities(
    context_window: Sourced[int] | None = None,
    max_output_tokens: Sourced[int] | None = None,
    features: Sourced[Feature] = Sourced(
        Feature(0), "default"
    ),
    pricing: Sourced[Pricing] | None = None,
    default_temperature: Sourced[float] | None = None,
    default_top_p: Sourced[float] | None = None,
    local: LocalModelInfo | None = None,
    operations: Sourced[frozenset[InferenceOperation]]
    | None = None,
    embedding: EmbeddingCapabilities | None = None,
)

What a model can do, as far as we know.

Attributes:

Name	Type	Description
context_window	Sourced[int] | None	Maximum tokens of input context, with provenance; None when unknown.
max_output_tokens	Sourced[int] | None	Maximum tokens one response may contain, with provenance; None when unknown.
features	Sourced[Feature]	Which Feature flags the model supports, with provenance.
pricing	Sourced[Pricing] | None	Per-million-token pricing, when known.
default_temperature	Sourced[float] | None	The temperature this provider applies when a request sends none, with provenance; None when the provider does not document one.
default_top_p	Sourced[float] | None	The nucleus-sampling cutoff this provider applies when a request sends none, with provenance; None when undocumented.
local	LocalModelInfo | None	Facts about the local artifact, for locally-run models only.
operations	Sourced[frozenset[InferenceOperation]] | None	Which inference operations this model serves, with provenance; None when unknown. Deliberately not a generation Feature flag — "can embed" is a different operation, not a feature of chat — and deliberately not invented from the provider-level operations set, which says what the adapter speaks, not what one model does.
embedding	EmbeddingCapabilities | None	Vector facts this model states about itself — the dimensions and input ceiling a listing or a pinned catalog row declares. None when nothing model-level is known, which is the honest state for most providers: it is filled from what a model actually reports, never from the provider's documentation about a different model.

The two sampling defaults exist so an application can say what "provider default" means instead of only that it is one. They are populated from a provider's own documentation and nowhere else; never probed, never inferred from a sibling provider, never carried over from a model family. A provider whose documentation states no default keeps None indefinitely, and that is the correct final state for it rather than a gap waiting to be filled: an invented number presented beside a provenance tag is precisely the estimate-as-authority this type exists to prevent.

overlay

overlay(other: ModelCapabilities) -> ModelCapabilities

Layer other on top of this, field by field, stronger provenance winning.

This is the assembly rule: later layers override earlier ones, but a weaker-provenance value never displaces a stronger one.

anyinfer.Pricing dataclass

Pricing(
    input_per_1m: Decimal,
    output_per_1m: Decimal,
    cache_read_per_1m: Decimal | None = None,
    cache_write_per_1m: Decimal | None = None,
    currency: str = "USD",
    per_search_unit: Decimal | None = None,
    per_server_tool_use: Mapping[str, Decimal] = dict(),
)

Per-million-token pricing used to compute cost_usd.

Cache rates are optional and default to unknown rather than to the input rate. A provider that discounts cached prompt tokens but whose discount we have not recorded must not be billed as though the discount were zero or as though it were free — an unknown rate leaves cached tokens priced as ordinary input, which is the same answer this library gave before cache accounting existed, and is wrong in only one direction that a caller can reason about.

Attributes:

Name	Type	Description
input_per_1m	Decimal	Price per one million prompt tokens.
output_per_1m	Decimal	Price per one million generated tokens.
cache_read_per_1m	Decimal | None	Price per one million prompt tokens served from the provider's cache, or None when the rate is not recorded.
cache_write_per_1m	Decimal | None	Price per one million prompt tokens written into the cache, or None when the rate is not recorded. Several providers charge a premium for a write, so this is not assumed to be a discount.
currency	str	Currency code the prices are quoted in.
per_search_unit	Decimal | None	Price per one billed search unit, for rerank providers that bill by searches rather than tokens. None when the provider does not bill this way or the rate is not recorded — a rerank cost stays unknown rather than being priced through an invented token equivalence.
per_server_tool_use	Mapping[str, Decimal]	Price per invocation of a provider-run tool, keyed by kind. Web search is billed per search rather than per token, so a generation that searched costs more than its token counts say — and a caller who set max_uses did so precisely because that line item is real. A kind absent from the mapping is not free; it is unpriced, and makes the whole cost unknown rather than being silently counted as zero.

anyinfer.Feature

Bases: Flag

Capabilities a model may support.

Structured-output mechanism selection reads these in the order GRAMMAR > JSON_SCHEMA > JSON_MODE > prompt injection.

CACHE_USAGE and CACHE_PLACEMENT are deliberately separate facts: reporting what the prompt cache did is not the same as accepting instructions about where it should apply, and a provider may do either without the other.

LOGPROBS is a model fact rather than a provider one: several dialects accept the field on their completion models and reject it on their reasoning models, so it is carried here beside the input modalities instead of on the descriptor.

anyinfer.Mechanism module-attribute

Mechanism = Literal[
    "grammar", "json_schema", "json_mode", "prompt"
]

How structured output was requested of the provider.

anyinfer.Sourced dataclass

Sourced(value: _T, provenance: Provenance = 'default')

Bases: Generic[_T]

A capability value paired with its provenance.

outranks

outranks(other: Sourced[_T] | None) -> bool

Whether this value's provenance is at least as strong as other's.

anyinfer.Provenance module-attribute

Provenance = Literal[
    "catalog", "discovered", "probed", "default", "override"
]

Where a capability value came from, weakest (default) to strongest (override).

anyinfer.Health dataclass

Health(ok: bool, detail: str = '')

Result of a provider's cheap readiness probe.

Attributes:

Name	Type	Description
ok	bool	Whether the provider answered its readiness probe successfully.
detail	str	Short human-readable explanation, most useful when ok is false.

anyinfer.DiscoveredModel dataclass

DiscoveredModel(
    id: str, capabilities: ModelCapabilities | None = None
)

A model reported by a provider's listing endpoint.

capabilities carries only fields the provider actually reported; the capability assembler tags them "discovered".

Attributes:

Name	Type	Description
id	str	The model identifier exactly as the provider lists it.
capabilities	ModelCapabilities | None	Capability fields the listing reported; None when the provider lists ids only.

anyinfer.LocalModelInfo dataclass

LocalModelInfo(
    artifact_size_bytes: int | None = None,
    parameter_size: str | None = None,
    quantization: str | None = None,
    est_ram_bytes: int | None = None,
    est_vram_bytes: int | None = None,
    observed_vram_bytes: int | None = None,
)

Facts about a local model artifact, used for tuning and recommendation.

Attributes:

Name	Type	Description
artifact_size_bytes	int | None	On-disk size of the model weights.
parameter_size	str | None	Parameter count as the runtime reports it (e.g. "7B").
quantization	str | None	Quantization scheme of the artifact (e.g. "Q4_K_M").
est_ram_bytes	int | None	Estimated system memory needed to run the model.
est_vram_bytes	int | None	Estimated GPU memory needed to run the model.
observed_vram_bytes	int | None	GPU memory actually measured in use while the model was loaded, when the runtime reports it.

anyinfer.ContextBudget dataclass

ContextBudget(
    context_window: Sourced[int] | None,
    estimate: RequestEstimate,
    output_reserve_tokens: int,
    headroom_tokens: int,
    pricing: Sourced[Pricing] | None = None,
)

A request's estimated size held against a model's known capacity.

The verdict is tri-state: fits is True/False when the context window is known, and None when it is not — an unknown capacity is reported as unknown, never guessed.

Attributes:

Name	Type	Description
context_window	Sourced[int] | None	The model's context window with its provenance, or None when nothing trustworthy is known.
estimate	RequestEstimate	The per-component input-token estimate.
output_reserve_tokens	int	Tokens reserved for the response.
headroom_tokens	int	Safety margin against estimation error.
pricing	Sourced[Pricing] | None	The model's per-token rates with their provenance, when known.

input_allowance_tokens property

input_allowance_tokens: int | None

Tokens the input may spend, or None when the window is unknown.

remaining_tokens property

remaining_tokens: int | None

Allowance left after the estimated input; negative when over budget.

This is the number an app packs context against: keep adding material while it stays positive.

fits property

fits: bool | None

Whether the estimated request fits the allowance; None when unknowable.

estimated_cost property

estimated_cost: CostEstimate | None

A preflight cost range, or None when no trustworthy pricing exists.

Estimated money never mixes with reported money: cost_usd is only ever computed from provider-reported usage, and this range is only ever computed from the estimate.

anyinfer.TokenEstimate dataclass

TokenEstimate(tokens: int, floor: int)

A token count carried as a planning estimate and a defensible lower bound.

Attributes:

Name	Type	Description
tokens	int	The planning figure, deliberately conservative-high.
floor	int	A lower bound the true count is not realistically below. An exact tokenizer sets floor == tokens.

__add__

__add__(other: TokenEstimate) -> TokenEstimate

Sum two estimates component-wise.

anyinfer.TokenEstimator

Bases: Protocol

Pluggable token counting.

Implementations may be heuristic (the shipped default) or exact (tiktoken, a provider's tokenize endpoint). Exact implementations should return TokenEstimate(n, n) so the gate can act on their counts with full force.

estimate

estimate(text: str) -> TokenEstimate

Estimate the token count of text.

anyinfer.HeuristicTokenEstimator dataclass

HeuristicTokenEstimator(multiplier: float = 1.0)

The dependency-free default: token counts from UTF-8 byte counts.

Attributes:

Name	Type	Description
multiplier	float	Calibration factor applied to the planning estimate, for providers whose transport envelope inflates reported prompt tokens beyond the serialized bytes. The floor is never inflated — envelope overhead is not something a lower bound may claim.

__post_init__

__post_init__() -> None

Reject non-finite or non-positive calibration factors.

estimate

estimate(text: str) -> TokenEstimate

Estimate tokens as ceil(bytes/3), with a bytes//8 floor.

anyinfer.TiktokenEstimator

TiktokenEstimator(encoding: str | None = None)

Exact token counts via tiktoken, for models whose encoding it publishes.

Exact for OpenAI's own models and for the many open-weight families that adopted the same encodings. Not exact for Anthropic, Gemini, or Cohere, whose tokenizers are not published — those fall back to DEFAULT_ENCODING, which is a better guess than counting bytes but is still a guess, and is reported as one.

Instances cache their encodings process-wide, so for_model may be called per request. The first load of a given encoding is not cheap: tiktoken fetches the vocabulary over the network unless it is already in its cache (TIKTOKEN_CACHE_DIR). That happens at construction rather than at first count, deliberately — a server should discover a missing vocabulary while starting up, not part-way through a request. Deployments that must not reach the network at run time should pre-warm the cache in their image build.

Parameters:

Name	Type	Description	Default
encoding	str | None	Pin one encoding for every model instead of selecting per model. Use when serving a known open-weight family through an OpenAI-compatible endpoint, where the model id tells tiktoken nothing.	None

Raises:

Type	Description
ConfigError	If tiktoken is not installed.

tokenizer_kind class-attribute

tokenizer_kind: TokenizerKind = 'tiktoken'

Which exact-counting strategy this implements, matched against a provider's declared TokenCalibration.tokenizer before its counts are trusted as exact.

for_model

for_model(
    provider_id: str, model: str
) -> TiktokenEstimator

Return an estimator using the encoding this model actually uses.

Parameters:

Name	Type	Description	Default
provider_id	str	The resolved provider; unused today and accepted because the mapping from model id to encoding is a provider's fact, and a future provider-specific table belongs here rather than at the call site.	required
model	str	The resolved model id.	required

Returns:

Type	Description
TiktokenEstimator	self when the instance pins an encoding; otherwise an instance holding the
TiktokenEstimator	encoding tiktoken names for this model, or the default when it names none.

estimate

estimate(text: str) -> TokenEstimate

Count text with this instance's encoding.

Returns:

Type	Description
TokenEstimate	TokenEstimate(n, n) when the encoding is known to be this model's, so the
TokenEstimate	pre-dispatch gate can act on the floor with full force. Otherwise the count is
TokenEstimate	the planning figure and the floor is held slightly below it, because a
TokenEstimate	substituted encoding can over-count and a floor that over-claims would refuse
TokenEstimate	requests that fit.

anyinfer.AnthropicCountTokensEstimator

AnthropicCountTokensEstimator(
    *,
    api_key: str,
    model: str = "claude-sonnet-4-5",
    base_url: str = "https://api.anthropic.com",
    base: TokenEstimator | None = None,
    transport: Any = None,
    max_entries: int = DEFAULT_CACHE_ENTRIES,
)

Bases: _RemoteCountingEstimator

Exact counts from Anthropic's own POST /v1/messages/count_tokens.

Anthropic does not publish its tokenizer, so this endpoint is the only exact count available for Claude models — TiktokenEstimator falls back to an OpenAI encoding for them, which is a better guess than counting bytes but is still a guess.

The endpoint counts a whole message list, not a string, so each text is counted as a single user turn. That includes Anthropic's own per-message framing, which makes the count marginally high for a text that will travel as part of a longer conversation — high is the safe direction for a planning figure, and the framing is exactly what estimate_request would otherwise add by heuristic.

Parameters:

Name	Type	Description	Default
api_key	str	The credential to count with. Counting is free but authenticated.	required
model	str	The model whose tokenizer to count with. Anthropic requires one, and the count is model-specific.	'claude-sonnet-4-5'
base_url	str	Override the API root, for a proxy or a test transport.	'https://api.anthropic.com'
base	TokenEstimator | None	Estimator for texts that were not prewarmed.	None
transport	Any	An HTTP transport to use instead of the default.	None
max_entries	int	Counted texts to remember.	DEFAULT_CACHE_ENTRIES

tokenizer_kind class-attribute

tokenizer_kind: TokenizerKind = 'anthropic_count_tokens'

Matched against a provider's declared TokenCalibration.tokenizer before these counts are trusted as exact.

prewarm async

prewarm(texts: Sequence[str]) -> None

Count each unknown text, tolerating a service that will not answer.

aclose async

aclose() -> None

Close the counting transport.

anyinfer.LlamaServerTokenizeEstimator

LlamaServerTokenizeEstimator(
    *,
    base_url: str = "http://127.0.0.1:8080",
    api_key: str | None = None,
    base: TokenEstimator | None = None,
    transport: Any = None,
    max_entries: int = DEFAULT_CACHE_ENTRIES,
)

Bases: _RemoteCountingEstimator

Exact counts from a running llama-server's POST /tokenize.

The tokenizer here is the one loaded with the weights, so the count is exact for whatever GGUF the server is serving — including quantized community models whose vocabulary is not published anywhere a local tokenizer could find it.

Unlike the hosted case this is a loopback call, but the reasoning is the same: the estimator protocol is synchronous and the event loop is shared, so the fetch happens ahead of the counting rather than inside it.

Parameters:

Name	Type	Description	Default
base_url	str	Where the server is listening.	'http://127.0.0.1:8080'
api_key	str | None	A credential, for a server started with --api-key or behind a proxy.	None
base	TokenEstimator | None	Estimator for texts that were not prewarmed.	None
transport	Any	An HTTP transport to use instead of the default.	None
max_entries	int	Counted texts to remember.	DEFAULT_CACHE_ENTRIES

tokenizer_kind class-attribute

tokenizer_kind: TokenizerKind = 'llama_server_tokenize'

Matched against a provider's declared TokenCalibration.tokenizer before these counts are trusted as exact.

prewarm async

prewarm(texts: Sequence[str]) -> None

Tokenize each unknown text, tolerating a server that is not up.

aclose async

aclose() -> None

Close the counting transport.

anyinfer.PrewarmsCounts

Bases: Protocol

An estimator whose exact counts come from a service rather than from local code.

The optional async half of TokenEstimator. A client that knows it is about to size a request awaits prewarm once, and the synchronous estimate calls that follow read what it fetched. An estimator that does not implement this is used exactly as before — this is an extension, not a requirement.

prewarm async

prewarm(texts: Sequence[str]) -> None

Fetch exact counts for these texts, ignoring ones already known.

Never raises for a service problem: an estimator that cannot reach its counting endpoint degrades to its base estimator, because a request that could have been sized approximately should not fail outright for want of an exact number.

anyinfer.TargetAwareTokenEstimator

Bases: Protocol

An estimator that can specialize itself for one resolved target.

An optional extension to TokenEstimator, not a replacement: the tokenizer a count needs depends on the model, and TokenEstimator.estimate sees only text. An estimator implementing this is asked for a specialized instance once per target; one that does not is used as-is, which is why the shipped heuristic needs no change.

for_model

for_model(provider_id: str, model: str) -> TokenEstimator

Return the estimator to use for one provider and model.

anyinfer.RequestEstimate dataclass

RequestEstimate(
    messages: TokenEstimate,
    tools: TokenEstimate,
    schema: TokenEstimate,
    envelope: TokenEstimate = TokenEstimate(0, 0),
    unpriced_parts: int = 0,
)

Content-free size accounting for one request, by component.

The breakdown follows the typed request itself: what the caller said, what tools were offered, and what schema was attached — the three things that occupy input tokens on any provider.

Attributes:

Name	Type	Description
messages	TokenEstimate	The conversation, including per-message wire-framing overhead.
tools	TokenEstimate	Serialized tool specifications, when any were offered.
schema	TokenEstimate	The structured-output schema, when one was requested. Counted whether the wire carries it natively or the core injects it into the prompt — either way it occupies input tokens.
envelope	TokenEstimate	What the provider's own transport adds around all of the above, from its declared TokenCalibration. Zero for every provider that counts what it was sent, and floor-free always: an envelope correction is believed, not proven.

tokens property

tokens: int

Total planning estimate across all components.

floor property

floor: int

Total lower bound across all components.

anyinfer.TokenCalibration dataclass

TokenCalibration(
    multiplier: float = 1.0,
    overhead_tokens: int = 0,
    tokenizer: TokenizerKind | None = None,
    tokenizer_provenance: Provenance = "default",
)

How much a provider's own envelope inflates the prompt it is sent.

Serialized request bytes are not what every provider counts. Some wrap the caller's messages in a transport of their own before the model ever sees them — a session API that prepends its harness, a tool scaffold, a service-side system preamble, and then bill (and window-check) the inflated total. Estimating such a provider from message bytes alone under-counts every request, and the under-count is systematic rather than noise, so budgets stay optimistic right up to the overflow.

A provider therefore declares its own correction, and only the planning figure moves:

• multiplier scales content that grows with the prompt.
• overhead_tokens adds what the envelope costs regardless of prompt size.

Neither touches the estimate's floor. The floor exists to refuse requests before dispatch, and a lower bound may only claim tokens the provider certainly charges — envelope overhead is a correction we believe, not one we can prove.

A provider also declares how its tokens can be counted exactly, when they can be. That belongs here rather than in a table beside it because it is the same fact from the same source: what this provider's numbers mean and where an exact one comes from.

Attributes:

Name	Type	Description
multiplier	float	Factor applied to the planning estimate of prompt-proportional content. 1.0 means the provider counts what was sent.
overhead_tokens	int	Flat tokens the envelope adds per request, counted once.
tokenizer	TokenizerKind | None	Which exact-counting strategy this provider's models can use, or None where none exists and the heuristic is the honest answer. Naming a strategy does not install one — the estimators live behind extras, and a caller who has not configured one still gets the heuristic.
tokenizer_provenance	Provenance	How the tokenizer claim was arrived at, under the same trust rules as every other capability. Only TRUSTED_PROVENANCE values let the gate treat a resulting floor as exact: a default guess about which tokenizer applies could produce a floor that is wrong in the direction that matters, and the gate refuses on the floor.

is_identity property

is_identity: bool

Whether this calibration leaves an estimate unchanged.

__post_init__

__post_init__() -> None

Reject calibrations that would corrupt every estimate downstream.

Raises:

Type	Description
ValueError	If the multiplier is not a positive finite number, or the overhead is negative.

anyinfer.TokenizerKind module-attribute

TokenizerKind = Literal[
    "tiktoken",
    "anthropic_count_tokens",
    "llama_server_tokenize",
]

Exact-counting strategies AnyInfer knows how to drive.

Each names a real endpoint or library, not a family of them: tiktoken is the published OpenAI-family vocabulary, anthropic_count_tokens is Anthropic's own counting endpoint, and llama_server_tokenize is a supervised llama-server's /tokenize. A provider whose tokenizer is neither published nor exposed declares nothing.

anyinfer.build_context_budget

build_context_budget(
    request: GenerationRequest,
    capabilities: ModelCapabilities | None,
    *,
    estimator: TokenEstimator | None = None,
    calibration: TokenCalibration | None = None,
    output_reserve_tokens: int | None = None,
    headroom_tokens: int | None = None,
) -> ContextBudget

Compute the context budget for one request against one model's capabilities.

Parameters:

Name	Type	Description	Default
request	GenerationRequest	The request to size.	required
capabilities	ModelCapabilities | None	Assembled capabilities supplying the context window and maximum output size. None means nothing is known — the budget stays tri-state.	required
estimator	TokenEstimator | None	Token counting strategy; defaults to the byte heuristic.	None
calibration	TokenCalibration | None	The target provider's declared envelope correction, from its descriptor. None means the provider counts what it was sent.	None
output_reserve_tokens	int | None	Overrides the derived output reserve.	None
headroom_tokens	int | None	Overrides the default clamped headroom.	None

Returns:

Type	Description
ContextBudget	The computed ContextBudget.

Raises:

Type	Description
ValueError	If an explicit reserve or headroom is negative.

anyinfer.estimate_request

estimate_request(
    request: GenerationRequest,
    *,
    estimator: TokenEstimator | None = None,
    calibration: TokenCalibration | None = None,
) -> RequestEstimate

Estimate the input tokens a request will occupy.

Derived from the typed request rather than hand-fed strings: messages (every content part, plus per-message framing overhead), offered tools, and the schema.

Parameters:

Name	Type	Description	Default
request	GenerationRequest	The request to size.	required
estimator	TokenEstimator | None	Token counting strategy; defaults to the byte heuristic.	None
calibration	TokenCalibration | None	The target provider's declared envelope correction. Applied to the planning figure only, and reported as its own component so the breakdown still adds up. None means the identity.	None

Returns:

Type	Description
RequestEstimate	The per-component estimate.

anyinfer.check_context_fit

check_context_fit(
    request: GenerationRequest,
    capabilities: ModelCapabilities | None,
    *,
    estimator: TokenEstimator | None = None,
    calibration: TokenCalibration | None = None,
    output_reserve_tokens: int | None = None,
    provider: str | None = None,
    model: str | None = None,
) -> ContextBudget

Build the budget for a request and raise if it provably cannot fit.

Parameters:

Name	Type	Description	Default
request	GenerationRequest	The GenerationRequest to size.	required
capabilities	ModelCapabilities | None	The target's assembled capabilities.	required
estimator	TokenEstimator | None	Token counting strategy; defaults to the byte heuristic.	None
calibration	TokenCalibration | None	The target provider's declared envelope correction. It never affects the gate's decision — the gate reads the floor, which no calibration moves — but it keeps the returned budget consistent with the one budget() reports.	None
output_reserve_tokens	int | None	Overrides the derived output reserve.	None
provider	str | None	Provider id, for the error's structured fields.	None
model	str | None	Model id, for the error message.	None

Returns:

Type	Description
ContextBudget	The computed ContextBudget when the
ContextBudget	request may proceed.

Raises:

Type	Description
ContextLengthError	When the estimate's floor exceeds a trusted-provenance context window.

anyinfer.ProbeReport dataclass

ProbeReport(
    target: ResolvedTarget,
    probes: tuple[FeatureProbe, ...] = (),
    capabilities: ModelCapabilities | None = None,
    requests: int = 0,
    usage: Usage = Usage(),
)

Everything one probing run learned, and what it cost.

Attributes:

Name	Type	Description
target	ResolvedTarget	The target that was probed.
probes	tuple[FeatureProbe, ...]	One result per feature tested, in the order tested.
capabilities	ModelCapabilities | None	What to record at probed provenance, or None when nothing was settled. Already merged with what was known, since a feature flag is one value and a probe that clears a bit must not clear the others with it.
requests	int	Round trips spent.
usage	Usage	Tokens spent, summed across the probes.

summary property

summary: str

One line naming what was settled.

outcome_for

outcome_for(feature: Feature) -> ProbeOutcome | None

What was established for one feature, or None if it was not tested.

anyinfer.FeatureProbe dataclass

FeatureProbe(
    feature: Feature,
    outcome: ProbeOutcome,
    detail: str = "",
)

The result of testing one feature against one target.

Attributes:

Name	Type	Description
feature	Feature	The feature that was tested.
outcome	ProbeOutcome	What the attempt established.
detail	str	One sentence explaining the outcome — the provider's rejection, or what came back instead of what was asked for.

conclusive property

conclusive: bool

Whether this probe settled anything worth recording.

anyinfer.ProbeOutcome module-attribute

ProbeOutcome = Literal[
    "supported", "unsupported", "inconclusive"
]

What one probe established, if anything.

anyinfer.PROBEABLE_FEATURES module-attribute

PROBEABLE_FEATURES: tuple[Feature, ...] = (
    Feature.JSON_SCHEMA,
    Feature.JSON_MODE,
    Feature.GRAMMAR,
    Feature.TOOLS,
    Feature.STREAMING,
)

Every feature a probe can settle.

Absent by design: REASONING (providers overwhelmingly accept the field and ignore it, so a probe would return inconclusive nearly always), SYSTEM_PROMPT and CACHE_USAGE (no single reply distinguishes honored from ignored), and the numeric bounds — finding a context window by bisection would cost dozens of requests to learn what one catalog entry already says.

anyinfer.DEFAULT_PROBE_FEATURES module-attribute

DEFAULT_PROBE_FEATURES: tuple[Feature, ...] = (
    Feature.JSON_SCHEMA,
    Feature.JSON_MODE,
    Feature.TOOLS,
    Feature.STREAMING,
)

What probe() tests when the caller names nothing: the four an OpenAI-compatible endpoint most often misreports. Four requests. GRAMMAR is excluded because the engines that have it declare it accurately, so paying a request to confirm buys nothing.

anyinfer.CostEstimate dataclass

CostEstimate(
    low: Decimal, high: Decimal, currency: str = "USD"
)

A preflight cost range for one request.

Deliberately a range, never one number: the input estimate is two-sided (anyinfer.capabilities.estimate) and the output spend is unknown until the model stops. Kept strictly separate from cost_usd, which is only ever computed from reported usage — estimated and actual money must never be indistinguishable.

Attributes:

Name	Type	Description
low	Decimal	Floor input tokens priced, with zero output — the least this can cost.
high	Decimal	Planning-estimate input plus the full output reserve priced — a spend ceiling under the budget's own assumptions.
currency	str	The pricing currency.

anyinfer.PricingTable

PricingTable(entries: dict[str, tuple[PricingEntry, ...]])

Per-provider model pricing with prefix-aware lookup.

providers property

providers: tuple[str, ...]

Provider ids the table covers, sorted.

entries_for

entries_for(provider_id: str) -> tuple[PricingEntry, ...]

Every entry for one provider, or an empty tuple.

lookup

lookup(
    provider_id: str, model: str
) -> Sourced[Pricing] | None

Find pricing for a model: exact match first, then longest boundary prefix.

Returns:

Type	Description
Sourced[Pricing] | None	The pricing tagged catalog provenance, or None when the model has no
Sourced[Pricing] | None	entry; never a fallback price.

from_mapping classmethod

from_mapping(data: Any) -> PricingTable

Build and validate a table from parsed JSON.

Raises:

Type	Description
ConfigError	On a malformed document — wrong format version, missing fields, or prices that do not parse as non-negative decimals.

anyinfer.load_default_pricing cached

load_default_pricing() -> PricingTable

Load the pricing table bundled with this release.

anyinfer.fetch_pricing

fetch_pricing(
    url: str = DEFAULT_PRICING_URL,
    *,
    timeout_s: float = 30.0,
    transport: Any | None = None,
) -> PricingTable

Fetch a maintained pricing table over HTTPS — the explicit freshness opt-in.

Nothing in the library calls this implicitly. An application that wants prices newer than its installed release calls it on its own schedule and passes the result to the client's pricing_table.

Parameters:

Name	Type	Description	Default
url	str	Where to fetch from; defaults to the repo's continuously-updated file.	DEFAULT_PRICING_URL
timeout_s	float	Request timeout.	30.0
transport	Any | None	Test seam — an httpx2 transport.	None

Returns:

Type	Description
PricingTable	The fetched, validated table.

Raises:

Type	Description
ConfigError	If the fetch fails or the response is not a valid pricing document.

Spend Accounting

An in-process rollup of what a client spent, and an optional ceiling checked before dispatch. Concepts: cost and spending.

anyinfer.SpendLedger

SpendLedger(currency: str = 'USD')

A thread-safe rollup of one client's observed spend.

Subscribe it like any other observer::

ledger = SpendLedger()
client = ai.Client(providers, observers=[ledger])
...
print(ledger.totals().cost)

Two clients that should share a total are given the same ledger. There is deliberately no process-wide instance: a global would make a total depend on import order, and would silently merge the accounting of two libraries that happen to share a process.

Only completed requests are counted. A failed attempt that a retry replaced is not separately visible in the event stream, so its tokens are not in these totals — the figure is "what the successful requests cost", not "everything the provider might bill". Where that distinction matters, compare against the provider's own invoice.

on_event

on_event(event: TelemetryEvent) -> None

Absorb one telemetry event.

Fast and non-blocking, as the observer contract requires: this is arithmetic under a lock, with no I/O.

record

record(
    target: ResolvedTarget,
    usage: Usage,
    *,
    request_id: str | None = None,
) -> None

Fold one completed request into the totals.

Parameters:

Name	Type	Description	Default
target	ResolvedTarget	What served the request.	required
usage	Usage	Its reported usage, with cost already computed by the core.	required
request_id	str | None	Correlation id, used to attribute the request to the labels its caller supplied.	None

Raises:

Type	Description
ValueError	If the usage carries a cost in a different currency than this ledger's. Converting would require a rate source, and a converted figure would have no provenance.

reserve

reserve(
    request_id: str,
    estimate: Decimal,
    ceiling: Decimal | None,
) -> tuple[bool, Decimal, Decimal]

Atomically reserve a preflight estimate against a cumulative ceiling.

Re-reserving the same request replaces its prior estimate, which lets fallback targets update the bound without double-counting one caller request.

release

release(request_id: str) -> None

Release a reservation that will not be replaced by a completion event.

reserved

reserved() -> Decimal

Total preflight spend currently reserved by in-flight requests.

totals

totals() -> SpendTotals

Everything observed so far.

by_target

by_target() -> Mapping[str, SpendTotals]

Totals per provider:model, in first-seen order.

by_label

by_label(key: str) -> Mapping[str, SpendTotals]

Totals per value of one caller-supplied metadata label.

The library never interprets these labels — a tenant id, a feature name, a job id are all the application's vocabulary, carried through untouched.

reset

reset() -> None

Forget everything recorded so far.

anyinfer.SpendTotals dataclass

SpendTotals(
    cost: Decimal = Decimal(0),
    currency: str = "USD",
    requests: int = 0,
    unknown_requests: int = 0,
    input_tokens: int = 0,
    output_tokens: int = 0,
    cache_read_tokens: int = 0,
)

What was spent, and what could not be priced.

unknown_requests is the honest counterpart to cost. A provider whose pricing is absent or untrusted produces no cost at all; never a zero, so a total that reported only cost would quietly understate spend by however many calls it could not price. Both numbers travel together for that reason, and every rendering of one should render the other.

Attributes:

Name	Type	Description
cost	Decimal	Summed cost of the requests that could be priced.
currency	str	Currency the costs are in. A ledger refuses to mix currencies rather than converting, because a conversion needs a rate source this library must not have.
requests	int	Completed requests observed.
unknown_requests	int	Completed requests whose cost could not be known.
input_tokens	int	Prompt tokens reported across those requests.
output_tokens	int	Generated tokens reported across those requests.
cache_read_tokens	int	Prompt tokens the providers reported serving from cache.

complete property

complete: bool

Whether every observed request could be priced.

plus

plus(usage: Usage) -> SpendTotals

Fold one request's usage into these totals.

to_json

to_json() -> dict[str, Any]

Serialize for a SpendStore.

from_json classmethod

from_json(data: Mapping[str, Any]) -> SpendTotals | None

Deserialize, returning None for anything unreadable.

anyinfer.SpendStore

SpendStore(path: Path | str)

An optional, caller-owned file of accumulated spend.

The library persists nothing on its own. An application that wants a total that survives a restart constructs one of these and points it somewhere — the same contract anyinfer.benchmark.MeasurementStore uses, including its most important property: reads are total. A missing, truncated, or foreign file yields nothing rather than raising, because a cache that can break a program is worse than no cache.

path property

path: Path

Where this store reads and writes.

load

load() -> Mapping[str, SpendTotals]

Every stored bucket, keyed by name. Unreadable content yields nothing.

accumulate

accumulate(
    ledger: SpendLedger, *, bucket: str = "total"
) -> None

Add a ledger's current totals to the stored bucket, atomically.

Parameters:

Name	Type	Description	Default
ledger	SpendLedger	The ledger whose totals to fold in.	required
bucket	str	Which stored bucket to add to — a process name, a job id, or the default single bucket.	'total'

Raises:

Type	Description
ValueError	If the ledger's currency differs from the stored bucket's.

anyinfer.SpendPolicy dataclass

SpendPolicy(
    max_total_usd: Decimal | None = None,
    max_request_usd: Decimal | None = None,
    on_unknown: Literal["allow", "refuse"] = "allow",
)

A ceiling on what one client may spend. Off unless supplied.

Checked before dispatch, beside the context gate, so a refusal costs nothing. This is the caller's own policy on their own client — it shares no state with any other process and enforces no organization quota, which this library deliberately leaves to the deployment around it.

Attributes:

Name	Type	Description
max_total_usd	Decimal | None	Ceiling on this client's cumulative spend. None means no ceiling.
max_request_usd	Decimal | None	Ceiling on any single request's estimated cost.
on_unknown	Literal['allow', 'refuse']	What to do when a target's cost cannot be estimated, because its pricing is missing or untrusted. allow preserves today's behaviour; refuse is for callers who would rather fail than spend blind. There is no third option that treats unknown as zero — a guard that does that enforces nothing while appearing to.

active property

active: bool

Whether this policy can refuse anything.

__post_init__

__post_init__() -> None

Reject a ceiling that cannot be enforced.

Raises:

Type	Description
ValueError	On a negative ceiling or an unknown on_unknown value.

Rate Governance

Client-side pacing for one provider instance, and the header dialect a provider reports its window in. Both are inert until configured. Concepts: routing and rate limits.

anyinfer.RateLimits dataclass

RateLimits(
    max_concurrent: int | None = None,
    requests_per_minute: float | None = None,
    min_interval_s: float = 0.0,
    respect_headers: bool = True,
    reserve_fraction: float = 0.0,
)

Client-side pacing for one provider instance. Inert unless configured.

This paces this process's own requests to one provider so an application that fans out does not discover the provider's limits by being throttled by them. It shares no state with any other process, enforces no quota the provider did not state, and never influences which target is chosen — a limiter that picked a different provider because this one was busy would be load balancing, which this library deliberately does not do.

With every field left at its default, a request is dispatched exactly as it was before this existed: no permit, no delay, no bookkeeping.

Attributes:

Name	Type	Description
max_concurrent	int | None	Most requests in flight at once for this instance. None means unbounded, which is today's behaviour.
requests_per_minute	float | None	Sustained request rate. Enforced as a token bucket, so a burst up to the per-minute allowance is permitted and then paced.
min_interval_s	float	Smallest gap between two dispatches, for providers that object to bursts regardless of the rate.
respect_headers	bool	Whether to slow down when the provider's own rate-limit headers say its window is nearly exhausted. Inert when the provider declares no header dialect, since there is nothing to read.
reserve_fraction	float	Fraction of the provider's stated remaining allowance to leave untouched, between 0 and 1. Matters whenever this process is not the only consumer of the key: stopping at the last request in the window means the other consumer is the one that gets throttled.

active property

active: bool

Whether this policy can delay anything.

A bare RateLimits() is active: it means "pace me by what the provider reports", which is the least a caller who asked for governance at all can mean. Opting out is spelled by supplying no limits, not by supplying empty ones — so RateLimits(respect_headers=False) with no bounds is the one inert instance, and it is inert honestly rather than by accident.

__post_init__

__post_init__() -> None

Reject a limit that cannot be honoured.

Raises:

Type	Description
ValueError	On a non-positive bound, a negative interval, or a reserve fraction outside the unit interval.

anyinfer.RateLimitHeaders dataclass

RateLimitHeaders(
    requests_remaining: str = "",
    requests_reset: str = "",
    tokens_remaining: str = "",
    tokens_reset: str = "",
    limit_requests: str = "",
    limit_tokens: str = "",
)

Which response headers a provider reports its rate-limit state in.

Header names are wire facts and differ per provider, so they are declared on the descriptor and recorded in that provider's contract snapshot; never branched on by provider id in the core.

A provider whose dialect cannot be verified from its documentation declares nothing. An empty dialect is not a failure: pacing falls back to whatever bounds the caller configured, which is a smaller promise honestly kept rather than a guessed header name that silently reads None forever.

Attributes:

Name	Type	Description
requests_remaining	str	Requests left in the current window.
requests_reset	str	When the request window resets. Read as seconds, or as a duration like "1m30s" for the providers that spell it that way.
tokens_remaining	str	Tokens left in the current window.
tokens_reset	str	When the token window resets, in the same two spellings.
limit_requests	str	The window's full request allowance, when the provider states it. Only needed to turn reserve_fraction into an absolute floor.
limit_tokens	str	The window's full token allowance, for the same reason.

declared property

declared: bool

Whether this provider reports anything worth reading.