Clients and Streams

The two entry points, Client (sync) and AsyncClient (async), expose the same surface; the sync client is a facade over the async core (see the architecture overview).

anyinfer.Client

Client(
    providers: Sequence[ProviderSettings] | None = None,
    *,
    registry: ProviderRegistry | None = None,
    catalog: Catalog | None = None,
    route: Route | None = None,
    operation_routes: Mapping[str, Route] | None = None,
    observers: Sequence[Observer] | None = None,
    resolver: ResolverChain | None = None,
    retain_raw: bool = False,
    repair: Repair | None = None,
    use_default_catalog: bool = True,
    estimator: TokenEstimator | None = None,
    context_gate: bool = True,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    arenas: Mapping[str, ArenaPolicy] | None = None,
    spend: SpendPolicy | None = None,
    ledger: SpendLedger | None = None,
    pricing_table: PricingTable | None = None,
    manifests: bool = True,
    manifest_payloads: bool = False,
    capability_overrides: Mapping[str, ModelCapabilities]
    | None = None,
    model_dir: Path | None = None,
)

The synchronous inference client.

A thin facade: every method schedules work on one background event loop that owns the real AsyncClient. Safe to call from multiple threads, and concurrent requests still overlap on the loop.

Args are identical to AsyncClient.

model_store property

model_store: ModelStore

The store acquired model weights live in.

__enter__

__enter__() -> Client

Enter a context that closes the client on exit.

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the client on context exit.

close

close() -> None

Close adapters, stop the background loop, and join its thread.

subscribe

subscribe(
    observer: Observer, *, payloads: bool = False
) -> None

Register a telemetry observer.

unsubscribe

unsubscribe(observer: Observer) -> None

Remove a telemetry observer.

models

models(
    provider_id: str,
    *,
    operation: InferenceOperation | None = None,
) -> Sequence[DiscoveredModel]

List a provider's models. See AsyncClient.models.

health

health(provider_id: str) -> Health

Probe a provider's readiness.

diagnostics

diagnostics(provider_id: str) -> Sequence[Diagnostic]

Ask a provider what it has noticed about its own runtime.

See AsyncClient.diagnostics.

resolve

resolve(target: Target) -> ResolvedTarget

Resolve a target string without issuing a request.

pull_model

pull_model(
    provider_id: str,
    model: str,
    *,
    progress: Any | None = None,
    timeout_s: float = PULL_TIMEOUT_S,
) -> PullReport

Tell an engine that keeps its own store to make a model available.

See AsyncClient.pull_model.

session

session(target: Target) -> Session

Open a handle that lets a provider keep what it already knows.

See AsyncClient.session.

benchmark

benchmark(
    target: Target,
    *,
    prompt_tokens: int = BENCHMARK_PROMPT_TOKENS,
    output_tokens: int = BENCHMARK_OUTPUT_TOKENS,
    timeout_s: float = 120.0,
    store: MeasurementStore | None = None,
    progress: Callable[[BenchmarkSample], None]
    | None = None,
) -> Measurement

Measure what a target actually does, with one deterministic request.

See AsyncClient.benchmark.

probe

probe(
    target: Target,
    *,
    features: Sequence[Feature] | None = None,
    timeout_s: float = 30.0,
    record: bool = True,
) -> ProbeReport

Measure what a target actually supports, one request per feature.

See AsyncClient.probe.

verify

verify(
    target: Target,
    *,
    timeout_s: float = 60.0,
    operation: InferenceOperation = "generation",
) -> Verification

Prove a target works by asking it something, end to end.

See AsyncClient.verify.

probe_embedding

probe_embedding(
    target: Target,
    *,
    timeout_s: float = 30.0,
    record: bool = True,
) -> EmbeddingProbeReport

Measure an embedding target with one real call. See AsyncClient.probe_embedding.

local_catalog

local_catalog(
    provider_id: str | None = None,
    *,
    hardware: HardwareProfile | None = None,
    best_at: str | None = None,
    posture: Posture = "balanced",
) -> CatalogView

Browse the local model catalog, annotated with how each entry fits.

See AsyncClient.local_catalog.

acquire_model

acquire_model(
    model_id: str,
    *,
    engine: str | None = None,
    variant_id: str | None = None,
    hardware: HardwareProfile | None = None,
    progress: ProgressSink | None = None,
    prefs: VariantPrefs | None = None,
    dry_run: bool = False,
    token: str | None = None,
) -> AcquisitionReport

Download a catalog model's weights. See AsyncClient.acquire_model.

The progress sink is invoked from the background loop thread, so it must not block and must not call back into this client.

installed_models

installed_models() -> Sequence[StoreEntry]

Every model acquired into this client's store.

locate_model

locate_model(
    model_id: str,
    *,
    variant_id: str | None = None,
    engine: str | None = None,
    verify: bool = False,
) -> ResolvedModel | None

Find an acquired model on disk. See AsyncClient.locate_model.

remove_model

remove_model(entry_id: str) -> RemovalReport

Delete an acquired model. See AsyncClient.remove_model.

spend

spend() -> SpendTotals

What this client has spent so far. See AsyncClient.spend.

budget

budget(
    messages: MessagesInput,
    *,
    target: Target,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    sampling: Sampling | None = None,
    output_reserve_tokens: int | None = None,
) -> ContextBudget

Compute the context budget for a request without sending it.

Pure computation — runs directly on the calling thread. See AsyncClient.budget().

compare

compare(
    messages: MessagesInput | GenerationRequest,
    *,
    targets: Sequence[Target],
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    refresh: bool = False,
) -> tuple[TargetComparison, ...]

Compare request portability without generating. See AsyncClient.compare.

compare_embedding

compare_embedding(
    inputs: str | Sequence[str],
    *,
    targets: Sequence[Target],
    input_type: EmbeddingInputIntent | None = None,
    refresh: bool = False,
) -> tuple[EmbeddingTargetComparison, ...]

Compare embedding request portability without dispatching.

See AsyncClient.compare_embedding.

generate

generate(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    logprobs: int | None = None,
    cite_documents: bool = False,
    server_tools: Sequence[ServerToolSpec] = (),
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    max_input_part_bytes: int | None = None,
    max_input_bytes: int | None = None,
    session: Session | None = None,
    manifest: bool | None = None,
) -> Generation

Generate a single result. See AsyncClient.generate().

submit_batch

submit_batch(
    batch: BatchGenerationRequest, *, target: Target
) -> BatchHandle

Submit a batch. See AsyncClient.submit_batch().

batch_status

batch_status(handle: BatchHandle) -> BatchReport

Ask where a batch is. See AsyncClient.batch_status().

fetch_batch

fetch_batch(
    handle: BatchHandle, *, schema: SchemaSpec | None = None
) -> BatchResult

Download a finished batch. See AsyncClient.fetch_batch().

cancel_batch

cancel_batch(handle: BatchHandle) -> BatchReport

Cancel a batch. See AsyncClient.cancel_batch().

embed

embed(
    inputs: str | Sequence[str],
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    input_type: Literal[
        "query", "document", "classification", "clustering"
    ]
    | None = None,
    dimensions: int | None = None,
    expected_space: EmbeddingSpace | None = None,
    allow_incompatible_fallback: bool = False,
    batch: BatchPolicy | None = None,
    timeout_s: float | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    retain_raw: bool | None = None,
    manifest: bool | None = None,
) -> EmbeddingResult

Embed one or more texts into vectors. See AsyncClient.embed().

rerank

rerank(
    query: str,
    documents: Sequence[str | RerankDocument],
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    top_n: int | None = None,
    batch: BatchPolicy | None = None,
    timeout_s: float | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    return_documents: bool = False,
    retain_raw: bool | None = None,
    manifest: bool | None = None,
) -> RerankResult

Rank documents by relevance to a query. See AsyncClient.rerank().

run_tools

run_tools(
    messages: MessagesInput,
    *,
    tools: Sequence[Any],
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    max_rounds: int = DEFAULT_MAX_ROUNDS,
    **kwargs: Any,
) -> Generation

Generate, dispatching tools until the model answers.

See AsyncClient.run_tools().

stream

stream(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    logprobs: int | None = None,
    cite_documents: bool = False,
    server_tools: Sequence[ServerToolSpec] = (),
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    max_input_part_bytes: int | None = None,
    max_input_bytes: int | None = None,
    session: Session | None = None,
    manifest: bool | None = None,
) -> SyncStream

Start a streaming generation, returning a blocking iterator.

See AsyncClient.stream(). Use the result as a context manager so that leaving the block early cancels the in-flight request.

anyinfer.AsyncClient

AsyncClient(
    providers: Sequence[ProviderSettings] | None = None,
    *,
    registry: ProviderRegistry | None = None,
    catalog: Catalog | None = None,
    route: Route | None = None,
    operation_routes: Mapping[str, Route] | None = None,
    observers: Sequence[Observer] | None = None,
    resolver: ResolverChain | None = None,
    retain_raw: bool = False,
    repair: Repair | None = None,
    use_default_catalog: bool = True,
    estimator: TokenEstimator | None = None,
    context_gate: bool = True,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    arenas: Mapping[str, ArenaPolicy] | None = None,
    spend: SpendPolicy | None = None,
    ledger: SpendLedger | None = None,
    pricing_table: PricingTable | None = None,
    manifests: bool = True,
    manifest_payloads: bool = False,
    capability_overrides: Mapping[str, ModelCapabilities]
    | None = None,
    model_dir: Path | None = None,
    credential_ttl_s: float | None = None,
    limiters: Mapping[str, RateLimiter] | None = None,
)

Bases: GenerationExecutionMixin, ArenaExecutionMixin, SpendGovernanceMixin

The asynchronous inference client.

Parameters:

Name	Type	Description	Default
providers	Sequence[ProviderSettings] | None	Per-provider settings. The order given is the preference order used when resolving catalog aliases.	None
registry	ProviderRegistry | None	Provider registry; defaults to the process-wide one.	None
catalog	Catalog | None	Alias catalog; defaults to the bundled catalog. Pass None explicitly via use_default_catalog=False to disable alias resolution.	None
route	Route | None	Default route applied when a call names no target.	None
observers	Sequence[Observer] | None	Telemetry observers, registered payload-free.	None
resolver	ResolverChain | None	Credential resolver chain.	None
retain_raw	bool	Keep the provider's raw payload on results. Off by default because raw payloads carry response text that payload-free telemetry deliberately omits.	False
repair	Repair | None	Default repair budget for schema violations.	None
use_default_catalog	bool	Load the bundled catalog when catalog is not supplied.	True
estimator	TokenEstimator | None	Token counting strategy for budgets and the pre-dispatch gate. Defaults to the dependency-free byte heuristic.	None
context_gate	bool	Fail a target before dispatch when the request provably cannot fit its known context window. Only trusted-provenance windows gate, and only on the estimate's floor, so a heuristic never refuses a request that might have fit.	True
history	HistoryPolicy | None	Conversation-compaction policy applied when a request outgrows its target's window. None — the default; never compacts. This is the client's half of the overflow answer; Route.context_window_targets is the other half, and the policy's mode decides which is tried first.	None
spend	SpendPolicy | None	Ceiling on what this client may spend, checked before dispatch. None — the default; never refuses anything. Not an organization quota: it governs this client object in this process and nothing else.	None
ledger	SpendLedger | None	Spend rollup to record into. One is created automatically when spend is set; supply your own to share a total between clients or to read it without a policy in force.	None
cache	CachePolicy | None	Prompt-cache placement applied to every request that does not carry its own. None — the default; never engages a provider's cache, because caching changes what a provider bills and how long it keeps a copy of the prompt. A request's own cache overrides this. Every frontend built on this client inherits it.	None
pricing_table	PricingTable | None	Model pricing supplying the catalog layer of capability assembly. Defaults to the table bundled with this release; pass the result of fetch_pricing() for newer numbers.	None
manifests	bool	Assemble a RunManifest for every call, reachable as Generation.manifest. On by default: it allocates one small object per in-flight request, writes nothing, sends nothing, and is content-free — the invited/uninvited line in this library has always been about spend and side effects, and a manifest has neither. Switch it off to skip the allocation entirely.	True
manifest_payloads	bool	Capture prompt, response, schema, and tool-call text into the manifest's payloads facet, redacted. Off by default and independent of observer payload opt-in, so a manifest cannot start carrying prompt text because some unrelated telemetry sink asked for it.	False
capability_overrides	Mapping[str, ModelCapabilities] | None	Deliberate corrections keyed by "provider:model". Every supplied field is applied at override provenance — the strongest layer, outranking discovery and probes, so a wrong upstream number can always be fixed locally.	None
model_dir	Path | None	Where acquired model weights are stored. Defaults to the per-OS data directory, overridable with ANYINFER_MODEL_DIR.	None
credential_ttl_s	float | None	How often to re-check whether a provider's credential reference still resolves to the same secret. None — the default — resolves once at adapter build and never again, which is what a client holding literal keys wants. Set it when credentials come from a source that rotates underneath a long-running process (a keychain, a mounted secret, an env file a deployment rewrites): on expiry the reference is re-resolved, and the adapter is rebuilt only if the value actually changed, so a rotation costs one connection pool and a stable credential costs nothing. A provider that rejects a credential which had been working triggers the same re-resolution immediately, whatever the TTL says.	None
limiters	Mapping[str, RateLimiter] | None	Pre-built RateLimiter instances keyed by provider instance id, used instead of constructing one from that instance's limits. For the one situation the constructed path cannot serve: a caller that builds a short-lived client per request around a long-lived credential, where a per-client limiter paces every call against an empty bucket and discards the windows the provider just reported. Limiter identity is the unit of pacing — one account at one provider — so handing in the limiter that identity owns is what carries the state across clients. Injection is deliberately not a cross-process quota mechanism: these are in-memory objects in one loop.	None

catalog property

catalog: Catalog | None

The alias catalog in force, if any.

model_store property

model_store: ModelStore

The store acquired model weights live in.

__aenter__ async

__aenter__() -> AsyncClient

Enter an async context managing this client's adapters.

__aexit__ async

__aexit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the client on context exit.

aclose async

aclose() -> None

Close every adapter this client built.

subscribe

subscribe(
    observer: Observer, *, payloads: bool = False
) -> None

Register a telemetry observer.

Parameters:

Name	Type	Description	Default
observer	Observer	The sink.	required
payloads	bool	Opt in to prompt and response text. Off by default, so no observer sees payload text it did not explicitly ask for.	False

unsubscribe

unsubscribe(observer: Observer) -> None

Remove a telemetry observer.

models async

models(
    provider_id: str,
    *,
    operation: InferenceOperation | None = None,
) -> Sequence[DiscoveredModel]

List a provider's models, recording what they report about capabilities.

Parameters:

Name	Type	Description	Default
provider_id	str	The configured provider to list.	required
operation	InferenceOperation | None	Keep only models known to serve this operation — via a discovered operation tag, or the descriptor's static embedding/rerank capability tables. A model whose operations are unknown is included only for "generation" on a generation-capable provider (the pre-filter behaviour); for embedding and rerank, unknown support is never guessed into the listing. None lists everything.	None

operations_for

operations_for(
    target: Target,
) -> frozenset[InferenceOperation]

Which inference operations the resolved target is known to serve.

Model-level facts win: a discovered operation tag, else membership in the descriptor's static embedding/rerank capability tables. A model with no model-level facts on a generation-capable provider reports {"generation"} — the assumption every listing made before operations existed — and never has embedding or rerank support guessed in.

Parameters:

Name	Type	Description	Default
target	Target	A target string or catalog alias; resolved without dispatching.	required

Raises:

Type	Description
ConfigError	If the target cannot be resolved at all.

health async

health(provider_id: str) -> Health

Probe a provider's readiness.

diagnostics async

diagnostics(provider_id: str) -> Sequence[Diagnostic]

Ask a provider what it has noticed about its own runtime.

Answers the question a health probe cannot: not "can I reach it" but "is it in good shape" — a model that spilled out of VRAM, a runtime that fell back to the CPU, a supervised server nearing its memory ceiling. Requests to such a provider succeed; they are simply much slower than the caller expects, with nothing in the result to explain why.

Providers that declare reports_diagnostics answer; the rest return nothing, as does one that fails to answer — this is advisory data and never raises.

Parameters:

Name	Type	Description	Default
provider_id	str	The configured provider to ask.	required

Returns:

Type	Description
Sequence[Diagnostic]	What the provider reported, most likely empty.

resolve

resolve(target: Target) -> ResolvedTarget

Resolve a target string without issuing a request.

session

session(target: Target) -> Session

Open a handle that lets a provider keep what it already knows.

Every request is independent by default, which is right for one-shot work and wrong for a conversation. Providers that can carry state between turns each save something different — Copilot keeps the conversation server-side, llama.cpp keeps the model and its KV cache resident, Ollama keeps the model loaded, and a session is how a caller says "these requests belong together" without having to know which.

A session never changes an answer; it is a performance and cost optimization. Opening one against a provider that cannot keep state is therefore allowed and merely inert: every request behaves exactly as it would have, and Session.supported and Session.reuse say so.

with client.session("copilot:auto") as chat:
    await client.generate("Summarize this report.", session=chat)
    await client.generate("Now list the risks.", session=chat)

Parameters:

Name	Type	Description	Default
target	Target	The target this session's state belongs to. State is not portable, so a turn routed anywhere else runs without it.	required

Returns:

Type	Description
Session	The Session handle.

Raises:

Type	Description
ConfigError	If the target cannot be resolved.

verify async

verify(
    target: Target,
    *,
    timeout_s: float = 60.0,
    operation: InferenceOperation = "generation",
) -> Verification

Prove a target works by asking it something, end to end.

health() answers "can I reach this endpoint", which is not the question behind a Test connection button: a credential can be valid for a model listing and not for inference, a model id can be a typo, a deployment can exist with no capacity, and a provider can answer fluently while never holding a schema. Only a real request distinguishes those, so this spends one — deliberately tiny, capped at VERIFY_MAX_OUTPUT_TOKENS output tokens, or VERIFY_REASONING_OUTPUT_TOKENS when the target is known to be a reasoning model and would otherwise spend the whole budget thinking before it said anything.

Never raises for a provider problem: "this target is broken" is the answer to the question, not a failure to answer it. A malformed target, on the other hand, is the caller's mistake and still raises.

Parameters:

Name	Type	Description	Default
target	Target	The target to verify. A catalog alias resolves as usual.	required
timeout_s	float	Wall clock for the probe.	60.0
operation	InferenceOperation	Which operation to prove. "embedding" embeds one tiny probe text and judges the vector; "rerank" ranks two probe documents. Both spend one deliberately small real request, exactly like the generation probe.	'generation'

Returns:

Type	Description
Verification	The Verification, whose reached
Verification	and ok distinguish "unreachable" from "reachable but could not hold the
Verification	shape".

Raises:

Type	Description
ConfigError	If the target cannot be resolved at all.

probe_embedding async

probe_embedding(
    target: Target,
    *,
    timeout_s: float = 30.0,
    record: bool = True,
) -> EmbeddingProbeReport

Measure an embedding target with one tiny real call.

Embedding capability tables only carry what a provider documents; a self-hosted or preset endpoint often documents nothing. This spends one deliberately small request and measures what came back — the vector length, and whether it was unit-normalized — recording both at probed provenance so later calls (and capabilities_for consumers) see measured facts instead of blanks.

This costs money and time, exactly like probe(): opt-in, one round trip.

Parameters:

Name	Type	Description	Default
target	Target	The embedding target to measure.	required
timeout_s	float	Wall clock for the probe call.	30.0
record	bool	Store the findings at probed provenance.	True

Returns:

Type	Description
EmbeddingProbeReport	The EmbeddingProbeReport with the measured facts.

Raises:

Type	Description
ConfigError	If the target cannot be resolved, or its provider does not declare the embedding operation.
AllTargetsFailedError	If the probe call itself failed.

probe async

probe(
    target: Target,
    *,
    features: Sequence[Feature] | None = None,
    timeout_s: float = 30.0,
    record: bool = True,
) -> ProbeReport

Measure what a target actually supports, one tiny request per feature.

The capability layer's third tier, and the only one that is a measurement. The catalog says what a model should support and discovery says what a provider claims; for the compatibility surface — every preset endpoint, every self-hosted OpenAI-compatible server — both are educated guesses, and a server that accepts response_format while ignoring it is indistinguishable from one that honors it until a schema silently stops being enforced.

This costs money and time: one round trip per feature, four by default. It is opt-in for that reason, and normally run once when an application first configures an endpoint rather than on every start.

A probe that settles nothing records nothing. A provider that accepts the request and answers something unexpected is inconclusive, because a weak model and an ignored parameter look identical in one reply.

Parameters:

Name	Type	Description	Default
target	Target	The target to measure.	required
features	Sequence[Feature] | None	Which features to test; defaults to DEFAULT_PROBE_FEATURES.	None
timeout_s	float	Wall clock for each individual probe.	30.0
record	bool	Store the findings at probed provenance, so later requests choose mechanisms from measurement rather than assumption. Pass False to look without committing.	True

Returns:

Type	Description
ProbeReport	The ProbeReport: per-feature outcomes,
ProbeReport	what was recorded, and what it cost.

Raises:

Type	Description
ConfigError	If the target cannot be resolved, or a feature was named that no probe can settle.

benchmark async

benchmark(
    target: Target,
    *,
    prompt_tokens: int = BENCHMARK_PROMPT_TOKENS,
    output_tokens: int = BENCHMARK_OUTPUT_TOKENS,
    timeout_s: float = 120.0,
    store: MeasurementStore | None = None,
    progress: Callable[[BenchmarkSample], None]
    | None = None,
) -> Measurement

Measure what a target actually does, with one deterministic request.

Capabilities describe a model; none of them says how fast it is here. For local inference that is the number that decides everything — the same weights on the same GPU differ by an order of magnitude depending on what else is resident and how many layers ended up offloaded, and it is the number an application needs to pick a default model or explain a slow session.

Prefill and decode are reported separately, because a machine can be fast at one and slow at the other, and prefill throughput is reported only when the provider timed its own prefill phase. Deriving it from time-to-first-token would fold queueing and network latency into a figure labelled compute.

This costs one real request of roughly prompt_tokens in and output_tokens out. Nothing is written anywhere unless store is passed.

Parameters:

Name	Type	Description	Default
target	Target	The target to measure.	required
prompt_tokens	int	Approximate prompt size. Large enough that prefill is a real phase rather than rounding error.	BENCHMARK_PROMPT_TOKENS
output_tokens	int	Output cap. Decode throughput needs enough tokens to average over.	BENCHMARK_OUTPUT_TOKENS
timeout_s	float	Wall clock for the request.	120.0
store	MeasurementStore | None	An application-owned store to record the result in. Omitted, the measurement is returned and forgotten.	None
progress	Callable[[BenchmarkSample], None] | None	Optional sink for live token-rate and local host-utilization samples. Token counts are estimated until the terminal provider usage arrives.	None

Returns:

Type	Description
Measurement	The Measurement, whose rates are
Measurement	None where nothing could be measured rather than zero.

Raises:

Type	Description
ConfigError	If the target cannot be resolved.
AllTargetsFailedError	If the request itself failed — an unmeasurable target is a failure, unlike an unverifiable one.

local_catalog async

local_catalog(
    provider_id: str | None = None,
    *,
    hardware: HardwareProfile | None = None,
    best_at: str | None = None,
    posture: Posture = "balanced",
) -> CatalogView

Browse the local model catalog, annotated with how each entry fits.

Performs no network I/O: the catalog is bundled data and hardware detection is local and cached.

Parameters:

Name	Type	Description	Default
provider_id	str | None	Restrict to models one configured engine can serve. Its locality also decides whether this machine is the right one to probe.	None
hardware	HardwareProfile | None	Specs to judge against, overriding detection. This is how an application answers for a remote host after asking the user.	None
best_at	str | None	One category from the catalog's closed vocabulary.	None
posture	Posture	How much of the machine to budget.	'balanced'

Returns:

Type	Description
CatalogView	The view. hardware_source == "unavailable" means the engine runs somewhere
CatalogView	AnyInfer cannot probe, and the application should collect the host's specs and
CatalogView	call again.

Raises:

Type	Description
ConfigError	If best_at names a category no catalog entry uses.

acquire_model async

acquire_model(
    model_id: str,
    *,
    engine: str | None = None,
    variant_id: str | None = None,
    hardware: HardwareProfile | None = None,
    progress: ProgressSink | None = None,
    prefs: VariantPrefs | None = None,
    dry_run: bool = False,
    token: str | None = None,
) -> AcquisitionReport

Download a catalog model's weights into this client's model store.

The quantization is chosen, not assumed: the highest curated rung whose weights and KV cache fit this machine's budget. Pass variant_id to override that.

dry_run=True resolves everything and reports the exact byte count without writing anything — what an application needs to confirm a large download with a user before starting it.

Parameters:

Name	Type	Description	Default
model_id	str	A catalog model id.	required
engine	str | None	"llama.cpp" or "vllm"; defaults to whatever the model offers.	None
variant_id	str | None	Acquire this exact variant, skipping selection.	None
hardware	HardwareProfile | None	Specs to select against, overriding detection.	None
progress	ProgressSink | None	Aggregate progress sink. See AcquisitionProgress for the threading contract it must honor.	None
prefs	VariantPrefs | None	Selection preferences, including the low-quality opt-in.	None
dry_run	bool	Plan and report without writing.	False
token	str | None	A credential for the source, when it needs one. Defaults to HF_TOKEN.	None

Returns:

Type	Description
AcquisitionReport	The report, naming the registered entry and what it cost.

Raises:

Type	Description
ConfigError	If there is no catalog, the model is unknown, or nothing fits.
LocalRuntimeError	On a transfer failure, digest mismatch, or full disk.

pull_model async

pull_model(
    provider_id: str,
    model: str,
    *,
    progress: Callable[[TelemetryEvent], None]
    | None = None,
    timeout_s: float = PULL_TIMEOUT_S,
) -> PullReport

Tell an engine that keeps its own store to make a model available.

Distinct from acquire_model(), which fetches weights this library places and indexes. Some local engines — Ollama — already have a store, a registry, and a downloader; for those the useful operation is not "download these bytes" but "make yourself ready", and the bytes land in the engine's store under the engine's own name. Nothing is written to this client's model store and locate_model() will not find it, because it is not ours to find.

Progress arrives as DownloadProgress events on the client's observers, and additionally on progress when one is given.

Parameters:

Name	Type	Description	Default
provider_id	str	The configured provider to pull on.	required
model	str	The model name in that engine's namespace, e.g. "qwen3:8b".	required
progress	Callable[[TelemetryEvent], None] | None	An extra sink for progress events, for a caller that wants them without registering an observer.	None
timeout_s	float	Wall clock for the whole transfer. Generous by default: a timeout that fires mid-download turns a slow link into a failure the user cannot act on.	PULL_TIMEOUT_S

Returns:

Type	Description
PullReport	The PullReport, which distinguishes a
PullReport	transfer from a model that was already present.

Raises:

Type	Description
ConfigError	If the provider is unknown, or cannot pull.
ModelNotFoundError	If the engine's registry has no such model.
LocalRuntimeError	If the engine is unreachable or the pull fails.

installed_models async

installed_models() -> Sequence[StoreEntry]

Every model acquired into this client's store.

locate_model async

locate_model(
    model_id: str,
    *,
    variant_id: str | None = None,
    engine: str | None = None,
    verify: bool = False,
) -> ResolvedModel | None

Find an acquired model on disk, with advisory launch arguments.

No network I/O. Verification is shallow by default — size and modification time against the index, because re-hashing forty gigabytes on every lookup would be absurd; verify=True forces the full check.

remove_model async

remove_model(entry_id: str) -> RemovalReport

Delete an acquired model's files and unregister it.

A model adopted from somebody else's cache is only unregistered; its files belong to whatever put them there.

generate async

generate(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    logprobs: int | None = None,
    cite_documents: bool = False,
    server_tools: Sequence[ServerToolSpec] = (),
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    max_input_part_bytes: int | None = None,
    max_input_bytes: int | None = None,
    session: Session | None = None,
    manifest: bool | None = None,
) -> Generation

Generate a single result, draining the event stream internally.

manifest overrides the client's manifest setting for this one call; None inherits it.

logprobs asks the target to report per-token log-probabilities: 0 for the chosen token's own, a positive count for that many alternatives beside it, and None — the default — for none at all. A target that cannot report them emits a ParameterDropped event rather than answering with an empty Generation.logprobs.

cite_documents asks the target to attribute its answer to the documents this request supplied, landing them on Generation.citations and on CitationDelta events as they arrive. Off by default and never inferred from the presence of a document, since every dialect treats it as a request-side opt-in and several bill a cited answer differently.

server_tools asks the provider to run capabilities of its own during the generation — web search, code execution. Off by default and never inferred: each is billed per invocation. A target that cannot run one refuses before dispatch rather than answering as though it had, since an answer built without the search that was asked for is a different answer, not a degraded one.

Returns:

Type	Description
Generation	The assembled Generation.

Raises:

Type	Description
AllTargetsFailedError	Every target failed.
SchemaViolationError	The response never satisfied the schema.

submit_batch async

submit_batch(
    batch: BatchGenerationRequest, *, target: Target
) -> BatchHandle

Submit a batch for deferred, discounted execution.

Every major provider sells this tier at roughly half price on a delayed window, which is the shape of an eval, a backfill, or an offline enrichment run. Each line is translated through the same wire builder a live call uses, so a batched request carries the schema, tools, cache marks, and reasoning effort its live twin would.

Nothing is stored here. The returned handle is the caller's to persist, wherever they already persist their own work. Run retention is a stated non-goal, and a job answered hours later in another process is exactly where it would be most tempting to break it.

Parameters:

Name	Type	Description	Default
batch	BatchGenerationRequest	The requests to run together.	required
target	Target	Where to run them. One target for the whole batch: a provider's batch endpoint takes one model, and splitting across targets would be routing, which a deferred job cannot do — there is no failure to fall back from until hours later.	required

Returns:

Type	Description
BatchHandle	The handle that reclaims this batch.

Raises:

Type	Description
ConfigError	The provider does not implement batching.
ProviderError	The provider refused the submission.

batch_status async

batch_status(handle: BatchHandle) -> BatchReport

Ask where a batch is, without downloading its results.

Polling is cheap and fetching is not — providers charge nothing to ask about a job and real bandwidth to download one — so a caller waiting on a 24-hour window asks many times and fetches once.

Raises:

Type	Description
ConfigError	The provider does not implement batching.

fetch_batch async

fetch_batch(
    handle: BatchHandle,
    *,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
) -> BatchResult

Download a finished batch's lines, in submission order.

Parameters:

Name	Type	Description	Default
handle	BatchHandle	The batch to collect.	required
schema	SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None	The structured-output contract every line was submitted under, applied to each answer here. Supplying it is what keeps a batch from being the one place this library stops enforcing schemas — the reason to batch through AnyInfer rather than around it is that the typed request model still holds, and a result that skipped validation would quietly give that up on a caller's highest-volume traffic. Taken as an argument rather than remembered: collection happens hours later in another process, and a client that stored the batch's schemas would be the job registry this design does not keep. One schema for the whole batch, because a batch runs one shape — a line that violates it is reported on that line, never by failing the batch.	None

Returns:

Type	Description
BatchResult	The finished batch, lines in submission order, each validated and priced.

Raises:

Type	Description
ConfigError	The provider does not implement batching.
ProviderError	The batch has not finished.

cancel_batch async

cancel_batch(handle: BatchHandle) -> BatchReport

Ask the provider to stop a batch, returning its state afterwards.

Raises:

Type	Description
ConfigError	The provider does not implement batching.

embed async

embed(
    inputs: str | Sequence[str],
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    input_type: Literal[
        "query", "document", "classification", "clustering"
    ]
    | None = None,
    dimensions: int | None = None,
    expected_space: EmbeddingSpace | None = None,
    allow_incompatible_fallback: bool = False,
    batch: BatchPolicy | None = None,
    timeout_s: float | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    retain_raw: bool | None = None,
    manifest: bool | None = None,
) -> EmbeddingResult

Embed one or more texts into vectors.

Parameters:

Name	Type	Description	Default
inputs	str | Sequence[str]	A single text, or an ordered sequence of texts to embed. Duplicates are preserved exactly.	required
target	Target | None	A single target, as for generate().	None
route	Route | Target | Sequence[Target] | None	A fallback chain, as for generate(). Embedding fallback is safe-by-default: a fallback target is dispatched only when it is the identical provider:model as the route's primary target; anything else is refused before any request is sent, unless allow_incompatible_fallback is set.	None
input_type	Literal['query', 'document', 'classification', 'clustering'] | None	What the embedded text will be used for, when the target model distinguishes it.	None
dimensions	int | None	Requested output dimensionality, for models supporting native dimensionality reduction.	None
expected_space	EmbeddingSpace | None	An anyinfer.EmbeddingSpace the result must match; a successful-but-incompatible response is rejected rather than returned.	None
allow_incompatible_fallback	bool	Explicit opt-in permitting fallback to a target that cannot be proven to share the primary target's embedding space. Off by default because wrong-space vectors fail silently when compared; a result served this way always carries a warning naming both targets.	False
batch	BatchPolicy | None	Core-owned batching policy. A request larger than the target's verified batch limit is split into ordered chunks and re-assembled in input order; an unknown limit is never guessed — see anyinfer.BatchPolicy.	None
timeout_s	float | None	Per-attempt wall-clock budget.	None
provider_options	Mapping[str, Mapping[str, Any]] | None	Escape hatch, namespaced by provider id.	None
metadata	Mapping[str, str] | None	Caller-supplied labels carried through telemetry.	None
max_response_bytes	int | None	Hard cap on one provider response body.	None
retain_raw	bool | None	Keep the provider's raw response payload on the result. Defaults to the client's retain_raw setting.	None
manifest	bool | None	Overrides the client's manifest setting for this one call; None inherits it.	None

Returns:

Type	Description
EmbeddingResult	The assembled EmbeddingResult.

Raises:

Type	Description
AllTargetsFailedError	Every target failed.
ConfigError	The resolved target does not support embedding, its response fails the embedding-space safety check, or an incompatible fallback was refused before dispatch.

rerank async

rerank(
    query: str,
    documents: Sequence[str | RerankDocument],
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    top_n: int | None = None,
    batch: BatchPolicy | None = None,
    timeout_s: float | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    return_documents: bool = False,
    retain_raw: bool | None = None,
    manifest: bool | None = None,
) -> RerankResult

Rank documents by relevance to a query.

Parameters:

Name	Type	Description	Default
query	str	The query text every document is scored against.	required
documents	Sequence[str | RerankDocument]	Document texts, or anyinfer.RerankDocument values carrying caller-owned ids. Plain strings are assigned ids "0", "1", ... in order.	required
target	Target | None	A single target, as for generate().	None
route	Route | Target | Sequence[Target] | None	A fallback chain, as for generate().	None
top_n	int | None	Return only the top N ranked items.	None
batch	BatchPolicy | None	Core-owned batching policy. A rerank request larger than the target's verified document limit is refused rather than split, unless BatchPolicy.rerank_cross_batch explicitly accepts chunk-local rankings — scores from separate calls are not globally comparable.	None
timeout_s	float | None	Per-attempt wall-clock budget.	None
provider_options	Mapping[str, Mapping[str, Any]] | None	Escape hatch, namespaced by provider id.	None
metadata	Mapping[str, str] | None	Caller-supplied labels carried through telemetry.	None
max_response_bytes	int | None	Hard cap on one provider response body.	None
return_documents	bool	Echo document text back on each ranked item.	False
retain_raw	bool | None	Keep the provider's raw response payload on the result. Defaults to the client's retain_raw setting.	None
manifest	bool | None	Overrides the client's manifest setting for this one call; None inherits it.	None

Returns:

Type	Description
RerankResult	The assembled RerankResult.

Raises:

Type	Description
AllTargetsFailedError	Every target failed.
ConfigError	The resolved target does not support reranking, or its response names a document index outside the request.

stream

stream(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    logprobs: int | None = None,
    cite_documents: bool = False,
    server_tools: Sequence[ServerToolSpec] = (),
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    max_input_part_bytes: int | None = None,
    max_input_bytes: int | None = None,
    session: Session | None = None,
    manifest: bool | None = None,
) -> AsyncStream

Start a streaming generation.

manifest overrides the client's manifest setting for this one call; None inherits it.

logprobs asks the target to report per-token log-probabilities: 0 for the chosen token's own, a positive count for that many alternatives beside it, and None — the default — for none at all. A target that cannot report them emits a ParameterDropped event rather than answering with an empty Generation.logprobs.

cite_documents asks the target to attribute its answer to the documents this request supplied, landing them on Generation.citations and on CitationDelta events as they arrive. Off by default and never inferred from the presence of a document, since every dialect treats it as a request-side opt-in and several bill a cited answer differently.

server_tools asks the provider to run capabilities of its own during the generation — web search, code execution. Off by default and never inferred: each is billed per invocation. A target that cannot run one refuses before dispatch rather than answering as though it had, since an answer built without the search that was asked for is a different answer, not a degraded one.

Returns:

Type	Description
AsyncStream	An AsyncStream: an async iterator of
AsyncStream	StreamEvent, usable as an async context manager,
AsyncStream	exposing the final result as AsyncStream.result.

run_tools async

run_tools(
    messages: MessagesInput,
    *,
    tools: Sequence[Tool | Any],
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    max_rounds: int = DEFAULT_MAX_ROUNDS,
    **kwargs: Any,
) -> Generation

Generate, dispatching any tools the model calls, until it answers.

Tools run sequentially, and a tool that raises becomes an error-flagged result the model can react to rather than an exception the caller must handle.

Parameters:

Name	Type	Description	Default
messages	MessagesInput	The starting conversation.	required
tools	Sequence[Tool | Any]	Callables or tool()-decorated tools.	required
target	Target | None	A single target, as for generate().	None
route	Route | Target | Sequence[Target] | None	A route, as for generate().	None
max_rounds	int	Maximum tool rounds before giving up.	DEFAULT_MAX_ROUNDS
**kwargs	Any	Forwarded to generate().	{}

Returns:

Type	Description
Generation	The final Generation, once the model stops
Generation	calling tools.

Raises:

Type	Description
ToolLoopError	If the model calls an unknown tool, or the round budget is exhausted.

spend

spend() -> SpendTotals

What this client has spent so far.

Returns zeros; never None, when no ledger is attached, so a caller reading this never has to branch on whether accounting was switched on. Check SpendTotals.unknown_requests before treating the figure as complete: requests against a target with no trusted pricing are counted there rather than being silently priced at zero.

budget

budget(
    messages: MessagesInput,
    *,
    target: Target,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    sampling: Sampling | None = None,
    output_reserve_tokens: int | None = None,
) -> ContextBudget

Compute the context budget for a request without sending it.

This is the preflight calculator: apps assembling large prompts read remaining_tokens to decide how much more material fits, instead of hand-rolling window arithmetic per provider. Pure computation — no request is issued, no network is touched; capabilities come from the catalog plus whatever discovery or probes have already been recorded.

Parameters:

Name	Type	Description	Default
messages	MessagesInput	The conversation as assembled so far.	required
target	Target	The target to budget against.	required
schema	SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None	Structured-output schema the real request will carry, if any.	None
tools	Sequence[ToolSpec]	Tools the real request will offer, if any.	()
sampling	Sampling | None	Sampling controls; max_output_tokens shapes the output reserve.	None
output_reserve_tokens	int | None	Overrides the derived output reserve.	None

Returns:

Type	Description
ContextBudget	The computed ContextBudget. When the
ContextBudget	target's context window is unknown, the budget's verdict is None — never
ContextBudget	a guess.

compare async

compare(
    messages: MessagesInput | GenerationRequest,
    *,
    targets: Sequence[Target],
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
    refresh: bool = False,
) -> tuple[TargetComparison, ...]

Compare how one request would behave across targets without generating.

Results preserve caller order and are never ranked or consumed by routing. With refresh=False (the default), no adapter is constructed and no network is touched. refresh=True may list models to refresh discovered capabilities.

compare_embedding async

compare_embedding(
    inputs: str | Sequence[str],
    *,
    targets: Sequence[Target],
    input_type: EmbeddingInputIntent | None = None,
    refresh: bool = False,
) -> tuple[EmbeddingTargetComparison, ...]

Compare how one embedding request would behave across targets, without dispatching.

Results preserve caller order and are never ranked. With refresh=False (the default), no adapter is constructed and no network is touched; refresh=True may list models to refresh discovered capabilities, exactly as compare().

anyinfer.SyncStream

SyncStream(loop: _LoopThread, factory: Any)

A blocking iterator over stream events, fed from the background loop.

Use it as a context manager so an early exit cancels the underlying request instead of leaving it running:

with client.stream(messages, target="ollama:qwen3:8b") as stream:
    for event in stream:
        ...
    final = stream.result

result property

result: Generation

The final result.

Raises:

Type	Description
RuntimeError	If the stream has not been consumed to completion.

manifest property

manifest: RunManifest | None

What this call has done so far, as a RunManifest.

The blocking mirror of AsyncStream.manifest, and readable at any point — including after close() cancelled the request, which is when it is most useful. None when the client was built with manifests switched off.

__iter__

__iter__() -> Iterator[StreamEvent]

Iterate events as they arrive.

__next__

__next__() -> StreamEvent

Block for the next event, re-raising loop-side exceptions here.

__enter__

__enter__() -> SyncStream

Enter a context that cancels the request on exit.

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the stream, cancelling it if it was not fully consumed.

close

close() -> None

Cancel the underlying request and drain any buffered events.

collect

collect() -> Generation

Drain the stream and return the final result.

anyinfer.AsyncStream

AsyncStream(
    source: AsyncIterator[StreamEvent],
    *,
    builder: ManifestBuilder | None = None,
)

An async iterator over stream events, with the final result attached.

Supports the three consumption shapes the design targets: iterate deltas, watch for the first-token mark and then read the result, or ignore events and read the result.

result property

result: Generation

The final result.

Raises:

Type	Description
RuntimeError	If the stream has not been fully consumed yet.

manifest property

manifest: RunManifest | None

What this call has done so far, as a RunManifest.

Available at any point, which is the whole reason the handle lives on the stream rather than only on the result: a stream that was cancelled or that failed part-way has no Generation to carry a manifest, and that is precisely the call whose story a caller needs. Such a record has complete=False.

None when the client was built with manifests switched off.

__aiter__

__aiter__() -> AsyncStream

Iterate stream events.

__anext__ async

__anext__() -> StreamEvent

Yield the next event, capturing the result when the stream ends.

__aenter__ async

__aenter__() -> AsyncStream

Enter a context that guarantees the stream is closed.

__aexit__ async

__aexit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the underlying generator, cancelling any in-flight request.

aclose async

aclose() -> None

Close the stream early, releasing the provider connection.

collect async

collect() -> Generation

Drain the stream and return the final result.

anyinfer.MessagesInput module-attribute

MessagesInput = str | Message | Sequence[Message]

What callers may pass as messages: a bare prompt, one message, or a sequence.

anyinfer.ProviderSettings dataclass

ProviderSettings(
    provider_id: str,
    base_url: str | None = None,
    api_key: str | None = None,
    api_version: str | None = None,
    headers: Mapping[str, str] = dict(),
    options: Mapping[str, Any] = dict(),
    timeout_s: float = 120.0,
    transport: Any | None = None,
    proxy: str | None = None,
    verify: str | bool | None = None,
    client_cert: str
    | tuple[str, str]
    | tuple[str, str, str]
    | None = None,
    alias: str | None = None,
    limits: RateLimits | None = None,
)

How one provider instance should be configured on a client.

A client may hold several instances of the same underlying engine — two Azure tenants, a local and a remote Ollama — by giving each one an alias. The alias is the instance's identity everywhere else: it is what a alias:model target names, what AdapterPool keys its adapters by, and what telemetry reports.

Attributes:

Name	Type	Description
provider_id	str	Registered provider id or alias, e.g. "openai" or "claude". This selects the engine, which adapter is built and how it talks.
alias	str | None	Instance id, when this is one of several instances of provider_id. Defaults to provider_id itself, which is the single-instance case.
base_url	str | None	Endpoint override. Optional for providers with a default; required for ones that have none (openai-compat, azure-foundry).
api_key	str | None	Credential for the provider. Accepts a reference ("env://OPENAI_API_KEY", "credential://system/openai") as well as a literal; it is resolved once, when the adapter is first built, and registered for redaction at that point.
api_version	str | None	Version pin for providers that take one (Azure, Anthropic).
headers	Mapping[str, str]	Extra headers merged into every request.
options	Mapping[str, Any]	Provider-specific settings, per the provider's documented ProviderSetupSpec fields.
timeout_s	float	Default per-request timeout for this provider.
transport	Any | None	Test seam — an httpx2 transport that intercepts this provider's traffic (used by the fake-server and cassette modes).
limits	RateLimits | None	Client-side pacing for this instance, or None for none. Rate limits belong to an account at a provider rather than to the application, which is why they are configured here and not as a client-wide policy.
proxy	str | None	Proxy URL for this instance's traffic, e.g. "http://corp-proxy:3128". None leaves httpx's HTTPS_PROXY/NO_PROXY environment handling in place, which is the default.
verify	str | bool | None	TLS verification for this instance: a CA-bundle path for a private or intercepting CA, False to disable verification entirely, or None for the default trust store. Per instance on purpose — one provider can trust a corporate CA while another keeps the public roots.
client_cert	str | tuple[str, str] | tuple[str, str, str] | None	Client certificate for mTLS: a combined PEM path, or (cert, key) / (cert, key, password). None when the endpoint needs no client certificate.

Note

proxy, verify, and client_cert are ignored when transport is supplied — a caller bringing its own transport has taken over connection handling. They also do not reach connections AnyInfer does not open for this instance: MCP servers, model downloads, and auth token endpoints other than Google's follow the process environment instead. An adapter that delegates transport to a vendor SDK declares honors_connection_settings=False and the config parser refuses the keys outright rather than accepting them and doing nothing.

instance_id property

instance_id: str

This instance's identity: the alias when set, else the provider id.

of classmethod

of(provider_id: str, **kwargs: Any) -> ProviderSettings

Build settings for a provider id, normalizing the id and any alias.

anyinfer.Session

Session(target: ResolvedTarget, *, supported: bool)

A handle threading related requests through one provider's own state.

Obtained from session(), passed to generate() or stream(), and threaded forward by the caller. Mutable by design — it is a handle, like a stream, not a domain value, and updated in place after each turn so a caller can keep passing the same object.

with client.session("copilot:auto") as chat:
    first = client.generate("Summarize this report.", session=chat)
    follow = client.generate("Now list the risks.", session=chat)
    chat.reuse      # 'resumed' — the provider kept the conversation

Closing stops the handle being used; it does not reach out to the provider. Server-side state expires on the provider's own schedule, and a library that pretended otherwise would be making a promise it cannot keep.

target property

target: ResolvedTarget

The provider and model this session's state belongs to.

supported property

supported: bool

Whether this provider declares it can keep state between requests.

False is not an error: the session is inert, every request behaves exactly as it would without one, and reuse says so on every turn.

reuse property

reuse: SessionReuse

What happened on the most recent turn.

turns property

turns: int

How many requests have been made with this session.

active property

active: bool

Whether the provider is currently holding state for this session.

state property

state: Mapping[str, Any]

The provider's opaque continuation data.

Exposed for diagnostics and persistence, never interpreted by the core. Its contents are the provider's business and may change between releases of that provider, so treat it as a token rather than a structure.

closed property

closed: bool

Whether this handle has been closed.

applies_to

applies_to(target: ResolvedTarget) -> bool

Whether this session's state may be sent to target.

Provider state is not portable: after a fallback to another provider, or a different model on the same one, the stored handle means nothing there.

close

close() -> None

Stop using this handle. Idempotent.

__enter__

__enter__() -> Session

Enter a context that closes the handle on exit.

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the handle.

__aenter__ async

__aenter__() -> Session

Enter an async context that closes the handle on exit.

__aexit__ async

__aexit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the handle.

__repr__

__repr__() -> str

Show the target, turn count, and last reuse outcome.

anyinfer.SessionReuse module-attribute

SessionReuse = Literal['fresh', 'resumed', 'unsupported']

What happened on the most recent turn.

fresh — the provider started new state (the first turn, or one it had expired). resumed — the provider continued state it already held. unsupported — nothing was reused, because this provider cannot or this turn went somewhere else.

anyinfer.Verification dataclass

Verification(
    target: ResolvedTarget | None,
    ok: bool,
    reached: bool = False,
    latency_ms: float = 0.0,
    detail: str = "",
    reply: str = "",
    mechanism: Mechanism | None = None,
    usage: Usage = Usage(),
    diagnostics: tuple[Diagnostic, ...] = (),
)

What one end-to-end probe of a target found.

Never raised, always returned: "this target is broken" is the answer to the question, not a failure to answer it. The two booleans are deliberately separate, because "unreachable" and "reachable but cannot hold a schema" call for completely different fixes.

Attributes:

Name	Type	Description
target	ResolvedTarget | None	What the target string resolved to, and — for a provider that picks the model itself, which model actually served the request.
ok	bool	The provider answered, in the shape asked for, with the expected content.
reached	bool	The provider answered at all. True with ok false means the connection and credential are fine and the model's output was not.
latency_ms	float	Wall-clock time for the whole probe, including any retry the route performed. Indicative only — one request is not a benchmark.
detail	str	What went wrong, or empty when nothing did.
reply	str	A bounded excerpt of what came back, for a human to look at.
mechanism	Mechanism | None	The structured-output mechanism actually used, when one was.
usage	Usage	Tokens the probe spent, as the provider reported them.
diagnostics	tuple[Diagnostic, ...]	Anything the provider said about its own runtime while serving this.

summary property

summary: str

One line suitable for a status area or a CLI.

anyinfer.Measurement dataclass

Measurement(
    identity: MeasurementIdentity,
    input_tokens: int | None = None,
    output_tokens: int | None = None,
    ttft_ms: float | None = None,
    total_ms: float = 0.0,
    prefill_tokens_per_s: float | None = None,
    decode_tokens_per_s: float | None = None,
    model_load_ms: float | None = None,
    measured_at: str | None = None,
)

One target's measured throughput.

Every rate is optional, and None means not measured rather than zero — the same tri-state rule cost and context windows follow.

Attributes:

Name	Type	Description
identity	MeasurementIdentity	What was measured.
input_tokens	int | None	Prompt tokens as the provider counted them.
output_tokens	int | None	Generated tokens as the provider counted them.
ttft_ms	float | None	Time to the first content delta, measured centrally.
total_ms	float	Whole-request wall clock.
prefill_tokens_per_s	float | None	Prompt tokens per second, only when the provider timed its own prefill phase. None otherwise, because deriving it from time-to-first-token would fold queueing and network latency into a figure labelled compute.
decode_tokens_per_s	float | None	Generated tokens per second, from first token to completion.
model_load_ms	float | None	How long the engine spent loading the model before it could answer, when it reported one. This is the warmth signal: a figure here means the run paid a cold start, and None means either the model was already resident or the engine does not report loads at all. Absent that distinction a caller has to measure every target twice and compare, which is what the demo did. Reported by Ollama on every request (load_duration, zero-ish when warm) and by the supervised llama.cpp runtime on the request that started its server. A hosted provider reports nothing: what a shared endpoint spent loading a model is not a property of this request.
measured_at	str | None	ISO-8601 timestamp the caller stamped, when they stamped one.

summary property

summary: str

One line for a status area or a CLI.

to_json

to_json() -> dict[str, Any]

A plain-data form suitable for storage or a machine-readable CLI.

from_json classmethod

from_json(payload: Any) -> Measurement | None

Rebuild a measurement from stored data, or None if it is unreadable.

Never raises: a stored measurement is a cache entry, and an unreadable one means measure again, not fail.

anyinfer.MeasurementIdentity dataclass

MeasurementIdentity(
    provider_id: str,
    model: str,
    endpoint: str | None = None,
    host: str | None = None,
    runtime: str | None = None,
)

What a measurement is a measurement of.

Throughput is not a property of a model; it is a property of a model on an endpoint on a machine with a runtime. Change any of those and the old number is not stale, it is about something else, which is what fingerprint is for.

Attributes:

Name	Type	Description
provider_id	str	The configured provider instance.
model	str	The concrete model that served the request.
endpoint	str | None	Normalized base URL, or None for a supervised in-process engine.
host	str | None	A signature of the machine, for locally-executed targets only. None for hosted providers, where this machine's specs are irrelevant.
runtime	str | None	The local runtime variant in use ("cuda", "metal", …), when one applies.

fingerprint property

fingerprint: str

A stable hash of every field, for use as a store key.

anyinfer.MeasurementStore

MeasurementStore(path: Path | str)

An optional, caller-owned file of past measurements.

The library persists nothing on its own; an application that wants a "last measured" figure across restarts constructs one of these and points it somewhere. Entries are keyed by MeasurementIdentity.fingerprint, so a measurement taken against a different endpoint, machine, or runtime never masquerades as a fresher version of this one.

Reads are total: a missing, truncated, or foreign file yields no entries rather than an exception, because a cache that can break a program is worse than no cache.

path property

path: Path

Where this store reads and writes.

get

get(identity: MeasurementIdentity) -> Measurement | None

The stored measurement for exactly this identity, if any.

all

all() -> tuple[Measurement, ...]

Every stored measurement, oldest entry first.

record

record(measurement: Measurement) -> None

Store a measurement, replacing any earlier one for the same identity.

Writes atomically — a store half-written by an interrupted process would fail every subsequent read.

anyinfer.BenchmarkSample dataclass

BenchmarkSample(
    elapsed_ms: float,
    phase: Literal["warmup", "decode", "complete"],
    estimated_output_tokens: int = 0,
    output_tokens_per_s: float | None = None,
    resources: ResourceSample = ResourceSample(),
)

One point in a live benchmark time series.

The token count and instantaneous rate are estimates while streaming because provider usage is authoritative only at the terminal event. Resource fields are best-effort host readings and remain None when the platform cannot report them.

anyinfer.BENCHMARK_PROMPT_TOKENS module-attribute

BENCHMARK_PROMPT_TOKENS = 2048

Default prompt size for a measurement.

Large enough that prefill is a real phase rather than rounding error, small enough that the whole measurement costs a fraction of a cent on a hosted provider.

anyinfer.BENCHMARK_OUTPUT_TOKENS module-attribute

BENCHMARK_OUTPUT_TOKENS = 128

Default output size. Decode throughput needs enough tokens to average over; a handful would mostly measure the first one.

anyinfer.tool

tool(
    func: Callable[..., Any] | None = None,
    *,
    name: str | None = None,
    description: str | None = None,
) -> Any

Turn a function into a Tool, deriving its schema from the signature.

Parameter types come from annotations and the description from the docstring, so a tool is declared once rather than being kept in sync with a hand-written schema:

@ai.tool
def read_file(path: str) -> str:
    """Read a project file."""
    return Path(path).read_text()

Parameters:

Name	Type	Description	Default
func	Callable[..., Any] | None	The function to wrap, when used bare.	None
name	str | None	Overrides the function's name.	None
description	str | None	Overrides the docstring summary.	None

Returns:

Type	Description
Any	A Tool, or a decorator producing one.

Raises:

Type	Description
ToolLoopError	If a parameter's annotation is not a supported JSON type.

anyinfer.Tool dataclass

Tool(spec: ToolSpec, func: Callable[..., Any])

A callable paired with the ToolSpec derived from its signature.

spec instance-attribute

spec: ToolSpec

The declaration advertised to the model: name, description, and parameter schema.

func instance-attribute

func: Callable[..., Any]

The wrapped callable. May be async def; the loop's dispatcher awaits its result.

name property

name: str

The tool's name, as advertised to the model.

call

call(arguments: Mapping[str, Any]) -> Any

Invoke the underlying function.

An async def tool returns a coroutine here; the loop's dispatcher awaits it.

Deferred Batches

anyinfer.BatchGenerationRequest dataclass

BatchGenerationRequest(
    requests: tuple[GenerationRequest, ...],
    custom_ids: tuple[str, ...] = (),
    completion_window: str | None = "24h",
    metadata: Mapping[str, str] = dict(),
)

A set of generation requests submitted together for deferred, discounted execution.

Every major provider sells a batch tier at roughly half price on a delayed completion window, which is exactly the shape of the workloads this library's audience runs — evals, backfills, offline enrichment. Without this they drop to a raw provider SDK for their highest-volume traffic and lose structured-output enforcement, capability provenance, and cost accounting on precisely the requests where those matter most.

The line-item type is GenerationRequest itself, not a reduced copy of it. A batched request is the same request with a later answer, and a parallel type would drift from the one every adapter already knows how to translate.

Attributes:

Name	Type	Description
requests	tuple[GenerationRequest, ...]	The generation requests to run, in order. Their positions are what custom_ids defaults to, and what a result's lines are ordered by.
custom_ids	tuple[str, ...]	Caller-chosen ids correlating each line to its result. Defaults to the request's index as a string. Supply your own when the results will be joined against something else — a row id, a document key — because a provider returns them in completion order, not submission order.
completion_window	str | None	How long the provider may take, in its own vocabulary ("24h"). None sends nothing and takes the provider's default.
metadata	Mapping[str, str]	Opaque labels carried to the provider where it accepts them, and echoed in telemetry.

line_ids property

line_ids: tuple[str, ...]

The id for each line, defaulting to its index.

__post_init__

__post_init__() -> None

Reject a batch no provider could run, or whose lines cannot be correlated.

Raises:

Type	Description
ValueError	The batch is empty, the id count disagrees with the request count, or an id repeats. A duplicate id is refused rather than tolerated because results come back keyed by it: two lines sharing one id are two answers the caller cannot tell apart.

anyinfer.BatchHandle dataclass

BatchHandle(
    batch_id: str,
    provider_id: str,
    model: str,
    line_count: int,
    line_ids: tuple[str, ...] = (),
    submitted_at: float = 0.0,
    provider_state: Mapping[str, str] = dict(),
)

What a caller keeps in order to reclaim a batch later.

AnyInfer stores no job registry. Run retention is a stated non-goal, and a deferred batch is exactly where it would be most tempting to break it — the answer arrives hours later, in another process. So the handle is a plain value the caller persists wherever they already persist their own work, and every batch call takes one back. Losing it means asking the provider, not asking us.

Attributes:

Name	Type	Description
batch_id	str	The provider's own id for the job.
provider_id	str	Which configured provider instance it was submitted to. Needed to reclaim it: the id is meaningless anywhere else.
model	str	The model every line was submitted against.
line_count	int	How many requests went in, so a partial result is recognizable as one.
line_ids	tuple[str, ...]	The custom ids, in the order they were submitted. Carried because nothing else can restore that order: providers return finished lines in completion order, and sorting them by id only recovers submission order when the ids happen to be the positional defaults. A caller who supplied their own row keys and then zipped the results against their inputs would otherwise get silently mispaired answers — every line correct, attached to the wrong row.
submitted_at	float	Unix timestamp of submission, for a caller's own bookkeeping.
provider_state	Mapping[str, str]	Opaque provider-specific facts the adapter needs to reclaim this job, carried here because there is nowhere else to put them. Where a provider answers into storage the caller owns — Bedrock writes to S3, Vertex to GCS — the output location is not derivable from the job id, and an adapter that remembered it would be the job registry this type exists to avoid. Opaque to the caller: read it for diagnostics, never construct it.

anyinfer.BatchReport dataclass

BatchReport(
    handle: BatchHandle,
    status: BatchStatus,
    completed: int = 0,
    failed: int = 0,
    detail: str = "",
)

A batch's current state, without its results.

Separate from BatchResult because polling should be cheap: providers charge nothing to ask about a job and meaningful bandwidth to download one, and a caller waiting on a 24-hour window will ask many times and fetch once.

Attributes:

Name	Type	Description
handle	BatchHandle	The batch this describes.
status	BatchStatus	Where it is now.
completed	int	Lines finished successfully, when the provider reports counts.
failed	int	Lines that failed, when the provider reports counts.
detail	str	A provider-supplied explanation for a terminal failure; empty otherwise.

finished property

finished: bool

Whether this batch will never change state again.

anyinfer.BatchResult dataclass

BatchResult(
    handle: BatchHandle,
    status: BatchStatus,
    lines: tuple[BatchLine, ...] = (),
)

A finished batch's lines, correlated back to what was submitted.

Attributes:

Name	Type	Description
handle	BatchHandle	The batch these lines belong to.
status	BatchStatus	The batch's terminal status.
lines	tuple[BatchLine, ...]	One entry per line the provider returned, in submission order rather than the completion order providers return them in — a caller zipping results against their own inputs should not have to sort first. Restored from the handle's line_ids; see there for why sorting cannot substitute.

succeeded property

succeeded: tuple[BatchLine, ...]

Only the lines that produced a result.

failed property

failed: tuple[BatchLine, ...]

Only the lines that failed.

anyinfer.BatchLine dataclass

BatchLine(
    custom_id: str,
    result: Generation | None = None,
    error: ErrorInfo | None = None,
)

One line's outcome within a finished batch.

Exactly one of result and error is set. A batch is not all-or-nothing — providers run and bill the lines that succeeded even when others failed — so a line type that could not carry a per-line failure would force the whole batch to be discarded over one bad request.

Attributes:

Name	Type	Description
custom_id	str	The id this line was submitted under.
result	Generation | None	The generation, when this line succeeded.
error	ErrorInfo | None	What went wrong, when it did.

ok property

ok: bool

Whether this line produced a result.

anyinfer.BatchStatus module-attribute

BatchStatus = Literal[
    "queued",
    "in_progress",
    "completed",
    "failed",
    "expired",
    "cancelled",
]

Where a deferred batch is in its life, normalized across providers.

Terminal states are completed, failed, expired, and cancelled: a provider that reports something else has moved, and the mapping is the adapter's to update rather than the caller's to guess at.