Clients and streams

The two entry points — Client (sync) and AsyncClient (async) — expose the same surface; the sync client is a facade over the async core (see the architecture overview).

anyinfer.Client

Client(
    providers: Sequence[ProviderSettings] | None = None,
    *,
    registry: ProviderRegistry | None = None,
    catalog: Catalog | None = None,
    route: Route | None = None,
    observers: Sequence[Observer] | None = None,
    resolver: ResolverChain | None = None,
    retain_raw: bool = False,
    repair: Repair | None = None,
    use_default_catalog: bool = True,
    estimator: TokenEstimator | None = None,
    context_gate: bool = True,
    pricing_table: PricingTable | None = None,
    capability_overrides: Mapping[str, ModelCapabilities]
    | None = None,
    model_dir: Path | None = None,
)

The synchronous inference client.

A thin facade: every method schedules work on one background event loop that owns the real AsyncClient. Safe to call from multiple threads, and concurrent requests still overlap on the loop.

Args are identical to AsyncClient.

model_store property

model_store: ModelStore

The store acquired model weights live in.

__enter__

__enter__() -> Client

Enter a context that closes the client on exit.

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the client on context exit.

close

close() -> None

Close adapters, stop the background loop, and join its thread.

subscribe

subscribe(
    observer: Observer, *, payloads: bool = False
) -> None

Register a telemetry observer.

unsubscribe

unsubscribe(observer: Observer) -> None

Remove a telemetry observer.

models

models(provider_id: str) -> Sequence[DiscoveredModel]

List a provider's models.

health

health(provider_id: str) -> Health

Probe a provider's readiness.

resolve

resolve(target: Target) -> ResolvedTarget

Resolve a target string without issuing a request.

local_catalog

local_catalog(
    provider_id: str | None = None,
    *,
    hardware: HardwareProfile | None = None,
    best_at: str | None = None,
    posture: Posture = "balanced",
) -> CatalogView

Browse the local model catalog, annotated with how each entry fits.

See AsyncClient.local_catalog.

acquire_model

acquire_model(
    model_id: str,
    *,
    engine: str | None = None,
    variant_id: str | None = None,
    hardware: HardwareProfile | None = None,
    progress: ProgressSink | None = None,
    prefs: VariantPrefs | None = None,
    dry_run: bool = False,
    token: str | None = None,
) -> AcquisitionReport

Download a catalog model's weights. See AsyncClient.acquire_model.

The progress sink is invoked from the background loop thread, so it must not block and must not call back into this client.

installed_models

installed_models() -> Sequence[StoreEntry]

Every model acquired into this client's store.

locate_model

locate_model(
    model_id: str,
    *,
    variant_id: str | None = None,
    engine: str | None = None,
    verify: bool = False,
) -> ResolvedModel | None

Find an acquired model on disk. See AsyncClient.locate_model.

remove_model

remove_model(entry_id: str) -> RemovalReport

Delete an acquired model. See AsyncClient.remove_model.

budget

budget(
    messages: MessagesInput,
    *,
    target: Target,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    sampling: Sampling | None = None,
    output_reserve_tokens: int | None = None,
) -> ContextBudget

Compute the context budget for a request without sending it.

Pure computation — runs directly on the calling thread. See AsyncClient.budget().

generate

generate(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
) -> Generation

Generate a single result. See AsyncClient.generate().

run_tools

run_tools(
    messages: MessagesInput,
    *,
    tools: Sequence[Any],
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    max_rounds: int = DEFAULT_MAX_ROUNDS,
    **kwargs: Any,
) -> Generation

Generate, dispatching tools until the model answers.

See AsyncClient.run_tools().

stream

stream(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
) -> SyncStream

Start a streaming generation, returning a blocking iterator.

See AsyncClient.stream(). Use the result as a context manager so that leaving the block early cancels the in-flight request.

anyinfer.AsyncClient

AsyncClient(
    providers: Sequence[ProviderSettings] | None = None,
    *,
    registry: ProviderRegistry | None = None,
    catalog: Catalog | None = None,
    route: Route | None = None,
    observers: Sequence[Observer] | None = None,
    resolver: ResolverChain | None = None,
    retain_raw: bool = False,
    repair: Repair | None = None,
    use_default_catalog: bool = True,
    estimator: TokenEstimator | None = None,
    context_gate: bool = True,
    pricing_table: PricingTable | None = None,
    capability_overrides: Mapping[str, ModelCapabilities]
    | None = None,
    model_dir: Path | None = None,
)

The asynchronous inference client.

Parameters:

Name	Type	Description	Default
providers	Sequence[ProviderSettings] | None	Per-provider settings. The order given is the preference order used when resolving catalog aliases.	None
registry	ProviderRegistry | None	Provider registry; defaults to the process-wide one.	None
catalog	Catalog | None	Alias catalog; defaults to the bundled catalog. Pass None explicitly via use_default_catalog=False to disable alias resolution.	None
route	Route | None	Default route applied when a call names no target.	None
observers	Sequence[Observer] | None	Telemetry observers, registered payload-free.	None
resolver	ResolverChain | None	Credential resolver chain.	None
retain_raw	bool	Keep the provider's raw payload on results. Off by default because raw payloads carry response text that payload-free telemetry deliberately omits.	False
repair	Repair | None	Default repair budget for schema violations.	None
use_default_catalog	bool	Load the bundled catalog when catalog is not supplied.	True
estimator	TokenEstimator | None	Token counting strategy for budgets and the pre-dispatch gate. Defaults to the dependency-free byte heuristic.	None
context_gate	bool	Fail a target before dispatch when the request provably cannot fit its known context window. Only trusted-provenance windows gate, and only on the estimate's floor, so a heuristic never refuses a request that might have fit.	True
pricing_table	PricingTable | None	Model pricing supplying the catalog layer of capability assembly. Defaults to the table bundled with this release; pass the result of fetch_pricing() for newer numbers.	None
capability_overrides	Mapping[str, ModelCapabilities] | None	Deliberate corrections keyed by "provider:model". Every supplied field is applied at override provenance — the strongest layer, outranking discovery and probes — so a wrong upstream number can always be fixed locally.	None
model_dir	Path | None	Where acquired model weights are stored. Defaults to the per-OS data directory, overridable with ANYINFER_MODEL_DIR.	None

catalog property

catalog: Catalog | None

The alias catalog in force, if any.

model_store property

model_store: ModelStore

The store acquired model weights live in.

__aenter__ async

__aenter__() -> AsyncClient

Enter an async context managing this client's adapters.

__aexit__ async

__aexit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the client on context exit.

aclose async

aclose() -> None

Close every adapter this client built.

subscribe

subscribe(
    observer: Observer, *, payloads: bool = False
) -> None

Register a telemetry observer.

Parameters:

Name	Type	Description	Default
observer	Observer	The sink.	required
payloads	bool	Opt in to prompt and response text. Off by default, so no observer sees payload text it did not explicitly ask for.	False

unsubscribe

unsubscribe(observer: Observer) -> None

Remove a telemetry observer.

models async

models(provider_id: str) -> Sequence[DiscoveredModel]

List a provider's models, recording what they report about capabilities.

health async

health(provider_id: str) -> Health

Probe a provider's readiness.

resolve

resolve(target: Target) -> ResolvedTarget

Resolve a target string without issuing a request.

local_catalog async

local_catalog(
    provider_id: str | None = None,
    *,
    hardware: HardwareProfile | None = None,
    best_at: str | None = None,
    posture: Posture = "balanced",
) -> CatalogView

Browse the local model catalog, annotated with how each entry fits.

Performs no network I/O: the catalog is bundled data and hardware detection is local and cached.

Parameters:

Name	Type	Description	Default
provider_id	str | None	Restrict to models one configured engine can serve. Its locality also decides whether this machine is the right one to probe.	None
hardware	HardwareProfile | None	Specs to judge against, overriding detection. This is how an application answers for a remote host after asking the user.	None
best_at	str | None	One category from the catalog's closed vocabulary.	None
posture	Posture	How much of the machine to budget.	'balanced'

Returns:

Type	Description
CatalogView	The view. hardware_source == "unavailable" means the engine runs somewhere
CatalogView	AnyInfer cannot probe, and the application should collect the host's specs and
CatalogView	call again.

Raises:

Type	Description
ConfigError	If best_at names a category no catalog entry uses.

acquire_model async

acquire_model(
    model_id: str,
    *,
    engine: str | None = None,
    variant_id: str | None = None,
    hardware: HardwareProfile | None = None,
    progress: ProgressSink | None = None,
    prefs: VariantPrefs | None = None,
    dry_run: bool = False,
    token: str | None = None,
) -> AcquisitionReport

Download a catalog model's weights into this client's model store.

The quantization is chosen, not assumed: the highest curated rung whose weights and KV cache fit this machine's budget. Pass variant_id to override that.

dry_run=True resolves everything and reports the exact byte count without writing anything — what an application needs to confirm a large download with a user before starting it.

Parameters:

Name	Type	Description	Default
model_id	str	A catalog model id.	required
engine	str | None	"llama.cpp" or "vllm"; defaults to whatever the model offers.	None
variant_id	str | None	Acquire this exact variant, skipping selection.	None
hardware	HardwareProfile | None	Specs to select against, overriding detection.	None
progress	ProgressSink | None	Aggregate progress sink. See AcquisitionProgress for the threading contract it must honor.	None
prefs	VariantPrefs | None	Selection preferences, including the low-quality opt-in.	None
dry_run	bool	Plan and report without writing.	False
token	str | None	A credential for the source, when it needs one. Defaults to HF_TOKEN.	None

Returns:

Type	Description
AcquisitionReport	The report, naming the registered entry and what it cost.

Raises:

Type	Description
ConfigError	If there is no catalog, the model is unknown, or nothing fits.
LocalRuntimeError	On a transfer failure, digest mismatch, or full disk.

installed_models async

installed_models() -> Sequence[StoreEntry]

Every model acquired into this client's store.

locate_model async

locate_model(
    model_id: str,
    *,
    variant_id: str | None = None,
    engine: str | None = None,
    verify: bool = False,
) -> ResolvedModel | None

Find an acquired model on disk, with advisory launch arguments.

No network I/O. Verification is shallow by default — size and modification time against the index — because re-hashing forty gigabytes on every lookup would be absurd; verify=True forces the full check.

remove_model async

remove_model(entry_id: str) -> RemovalReport

Delete an acquired model's files and unregister it.

A model adopted from somebody else's cache is only unregistered; its files belong to whatever put them there.

generate async

generate(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
) -> Generation

Generate a single result, draining the event stream internally.

Returns:

Type	Description
Generation	The assembled Generation.

Raises:

Type	Description
AllTargetsFailedError	Every target failed.
SchemaViolationError	The response never satisfied the schema.

stream

stream(
    messages: MessagesInput,
    *,
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling | None = None,
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    repair: Repair | None = None,
    provider_options: Mapping[str, Mapping[str, Any]]
    | None = None,
    metadata: Mapping[str, str] | None = None,
    max_response_bytes: int | None = None,
) -> AsyncStream

Start a streaming generation.

Returns:

Type	Description
AsyncStream	An AsyncStream: an async iterator of
AsyncStream	StreamEvent, usable as an async context manager,
AsyncStream	exposing the final result as AsyncStream.result.

run_tools async

run_tools(
    messages: MessagesInput,
    *,
    tools: Sequence[Tool | Any],
    target: Target | None = None,
    route: Route | Target | Sequence[Target] | None = None,
    max_rounds: int = DEFAULT_MAX_ROUNDS,
    **kwargs: Any,
) -> Generation

Generate, dispatching any tools the model calls, until it answers.

Tools run sequentially, and a tool that raises becomes an error-flagged result the model can react to rather than an exception the caller must handle.

Parameters:

Name	Type	Description	Default
messages	MessagesInput	The starting conversation.	required
tools	Sequence[Tool | Any]	Callables or tool()-decorated tools.	required
target	Target | None	A single target, as for generate().	None
route	Route | Target | Sequence[Target] | None	A route, as for generate().	None
max_rounds	int	Maximum tool rounds before giving up.	DEFAULT_MAX_ROUNDS
**kwargs	Any	Forwarded to generate().	{}

Returns:

Type	Description
Generation	The final Generation, once the model stops
Generation	calling tools.

Raises:

Type	Description
ToolLoopError	If the model calls an unknown tool, or the round budget is exhausted.

budget

budget(
    messages: MessagesInput,
    *,
    target: Target,
    schema: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any]
    | None = None,
    tools: Sequence[ToolSpec] = (),
    sampling: Sampling | None = None,
    output_reserve_tokens: int | None = None,
) -> ContextBudget

Compute the context budget for a request without sending it.

This is the preflight calculator: apps assembling large prompts read remaining_tokens to decide how much more material fits, instead of hand-rolling window arithmetic per provider. Pure computation — no request is issued, no network is touched; capabilities come from the catalog plus whatever discovery or probes have already been recorded.

Parameters:

Name	Type	Description	Default
messages	MessagesInput	The conversation as assembled so far.	required
target	Target	The target to budget against.	required
schema	SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None	Structured-output schema the real request will carry, if any.	None
tools	Sequence[ToolSpec]	Tools the real request will offer, if any.	()
sampling	Sampling | None	Sampling controls; max_output_tokens shapes the output reserve.	None
output_reserve_tokens	int | None	Overrides the derived output reserve.	None

Returns:

Type	Description
ContextBudget	The computed ContextBudget. When the
ContextBudget	target's context window is unknown, the budget's verdict is None — never
ContextBudget	a guess.

anyinfer.SyncStream

SyncStream(loop: _LoopThread, factory: Any)

A blocking iterator over stream events, fed from the background loop.

Use it as a context manager so an early exit cancels the underlying request instead of leaving it running:

with client.stream(messages, target="ollama:qwen3:8b") as stream:
    for event in stream:
        ...
    final = stream.result

result property

result: Generation

The final result.

Raises:

Type	Description
RuntimeError	If the stream has not been consumed to completion.

__iter__

__iter__() -> Iterator[StreamEvent]

Iterate events as they arrive.

__next__

__next__() -> StreamEvent

Block for the next event, re-raising loop-side exceptions here.

__enter__

__enter__() -> SyncStream

Enter a context that cancels the request on exit.

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the stream, cancelling it if it was not fully consumed.

close

close() -> None

Cancel the underlying request and drain any buffered events.

collect

collect() -> Generation

Drain the stream and return the final result.

anyinfer.AsyncStream

AsyncStream(source: AsyncIterator[StreamEvent])

An async iterator over stream events, with the final result attached.

Supports the three consumption shapes the design targets: iterate deltas, watch for the first-token mark and then read the result, or ignore events and read the result.

result property

result: Generation

The final result.

Raises:

Type	Description
RuntimeError	If the stream has not been fully consumed yet.

__aiter__

__aiter__() -> AsyncStream

Iterate stream events.

__anext__ async

__anext__() -> StreamEvent

Yield the next event, capturing the result when the stream ends.

__aenter__ async

__aenter__() -> AsyncStream

Enter a context that guarantees the stream is closed.

__aexit__ async

__aexit__(
    exc_type: type[BaseException] | None,
    exc: BaseException | None,
    tb: TracebackType | None,
) -> None

Close the underlying generator, cancelling any in-flight request.

aclose async

aclose() -> None

Close the stream early, releasing the provider connection.

collect async

collect() -> Generation

Drain the stream and return the final result.

anyinfer.MessagesInput module-attribute

MessagesInput = str | Message | Sequence[Message]

What callers may pass as messages: a bare prompt, one message, or a sequence.

anyinfer.ProviderSettings dataclass

ProviderSettings(
    provider_id: str,
    base_url: str | None = None,
    api_key: str | None = None,
    api_version: str | None = None,
    headers: Mapping[str, str] = dict(),
    options: Mapping[str, Any] = dict(),
    timeout_s: float = 120.0,
    transport: Any | None = None,
    alias: str | None = None,
)

How one provider instance should be configured on a client.

A client may hold several instances of the same underlying engine — two Azure tenants, a local and a remote Ollama — by giving each one an alias. The alias is the instance's identity everywhere else: it is what a alias:model target names, what AdapterPool keys its adapters by, and what telemetry reports.

Attributes:

Name	Type	Description
provider_id	str	Registered provider id or alias, e.g. "openai" or "claude". This selects the engine — which adapter is built and how it talks.
alias	str | None	Instance id, when this is one of several instances of provider_id. Defaults to provider_id itself, which is the single-instance case.
base_url	str | None	Endpoint override. Optional for providers with a default; required for ones that have none (openai-compat, azure-foundry).
api_key	str | None	Credential for the provider. Accepts a reference ("env://OPENAI_API_KEY", "credential://system/openai") as well as a literal; it is resolved once, when the adapter is first built, and registered for redaction at that point.
api_version	str | None	Version pin for providers that take one (Azure, Anthropic).
headers	Mapping[str, str]	Extra headers merged into every request.
options	Mapping[str, Any]	Provider-specific settings, per the provider's documented ProviderSetupSpec fields.
timeout_s	float	Default per-request timeout for this provider.
transport	Any | None	Test seam — an httpx2 transport that intercepts this provider's traffic (used by the fake-server and cassette modes).

instance_id property

instance_id: str

This instance's identity: the alias when set, else the provider id.

of classmethod

of(provider_id: str, **kwargs: Any) -> ProviderSettings

Build settings for a provider id, normalizing the id and any alias.

anyinfer.tool

tool(
    func: Callable[..., Any] | None = None,
    *,
    name: str | None = None,
    description: str | None = None,
) -> Any

Turn a function into a Tool, deriving its schema from the signature.

Parameter types come from annotations and the description from the docstring, so a tool is declared once rather than being kept in sync with a hand-written schema:

@ai.tool
def read_file(path: str) -> str:
    """Read a project file."""
    return Path(path).read_text()

Parameters:

Name	Type	Description	Default
func	Callable[..., Any] | None	The function to wrap, when used bare.	None
name	str | None	Overrides the function's name.	None
description	str | None	Overrides the docstring summary.	None

Returns:

Type	Description
Any	A Tool, or a decorator producing one.

Raises:

Type	Description
ToolLoopError	If a parameter's annotation is not a supported JSON type.

anyinfer.Tool dataclass

Tool(spec: ToolSpec, func: Callable[..., Any])

A callable paired with the ToolSpec derived from its signature.

spec instance-attribute

spec: ToolSpec

The declaration advertised to the model: name, description, and parameter schema.

func instance-attribute

func: Callable[..., Any]

The wrapped callable. May be async def; the loop's dispatcher awaits its result.

name property

name: str

The tool's name, as advertised to the model.

call

call(arguments: Mapping[str, Any]) -> Any

Invoke the underlying function.

An async def tool returns a coroutine here; the loop's dispatcher awaits it.