# AnyInfer - full documentation Version 0.1.2. Generated from the built site; the link index is at https://anyinfer.dev/llms.txt. --- # Home / Home Source: https://anyinfer.dev/ One typed inference layer for hosted APIs and a supervised local model — with a test kit for your failure paths, provenance on every number, and a confidentiality story no other BYOK library ships. Quickstart Installation Downloads approved context → budget + route → cloud or local Typed events A generation is an ordered stream of typed events; non-streaming is just the drained stream. Routing & fallback Deterministic retries, failure-specific fallback chains, and health gating — fully traceable after the fact. Local inference Hardware detection, tuning, verified downloads, and supervised llama-server — one target string. Structured output A schema is a contract: strongest native mechanism, always client-side validated, optional bounded repair. Context budgets Estimate fit and cost before dispatch, then reduce approved context with an explicit record of what was omitted. Confidentiality tiers Encrypted-at-rest templates, zero-retention orchestration, and one function that tells you if a box can run a model under hardware attestation — honestly tiered, nowhere else. AnyInfer SyncAsync import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")]) result = client.generate(prompt, target="anthropic:claude-sonnet-4-5") print(result.text) import anyinfer as ai async with ai.AsyncClient( [ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")] ) as client: result = await client.generate(prompt, target="anthropic:claude-sonnet-4-5") print(result.text) AnyInfer provides a provider-independent inference runtime for Python applications that span hosted providers and local models, as well as an OpenAI-compatible sidecar for everything that is not Python. Install pip install anyinfer The core depends on only httpx2 and jsonschema. Provider SDKs, the sidecar, and the demo app are optional extras. Local inference is part of the core. See installation and extras. What Makes This Different Most libraries in this space solve provider switching: one function, many APIs, one response shape. AnyInfer solves the problem that starts right after: being correct about what you sent, what you got back, what it cost, and what quietly didn't happen. Your fallback chain has a real test, with no credentials and no network. The test kit ships with the library (script a 503, a malformed schema response, a rate limit) and allows you to assert on the recovery, not on a mock of your own wrapper. → Testing your app Every number says where it came from. A context window, a price, a feature flag is tagged cataloged, discovered, probed, or defaulted, and an unknown cost is None, never $0.00. → Capabilities and provenance Structured output is a contract. A request carrying a schema always returns a client-side-validated result, using the strongest mechanism the provider offers, with an opt-in bounded repair loop. → Structured output Portability is a test result, not a claim. compare() reports exactly what a fixed request becomes on a different target before you spend anything, and the conformance matrix is generated from executed tests. → Comparing targets A local model is a target, not a separate product. Point the same call at llama-cpp:qwen3-8b-q4-k-m and AnyInfer acquires, verifies, and supervises the weights itself: same fallback chain, event stream, and structured-output contract, no separate daemon. → Run a model locally Context engineering is part of dispatch. A provenance-aware budget estimates input, reserve, and cost before a call; deterministic reducers fit approved corpora to it and report exactly what they omitted. → Context reduction → Read the full case, with runnable proof for every claim. One Engine, Four Kinds of Target Environment Examples What AnyInfer owns Hosted provider OpenAI, Anthropic, Gemini, Bedrock Native protocol translation and capability discovery Router or hub OpenRouter, compatible gateways Targeting, normalized events, and shared routing policy Existing local service Ollama, LM Studio, vLLM Native or compatible client behavior; the service keeps process ownership Managed local runtime llama.cpp Runtime and model acquisition, hardware fit, tuning, supervision, and loopback lifecycle → See the compatibility inventory: dedicated protocol adapters and declarative presets, from frontier APIs to local engines, with per-provider guides. Embeddings and reranking are typed, routed operations on the same client; see embeddings and reranking. Next Steps Deciding whether you need this layer? Start with why and when to use AnyInfer. It names the cases where a provider client, organization gateway, or dedicated local server is the better tool. Integrating into an app? The Quickstart is the five-minute route from install to a result; Integrate AnyInfer chooses between the SDK, CLI, and sidecar. Want existing OpenAI clients to use the same route? Run the sidecar, an OpenAI-compatible loopback service. Anything that can point at an OpenAI base URL can use the providers, routes, and local models you configured. Just want to see it? The pack-in demo app runs fully offline against in-process fakes (no credentials required). Grab a standalone build from Downloads. Pre-1.0 and under active development. Python 3.11+; Windows, macOS, and Linux are all first-class. MIT licensed. Sources, design documents, and the issue tracker live at github.com/anthturner/AnyInfer. v0.1.2 Python 3.11+ Conformance matrix → --- # Integrate / Integrate AnyInfer Source: https://anyinfer.dev/guides/ Integrate AnyInfer Three supported production paths share the same core behavior and configuration file; only the process boundary changes. Not every application needs this layer at all; if a provider-switching client, an organization gateway, or a dedicated local server already solves your whole problem, why and when to use AnyInfer names the better-shaped tool. The quickstart is the shortest path from installation to a result. Choose a Path Embed the Python SDK when you are writing Python and want typed results, the event stream, and in-process telemetry. The cost: AnyInfer is in your dependency tree and your process. Run the OpenAI-compatible sidecar when your application is not Python, when one process should hold provider credentials for several clients, or when existing OpenAI-speaking tools should use your configured hybrid route: pip install "anyinfer[serve]" anyinfer serve --config anyinfer.json The cost: an HTTP hop, and the OpenAI wire format cannot carry AnyInfer-native observability (timing marks and attempt records have no chunk representation, though usage and finish reasons survive). Use anyinfer run when a person or a shell script needs one answer with no server left running. The cost: process startup per call and no state between calls; it declares tools but never executes them (that is the tool loop). Python SDK Sidecar CLI run Language Python Any Any (a shell) Typed results Yes OpenAI JSON Text or JSON Event stream Full Text and tool-call deltas Text to stdout TTFT / attempt trail Yes Not on the wire --stats / --json (timing and usage; no attempt trail) Local models Yes Yes Yes Credentials In-process Held by the server Read per call Deployment A library A process A command The paths compose: anyinfer.serve.create_app(async_client, auth_token=token) returns a plain ASGI app, mountable inside an existing Starlette or FastAPI application, so one process can embed the SDK and expose the frontend. In order to evaluate everything offline first, the reference application runs against in-process fakes with no credentials. Python Tasks Stream typed events Enforce a JSON schema Add a fallback chain Run the tool loop Fit a corpus to a budget Test your application offline Compare targets without spending Add your own provider Embed, store, and query a small corpus Operations Run a model locally Keep the sidecar running across reboots Observe requests and bridge to OpenTelemetry Credentials and redaction Confidentiality Confidentiality tiers: protecting prompt IP shipped to customer machines, including the SOC 2 control mapping. Coding Agents Coding agents: anyinfer agents-md, llms.txt, and the integration procedure a skill can execute. --- # Integrate / Quickstart Source: https://anyinfer.dev/guides/quickstart/ Quickstart From pip install to a working result. Every example on this page is executed in CI against the fake providers, so none of it can quietly rot. Install pip install anyinfer The core depends on httpx2 and jsonschema and nothing else. Providers that need more come as extras; see installation. The fastest start is anyinfer init: it inspects the machine, reports which providers are already usable (a running Ollama, a set credential variable), and writes a valid anyinfer.json plus a runnable starter.py, without ever storing a secret or installing anything. The CLI guide covers what it detects and its flags. The file it writes is the shared configuration the SDK, CLI, and sidecar all read. Your First Call SyncAsync import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) result = client.generate( "Summarize this in one sentence:\n" + text, target="anthropic:claude-sonnet-4-5", ) print(result.text) client.close() import anyinfer as ai async with ai.AsyncClient( [ ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) as client: result = await client.generate( "Summarize this in one sentence:\n" + text, target="anthropic:claude-sonnet-4-5", ) print(result.text) The highlighted line is the only one that changes when you point the same call at a different provider or a local model. Note the credential: "env://ANTHROPIC_API_KEY" is a reference, safe to keep in a config file. It is resolved once and registered for redaction, so the key can never appear in a log line or an error message. See credentials. Client owns a background event loop, so use it as a context manager or call close(): with ai.Client([ai.ProviderSettings.of("ollama")]) as client: result = client.generate("Why is the sky blue?", target="ollama:qwen3:8b") Streaming with client.stream(messages, target="ollama:qwen3:8b") as stream: for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) final = stream.result print(f"\n\n{final.usage.output_tokens} tokens in {final.timing.total_ms:.0f} ms") Using the stream as a context manager matters: leaving the block early cancels the in-flight request instead of letting it run on. See stream typed events. Aliases: Don't Hardcode Model Names small, medium, and large resolve to a concrete model for whichever provider you have configured: client = ai.Client( [ ai.ProviderSettings.of("ollama"), ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) result = client.generate(prompt, target="medium") # -> ollama, since it is listed first The order you configure providers is the preference order. See targets and aliases. Structured Output Pass a JSON schema and get back a validated Python value: SUMMARY = { "type": "object", "properties": { "headline": {"type": "string"}, "topics": {"type": "array", "items": {"type": "string"}}, }, "required": ["headline", "topics"], } result = client.generate( article, target="medium", schema=SUMMARY, repair=ai.Repair(max_attempts=1), ) print(result.structured["headline"]) print(result.structured_mechanism) # "json_schema", "grammar", "json_mode", or "prompt" AnyInfer uses the strongest mechanism the provider supports, then validates the result against your schema regardless. repair allows the model to correct itself once if it gets the shape wrong. See structured output. Fallback Chains route = ai.Route( targets=("anthropic:claude-sonnet-4-5", "openai:gpt-5", "ollama:qwen3:8b"), retry=ai.Retry(max_attempts=3), ) result = client.generate(prompt, route=route) print(f"served by {result.target}") for attempt in result.attempts: print(f" {attempt.target} -> {attempt.outcome}") Every result carries its full routing trail, so "why was this slow?" is answerable after the fact. See routing and rate limits. Beyond Generation The same client embeds and reranks (client.embed(), client.rerank()), typed and routed like generation, with a safety rule that keeps fallback from mixing incompatible vector spaces. And a local model is just another target: with llama-cpp configured, one generate() call downloads a pinned, hash-verified model, tunes a server for your hardware, and answers on loopback; see run a model locally. Key Takeaways One call shape covers every provider; only the target= string changes. Credentials are references (env://…), resolved once and redacted everywhere. A schema is validated client-side no matter which mechanism the provider offers. Every result carries its attempt trail, so routing decisions are inspectable after the fact. See Also Concepts: the model behind the API. Integrate AnyInfer: SDK, CLI, or sidecar. Providers: the quirks of each backend. --- # Integrate / Installation and Extras Source: https://anyinfer.dev/guides/installation/ Installation and Extras pip install anyinfer The core depends on httpx2 and jsonschema and nothing else. That constraint is a security argument as much as an aesthetic one: a small mandatory dependency surface is a small supply-chain attack surface. Extras Extra Adds Needed for copilot github-copilot-sdk The copilot provider azure azure-identity Entra auth for azure-foundry, and m365-copilot vertex cryptography Signing a Vertex service-account assertion attest cryptography Tier 4 model-manifest signature verification (anyinfer.local.provenance) keyring keyring credential:// references otel opentelemetry-api The OpenTelemetry bridge serve starlette, uvicorn The OpenAI-compatible sidecar demo PySide6, markdown The pack-in demo app (anyinfer-demo) all everything above pip install "anyinfer[serve,otel]" Optional packages are imported only when their feature is used. A missing package produces an actionable ConfigError or CredentialError with the install command, never a raw ImportError traceback. Optional Add-On Packages Some features ship as separate, independently-versioned distributions rather than as anyinfer extras (never imported by core, never a dependency of it): Package Adds Needed for anyinfer-confidential SealedTemplate, TemplateVault, the AnyInfer Relay Confidentiality tiers 1-2 anyinfer-shared ConfidentialityReport Composing confidentiality facts from both anyinfer-confidential and anyinfer core in one type anyinfer-store VectorStore, query_and_rerank Small-scale embedded vector storage over embed()/rerank() results pip install -e src/anyinfer-confidential # from a repository checkout, until a first pip install -e src/anyinfer-shared # PyPI release ships pip install -e src/anyinfer-store Which Providers Need Nothing Extra openai, anthropic, openai-compat, openrouter, ollama, and llama-cpp are pure httpx2 and work with a bare install. The local subsystem, including hardware detection and the llama-server supervisor, is core. Requirements Python 3.11+ Windows, macOS, and Linux are all first-class and all tested in CI. llama-server runtimes are installed explicitly with anyinfer runtime install; catalog models are acquired on demand or with anyinfer models add. Neither is bundled in the wheel, and both paths verify the downloaded artifacts. For non-Python deployments, release builds of the demo and sidecar are listed on the Downloads page. The 0.1 beta native bundles are checksummed but not code-signed; verify SHA256SUMS from the GitHub Release before running them. Verify the Install anyinfer providers # every registered provider and what it needs anyinfer doctor # detected hardware and the recommended local tier --- # Integrate / Why and When to Use AnyInfer Source: https://anyinfer.dev/why-anyinfer/ Why and When to Use AnyInfer Most libraries in this space solve provider switching: one function, many APIs, one response shape. That is a real problem and several tools solve it well. AnyInfer is built for the problem that starts immediately afterward, when an application has to be correct about what it sent, what it got back, what it cost, and what quietly did not happen. An OpenAI-shaped request does not make providers behave alike. One supports a JSON schema natively, one has a grammar, one only has "JSON mode", one drops your top_p without saying so. One reports cached tokens inside the prompt total, one beside it. One tells you its context window, one guesses, one says nothing. A library that normalizes the syntax and leaves the behavior to you has moved the problem, not solved it. AnyInfer normalizes the behavior and reports every place it could not. This page argues both directions: what is genuinely unusual here, and when a smaller tool is the better boundary. Five Things That Are Unusual 1. You Can Unit-Test Your Integration's Failure Paths, Offline Inference code has behavior worth testing: it falls back when a provider is down, it repairs a malformed structured answer, it reduces a corpus to fit. Testing that normally means mocking your own wrapper, which mostly tests the mock, or provoking a real outage. The test kit ships with the library, so your fallback chain has a real test with no credentials and no network: from anyinfer.testing import ScriptedFailure, ScriptedModel, ScriptedProvider provider = ScriptedProvider( "acme", [ ScriptedModel("flaky", failures=(ScriptedFailure(status=503, retry_after_s=0.0),)), ScriptedModel( "structured", structured={"answer": "valid on the second try"}, failures=(ScriptedFailure(kind="malformed-json"),), ), ], ) Five failure kinds are declarable (an HTTP status with Retry-After, a stream cut mid-event, a body that will not validate, a read timeout, and a content-policy refusal), and each reaches a different part of the core. These are the failures you cannot schedule against a real provider, and they are exactly the ones your error handling is written for. See test your application offline. 2. Every Number Says Where It Came From A context window you read from a table and a context window the provider just told you are not the same fact, and code that cannot tell them apart will eventually gate a request on a guess. budget.context_window # Sourced(200000, 'catalog') budget.context_window # Sourced(8192, 'discovered') budget.context_window # None, and it stays None Five provenances layer from weakest to strongest (default, catalog, discovered, probed, override), and a weaker source never displaces a stronger one. Only trusted provenance may refuse a request pre-dispatch. The same rule governs money: usage.cost_usd is a Decimal or None, and None means unknown, never zero. See capabilities and provenance. 3. Portability Is a Test Result, Not a Claim 106 providers is inventory, not a feature; the useful part is knowing which of them does what you need. The conformance matrix is generated from real suite runs: every cell is one test case that executed against that adapter, and a ➖ is a declared limitation, not a pass. Providers without rows stay empty rather than turning missing evidence into a claim. Underneath it, 20 contract snapshots record exactly which upstream endpoints, fields, framing, and error shapes each adapter depends on, each dated, with a drift-check procedure that audits them against current provider documentation. Writing your own adapter puts it on the same footing: anyinfer conform runs the suite and emits its matrix row. 4. A Local Model Is a Target, Not a Separate Product client.generate(prompt, target="anthropic:claude-sonnet-4-5") client.generate(prompt, target="llama-cpp:qwen3-8b-q4-k-m") # one string changed If the weights are not there yet, the second call acquires a pinned, hash-verified artifact, picks a runtime for the detected hardware, tunes the launch flags for the memory actually available, starts llama-server on loopback, waits for readiness, serves the request, and evicts the model when idle. No separate daemon to install or operate. An already-running Ollama, LM Studio, or vLLM is equally a target, and both kinds sit in the same fallback chain with the same event stream, usage normalization, and structured-output contract. See run a model locally. 5. Context Fit Is Decided Before You Pay for It budget = client.budget(messages, target="anthropic:claude-sonnet-4-5") budget.remaining_tokens # what is left for context budget.fits # True / False / None; None means the window is unknown reduction = context.select(documents, query, max_tokens=budget.remaining_tokens) reduction.summary() # what was sent, what was dropped, and what bound the decision The same target capabilities drive budgeting, reduction, pre-dispatch refusal, cost estimation, and context-overflow routing, so they all use the same facts. Reduction reports its omissions rather than quietly truncating. See context budgets and context reduction. When a Smaller Tool Is the Better Boundary Provider count is not a reason to add a dependency. Use AnyInfer when the application needs to own a hybrid inference runtime; use the smaller tool when it already solves your whole problem: Your actual requirement Usually the better boundary Call one provider That provider's client or HTTP API Switch among cloud APIs with one Python function A focused provider client such as any-llm or aisuite Centralize credentials, virtual keys, quotas, organization spend, and admin policy A gateway such as LiteLLM, Bifrost, or Portkey Operate a dedicated local-model platform Ollama, LM Studio, or LocalAI Run high-throughput GPU serving infrastructure vLLM or another serving platform Build semantic retrieval over a changing corpus A retrieval or vector-index system; pass its approved results into AnyInfer if you still need the hybrid runtime Ship one application-owned route spanning cloud and a managed local fallback AnyInfer These tools compose. A gateway in front of AnyInfer is a reasonable architecture, and so is AnyInfer calling an Ollama you already operate. AnyInfer earns its place only when removing the boundary between them makes the application simpler or its behavior more reliable. Some things here are support, not differentiators: a long provider list, an OpenAI-compatible sidecar, basic retry and fallback, or calling an already-running local endpoint. Integrators need all of those, and many tools have them. The reason to pick AnyInfer is the runtime and correctness contract around them. Where It Sits Among the Alternatives The columns are categories of tool, not specific products: a category claim can be checked against what the category is for, while a product claim goes stale the week it is written. Snapshot date: 2026-08-09. When choosing against a specific tool, verify that tool's current behavior rather than trusting a generalization here. AnyInfer Provider-switching client Hosted gateway / proxy Local-model server Agent framework What your code holds Typed event stream An OpenAI-shaped response OpenAI wire format OpenAI wire format The framework's abstraction Runs in your process Yes Yes No; a service you operate No; a service you operate Yes Hosted and managed-local in one fallback chain Yes Hosted, usually Across endpoints you already run Local only Whatever its client does Acquires, verifies, and supervises a local model process Yes (llama.cpp) No No Yes; that is its job No Capability provenance, tri-state cost, degradation events Yes Not typically Not typically N/A Varies Structured output validated client-side, with bounded repair Yes Varies Passes the provider's mode through Passes through Commonly yes Test kit, per-adapter conformance matrix, dated contract snapshots Yes Rare Rare N/A Rare Mandatory dependencies 2 Varies N/A N/A Typically many Central keys, org quotas, admin plane No; use a gateway No Yes No No Retrieval / vector index / corpus persistence Opt-in add-on only (anyinfer-store, small-scale) No No No Often yes Prompt templates, chains, agents No No No No Yes; that is its job High-throughput GPU serving No No No Some No The last four rows matter most: AnyInfer is a runtime, not a platform, and the tools in those columns are boundaries to compose with, not competitors. Check the Claims Yourself Nothing on this page requires taking a documentation page's word for it: pip install anyinfer anyinfer init # what this machine can already use, written to a file anyinfer providers --json # every provider and the fields it needs anyinfer verify --config anyinfer.json # does each target actually answer? anyinfer run "..." --dry-run --target ollama:qwen3:8b # cost and fit, nothing spent anyinfer verify sends a real, tiny request, because a credential can be valid for a model listing and useless for inference, and it distinguishes unreachable from reachable but could not hold the requested shape, which need different fixes. Nothing requires an account: the demo app runs entirely offline against in-process fakes, and every code example in this documentation is executed in CI against those same fakes, so none of it can quietly rot. Confidential Execution BYOK inference protects your customer's data from you; it does nothing to protect your prompt templates and orchestration IP from a customer who owns the machine they run on. AnyInfer ships a four-tier ladder for that problem, from encrypted-at-rest templates up to attested execution in a trusted environment with signed-model verification, each tier stating exactly what it does and does not guarantee. See confidentiality tiers. Who This Is For It fits an application that ships: a desktop tool, a developer tool, an offline-capable service, a distributable Python product. That is, something that needs a cloud route and a local route to behave the same way, that has to explain its costs, and whose inference code deserves tests. It does not fit a notebook experiment, a single-provider script, or an organization looking for a central control plane; the table above names better-shaped tools for those. See Also Quickstart: install to first result. Integrate AnyInfer: choosing between the SDK, CLI, and sidecar. Concepts: the eighteen ideas the API follows from. --- # Integrate / Shared Configuration Source: https://anyinfer.dev/reference/configuration/ Shared Configuration AnyInfer has one JSON configuration format for every integration method: load it from Python with load_config, pass it to anyinfer run, or start the OpenAI-compatible sidecar with it; each path is walked through in the guides. Since every deployment reads the same file, provider identity, credentials, endpoint overrides, and the default route do not drift between them. Full API signatures for the loader and the validated config object are in the configuration API. { "format_version": 1, "providers": [ {"id": "anthropic", "api_key": "env://ANTHROPIC_API_KEY"}, {"id": "ollama"} ], "default_route": ["ollama:qwen3:8b", "anthropic:claude-sonnet-4-5"] } The current format_version is 1. Omitting it is accepted for files written before the field was introduced. Unknown versions, unknown keys, duplicate instance ids, invalid types, and files larger than 1 MiB fail with ConfigError before a client is created. The one exception: an entry with enabled: false is skipped without validating its other keys, so a disabled entry can hold settings for a provider that is not installed. The demo app writes a few UI-only fields of its own alongside the shared ones; the SDK, CLI, and sidecar ignore them. See the demo app guide. Provider Settings ai.ProviderSettings.of( "openai", alias=None, # instance id; defaults to the provider id base_url="https://api.openai.com/v1", # provider default when omitted api_key="env://OPENAI_API_KEY", # literal, env://, or credential:// api_version=None, # Azure and Anthropic headers={}, # extra request headers options={}, # adapter-specific settings timeout_s=120.0, # default per-request timeout ) The order providers are listed in is the preference order for alias resolution. Configuring One Engine More Than Once alias gives an entry its own identity, so the same engine can be configured several times; two Azure tenants, a local and a remote Ollama; each with its own endpoint and credentials. The alias is what a target string names: client = ai.Client( [ ai.ProviderSettings.of( "azure-foundry", alias="work", base_url="https://work.openai.azure.com", api_key="env://WORK_KEY", ), ai.ProviderSettings.of( "azure-foundry", alias="lab", base_url="https://lab.openai.azure.com", api_key="env://LAB_KEY", ), ] ) client.generate(messages, target="work:gpt-4o") # the work tenant, not the lab one Each alias becomes its own adapter with its own connection pool. Omitting alias is the ordinary single-instance case, where the provider id is the instance id. Two entries sharing an instance id is a ConfigError, as is an alias that would shadow a registered provider id. Providers That Take a Choice of Credential api_key is the top-level slot for the usual credential. A provider that accepts more than one kind declares each as its own setup field and reads the extra ones from options. Anthropic is the case in point: a console API key and a claude.ai subscription token authenticate with different headers, so they are separate fields rather than two spellings of one. # An Anthropic API key; sent as x-api-key. ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY") # A claude.ai OAuth token; sent as a bearer token, with the beta flag the API requires. # Obtain one with: ant auth print-credentials --access-token ai.ProviderSettings.of("anthropic", options={"oauth_token": "env://ANTHROPIC_OAUTH_TOKEN"}) Supply one or the other; if both are set the OAuth token wins. Option values for fields a provider declares as secret go through the same credential resolver as api_key, so they accept env:// and credential:// references and are registered for redaction. Bedrock's explicit AWS credentials work the same way: ai.ProviderSettings.of( "bedrock", options={ "aws_access_key_id": "AKIA…", # an identifier, passed through "aws_secret_access_key": "env://AWS_SECRET_ACCESS_KEY", # resolved and redacted }, ) Fields a provider declares as anything other than secret are passed through verbatim, since resolving them would corrupt any literal value that merely looked like a reference. To discover what a provider accepts without hardcoding it, read its setup spec; the any_of groups are the ones where one of several fields will do: setup = ai.default_registry.get("anthropic").setup [(f.key, f.required) for f in setup.fields] # declared fields setup.any_of # (('api_key', 'oauth_token'),) setup.requirement_note # why, in one line Which Fields to Actually Ask For Not every declared field is a question. A provider knows its own endpoint, its API version, and where AWS keeps its credentials; what it cannot know is the developer's key or account. The spec draws that line itself, so a setup form does not have to infer it from help text: setup = ai.default_registry.get("openai").setup [f.key for f in setup.essential_fields] # ['api_key'] ; ask for these [f.key for f in setup.advanced_fields] # ['base_url']; offer these, folded away An advanced field is never required and never part of an any_of group, so a form built from essential_fields alone can always be saved. Each one carries its fallback in SetupField.default_value (https://api.openai.com/v1 here), which lets a collapsed field still say what it will do. Render that value rather than pre-filling the editor with it: a saved copy of today's default keeps overriding the real default after it has moved on. The two extremes: ollama, vllm, and the other local engines have no essential fields at all, while azure-foundry, runpod, and anything else whose URL embeds an account or endpoint id keeps base_url essential, because no default could be right. Client Settings ai.Client( providers, registry=None, # defaults to the process-wide registry catalog=None, # defaults to the bundled catalog route=None, # default route when a call names no target observers=[], # telemetry sinks, registered payload-free resolver=None, # credential resolver chain retain_raw=False, # keep raw provider payloads on results repair=None, # default repair budget use_default_catalog=True, # False disables alias resolution entirely estimator=None, # token counting; defaults to the byte heuristic context_gate=True, # refuse requests that provably cannot fit pre-dispatch history=None, # conversation compaction when a request overflows arena=None, # default fixed-target arena policy arenas={}, # named arena policies for CLI/sidecar model strings pricing_table=None, # defaults to the bundled table; see fetch_pricing() capability_overrides=None, # "provider:model"-keyed corrections, strongest layer model_dir=None, # where acquired model weights are stored ) retain_raw is off by default because raw payloads carry response text that payload-free telemetry omits. Per-Request Options client.generate( messages, target="medium", # or route= schema=None, tools=(), tool_choice="auto", # "auto" | "none" | "required" | a tool name sampling=ai.Sampling(...), reasoning=None, # "minimal" | "low" | "medium" | "high" timeout_s=None, # per attempt; defaults to 120 repair=None, provider_options={}, # namespaced escape hatch metadata={}, # opaque, echoed in telemetry max_response_bytes=1_048_576, arena=None, # fixed targets and post-run selection context=None, # caller-approved stateless corpus reduction ) Sampling fields default to None, meaning provider default. AnyInfer never invents a temperature; an unset value is omitted from the wire request entirely. Environment Variables Variable Effect ANYINFER_MODEL_DIR Override where downloaded models are stored (also Client(model_dir=...)). ANYINFER_RUNTIME_DIR Override where llama.cpp runtime variants are installed. ANYINFER_HARDWARE_CACHE_BYPASS Skip the hardware cache entirely, read and write. ANYINFER_HARDWARE_CACHE_REFRESH Ignore a cached profile, re-probe, and rewrite it. ANYINFER_SERVE_TOKEN Bearer token for anyinfer serve. COPILOT_CLI_PATH Override Copilot CLI discovery. Credential references (env://NAME) read any variable the reference names; there are no magic credential variable names. Generating a File anyinfer init writes a valid configuration from what the machine can already do (running loopback engines and credential variables that are actually set), plus a runnable starter program beside it. See the CLI guide. In Python, the same format is written by anyinfer.dumps_config and anyinfer.dump_config: import anyinfer as ai config = ai.AnyInferConfig( providers=(ai.ProviderSettings.of("openai", api_key="env://OPENAI_API_KEY"),), route=ai.Route(targets=("openai:gpt-5",)), ) ai.dump_config(config, "anyinfer.json") # refuses to replace an existing file text = ai.dumps_config(config, comments=True) # the same JSON, with a leading note Round-tripping is the contract: loads_config(dumps_config(c)) == c for every configuration the loader accepts. Two consequences follow: An opt-in policy left entirely at its defaults still writes its block, as {}. The block's presence is what asks for the policy; dropping it would turn "pace this provider by whatever it reports" back into "do not pace it at all". comments=True writes a _comment string at the root rather than // lines. The format is JSON, so a generated file explains itself in something the loader accepts; reading it back changes nothing. The writer emits credential settings exactly as they were given and never resolves a reference. An env:// or credential:// value therefore stays safe to store, while a literal credential stays literal. Review configurations constructed programmatically with literal secrets before writing or committing them; anyinfer init itself writes references. File Format Each provider entry needs an id. The other top-level provider settings are adapter, base_url, api_key, api_version, headers, timeout_s, and options. Setup fields declared by that provider may also be written directly, or grouped under a values object (the shape setup UIs write); unrecognized fields fail validation. Two more keys exist for compatibility: provider_id is the legacy spelling of adapter, and alias, when present, must simply restate the entry's id. Credential references such as env://ANTHROPIC_API_KEY are resolved only when the adapter is first used, so parsing a config never prints or expands a secret. The adapter Key id is the instance id used in target strings. The optional adapter key names the engine behind it, which is what lets one engine be configured more than once: { "providers": [ {"id": "openai", "api_key": "env://OPENAI_API_KEY"}, { "id": "work-azure", "adapter": "azure-foundry", "base_url": "https://wumbo.openai.azure.com", "api_key": "env://WUMBO_KEY" }, { "id": "ollama-local", "adapter": "ollama", "base_url": "http://127.0.0.1:11434" } ], "default_route": ["openai:gpt-5", "work-azure:gpt-4o"] } Omitting adapter keeps the single-instance spelling exactly as before: the id is both the engine selector and the instance id. A duplicate id fails fast with a ConfigError. The sidecar can advertise instance-scoped targets from /v1/models by writing them in instance terms: anyinfer serve --config anyinfer.json --expose work-azure:gpt-4o Provider limits Request pacing is configured per provider instance because two accounts at the same provider have independent allowances: { "providers": [ { "id": "openai", "limits": { "max_concurrent": 4, "requests_per_minute": 120, "min_interval_s": 0.1, "respect_headers": true, "reserve_fraction": 0.1 } } ] } Omitting limits disables pacing. An empty object opts into provider-reported rate-limit headers without imposing a local fixed limit. Values are validated by RateLimits; unknown keys and nonpositive or out-of-range values fail during configuration loading. How pacing behaves at run time is covered in pacing before the limit. The context Block Advanced settings for context reduction, parsed into a ContextTuning. Every key is a field of that record, so a setting is spelled the same way in the file, on the command line, and in Python: { "context": { "selection_order": "density", "diversity": 0.25, "split_identifiers": true, "query_expansion": true, "near_duplicate_threshold": 0.9, "compact_fallback": true, "salience_weight": 0.5, "carry_over_bonus": 0.5 } } The block is optional, and every field defaults to the behavior AnyInfer has always had; a file without it reduces exactly as before. The values above are what ContextTuning.recommended() sets, which anyinfer context --preset recommended applies without a file. Read it back with config.context and pass it straight through: config = ai.load_config("anyinfer.json") reduction = context.select(docs, query, max_tokens=8_000, tuning=config.context) A misspelled setting is a ConfigError, not a silent no-op; a tuning key that silently does nothing is worse than one that fails loudly. The sidecar reads the same file so one config serves every frontend, but it does not reduce context itself: it is a wire codec over a normal client, and reduction is the application's call about its own material. The full field list is on ContextTuning. The history Block Conversation compaction, applied by the client when a request outgrows its target's window. Because it is a client setting rather than a frontend one, this block makes the SDK, anyinfer run, and the sidecar behave identically: { "history": {"mode": "last_resort", "keep_recent": 6, "keep_system": true} } last_resort compacts only after the route's context_window_targets chain is exhausted, so a larger-window model is always preferred to losing history. proactive compacts to fit the resolved target before dispatch, which avoids a refused preflight but never reaches a larger-window target further down the route. Omitting the block means no compaction: an oversized request is rerouted or fails, exactly as before. Set "enabled": false to keep a tuned block switched off. Sidecar callers can override it per request with the anyinfer_history field; see the sidecar. The cache Block Prompt-cache placement is opt-in because it changes provider billing and retention: { "cache": { "mode": "auto", "min_segment_tokens": 1024, "max_marks": 4, "include_tools": true, "include_system": true } } Omitting the block disables placement. An empty object enables the default CachePolicy; auto chooses the strongest mechanism the resolved target offers. See prompt caching for the mechanism and billing semantics. The operation_routes Block Embedding and reranking calls get their own default routes, so a client configured for chat fallback never accidentally embeds with it: { "default_route": ["anthropic:claude-sonnet-4-5", "ollama:qwen3:8b"], "operation_routes": { "embedding": ["cohere:embed-v4.0", "ollama:nomic-embed-text"], "rerank": ["cohere:rerank-v3.5"] } } Valid keys are embedding and rerank; the generation default belongs in default_route, and the loader rejects a generation key here so the two can never be confused. embed() and rerank() use the matching entry when the caller names no target; an explicit target= or route= argument always wins. With no entry configured, they fall through to default_route, whose targets must actually declare the operation or the call is refused before dispatch. Note that an embedding fallback chain is still held to the embedding-space safety rule: targets that are not the identical provider:model are refused unless the caller opts in; see Embeddings and reranking. The arena and arenas Blocks Arena policies fan one request out to fixed targets and select only after the candidates finish. A default policy and named policies use the same complete field set: { "arena": { "targets": ["openai:gpt-5-mini", "anthropic:claude-haiku-4-5"], "strategy": "first_valid", "concurrency": 2, "min_candidates": 1, "reveal_targets": false, "memoize_tools": "read_only" }, "arenas": { "review-panel": { "targets": ["openai:gpt-5-mini", "anthropic:claude-haiku-4-5"], "strategy": "judge", "judge_target": "openai:gpt-5-mini", "instructions": "Choose the most precise supported answer." } } } Unknown keys fail validation. anyinfer run --arena-name review-panel and a sidecar model string of review-panel resolve the named policy without moving orchestration into either frontend. See Arena runs for cost ceilings, selection rules, tool loops, and the response evidence envelope. The mcp Block MCP entries are inert server descriptions. Loading the file never starts a subprocess or opens a connection: { "mcp": [ { "name": "files", "command": ["mcp-server-filesystem", "./docs"], "env": {"MCP_TOKEN": "env://MCP_TOKEN"}, "deny_tools": ["write_file"] }, { "name": "search", "url": "https://tools.example.invalid/mcp", "headers": {"authorization": "env://SEARCH_MCP_TOKEN"}, "timeout_s": 15, "allow_tools": ["search"] } ] } Every entry needs a unique name and exactly one of command or url. command, allow_tools, and deny_tools are string lists; env and headers map non-empty strings to strings; and timeout_s must be positive. Credential references in both env and headers resolve only when MCPToolset.connect() is called and are registered for redaction. See the tool-loop guide for discovery, trust boundaries, and the intentionally unsupported MCP surfaces. CLI anyinfer init [--output PATH] [--force] # write this file from what is available anyinfer serve --host 127.0.0.1 --port 8080 --config anyinfer.json anyinfer serve --token SECRET --host 0.0.0.0 --allow-remote-exposure anyinfer serve install --print # the service definition for this platform; writes nothing anyinfer run "PROMPT" --config anyinfer.json # one prompt, then exit anyinfer doctor [--json] # detected hardware, recommended tier anyinfer providers [--json] # every registered provider and what it needs anyinfer agents-md >> AGENTS.md # coding-agent instructions for this version anyinfer context src/ --query "how does auth work?" --max-tokens 8000 anyinfer context src/ --query "…" --max-tokens 8000 --plan # cost every strategy run reads the same config file as serve, so one file drives both. See run a prompt from the shell for its flags. A non-loopback bind requires both --allow-remote-exposure and a token. The CLI refuses otherwise, since an unauthenticated gateway would let anyone on the network spend the configured provider credentials. Cache and Data Locations Purpose Windows macOS Linux Hardware cache %LOCALAPPDATA%\anyinfer ~/Library/Caches/anyinfer $XDG_CACHE_HOME/anyinfer Model artifacts %LOCALAPPDATA%\anyinfer\models ~/Library/Application Support/anyinfer/models $XDG_DATA_HOME/anyinfer/models Override the model directory with options={"model_dir": Path(...)} on the llama-cpp provider. --- # Integrate / Python SDK Source: https://anyinfer.dev/guides/python-sdk/ Integrate the Python SDK Use the SDK when AnyInfer runs inside a Python application. Quickstart is the fastest path to a first result; this page is the reference for embedding the SDK properly: the client lifecycle and the error handling a long-lived application needs. Configure the Client For deployed applications, keep provider identity and routing in the shared configuration file: import anyinfer as ai config = ai.load_config("anyinfer.json") with ai.Client(config.providers, route=config.route) as client: result = client.generate("Give me a two-sentence status summary.") print(result.text) For a small script, construct the same settings directly: providers = [ ai.ProviderSettings.of( "anthropic", api_key="env://ANTHROPIC_API_KEY", ) ] with ai.Client(providers) as client: result = client.generate("Hello", target="anthropic:claude-sonnet-4-5") Credential references are resolved when an adapter is first used and registered for redaction. Prefer env:// or credential:// references to literals in source code and configuration files. One Client, Reused, Then Closed AsyncClient is the native implementation. Client is its thread-safe synchronous facade; both accept the same arguments and return the same domain types. AsyncSync async with ai.AsyncClient(config.providers, route=config.route) as client: result = await client.generate("Explain the result.") with ai.Client(config.providers, route=config.route) as client: result = client.generate("Explain the result.") Choose AsyncClient inside an async application and Client in a synchronous one. Create one client and reuse it; do not create one per request. Since a client owns connection pools and any supervised local servers, close it with a context manager or an explicit close()/aclose() call. One client serves many conversations; continuity across turns is a session concern, not a client-lifecycle one. generate() returns the finished result; to consume events as they arrive, see streaming. Handle Failures All public failures derive from AnyInferError and carry structured fields. Branch on those fields when behavior matters; show hint to the operator: try: result = client.generate("Hello", target="medium") except ai.AnyInferError as exc: logger.error("generation failed during %s: %s", exc.phase, exc) if exc.hint: logger.info("next step: %s", exc.hint) The error catalog lists every exception, when it is raised, and what the user will see. Key Takeaways AsyncClient is the native implementation; Client is its thread-safe synchronous facade over the same surface. Create one client, reuse it, and close it: it owns connection pools and any supervised local servers. Catch AnyInferError, branch on its structured fields, and surface hint. See Also Quickstart: from pip install to a working result. Stream typed events: consuming the event stream. Sessions: continuity across conversation turns. Clients and streams: the full client API. Error catalog: every exception and its fields. --- # Integrate / Command-Line Tool Source: https://anyinfer.dev/guides/cli/ Run a Prompt from the Shell anyinfer run sends one prompt through the same path the library uses (routing, fallback, structured output, telemetry) and then exits. It is the shell-shaped way to reach everything AnyInfer abstracts, without writing a script or starting a server. anyinfer run "Explain TCP slow start." --config anyinfer.json --target ollama:qwen3:8b The reply streams to stdout as it arrives. Getting a Config File in the First Place anyinfer init writes one from what this machine can already do, so the first five minutes end in a working call rather than in the configuration reference: anyinfer init detected Linux / x86_64, 32.0 GiB RAM, NVIDIA RTX 4070 (12.0 GiB) probed 17 loopback endpoint(s), every one a provider default: http://127.0.0.1:11434, http://127.0.0.1:1234/v1, … found ollama at http://127.0.0.1:11434 (4 models) found anthropic, credential env://ANTHROPIC_API_KEY recommend medium -> ollama:qwen3:8b wrote anyinfer.json wrote starter.py next python starter.py anyinfer verify --config anyinfer.json It discovers rather than guesses: a provider reaches the file only when a loopback endpoint it declares answered a model listing, or a credential variable it names is set. Detected keys are written as env:// references, never values, so the generated file is safe to commit, which init says once and then leaves your .gitignore alone. Flag What it does --output PATH Write the configuration somewhere other than anyinfer.json --force Replace an existing configuration and starter --no-probe Contact nothing; report credential evidence only --keyring Also look in the OS credential vault (may prompt to unlock) -y, --yes Do not ask before writing, on a terminal --json Emit the findings and the decisions for a script anyinfer doctor reports the same hardware without writing anything, and points here when no configuration exists yet. Instructions for a Coding Agent anyinfer agents-md prints a fragment describing how the installed version of this library is called, ready to append to a repository's AGENTS.md or CLAUDE.md. The command, its flags, and what the fragment contains are covered in coding agents. Pointing It at Providers run reads the same shared config file the Python SDK and sidecar use, so one file drives all three: { "providers": [ { "id": "ollama" }, { "id": "anthropic", "api_key": "env://ANTHROPIC_API_KEY" } ], "default_route": ["ollama:qwen3:8b", "anthropic:claude-sonnet-4-5"] } With a default_route configured, --target becomes optional: anyinfer run "Summarize the CAP theorem." --config anyinfer.json anyinfer providers lists every registered provider and the fields each one needs. Where the Prompt Comes From The prompt can be an argument, piped on stdin, or both; stdin is appended, which makes the usual Unix shapes work: anyinfer run "Say hello." < /dev/null # argument only cat notes.txt | anyinfer run # stdin only cat notes.txt | anyinfer run "Summarize this:" # instruction, then body Add a system prompt with --system, or continue a conversation with --messages, a JSON file of {"role", "content"} objects: anyinfer run "And in one sentence?" --messages history.json --config anyinfer.json Output Modes By default the text streams to stdout and nothing else does, so run composes: anyinfer run "Name three primes." --config anyinfer.json > primes.txt Flag Effect (default) Streams text to stdout as it is generated. --no-stream Waits for the whole reply, then prints it. --json Prints one object with the text, usage, timing, tool calls, and warnings. --stats Prints timing, token, and cost figures to stderr, leaving stdout clean. --show-reasoning Prints reasoning deltas to stderr, on models that emit them. Enforcing a JSON Schema Point --schema at a JSON Schema file and the reply is validated before you see it, using the strongest mechanism the provider offers. Output is the validated JSON: echo '{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}' > city.json anyinfer run "Which city hosted the 2004 Olympics?" \ --config anyinfer.json --schema city.json A reply that will not validate raises an error; allow bounded retries with --repair N: anyinfer run "..." --config anyinfer.json --schema city.json --repair 2 Schema mode implies --no-stream: a JSON document cannot be validated until it is complete. Attach Images, Documents, and Audio run collects attachment files at the CLI boundary and sends typed multimodal parts through the same client path as the SDK and sidecar: anyinfer run "Summarize these inputs" --image diagram.png --document report.pdf --audio note.wav Repeat a flag to attach multiple files. MIME types are inferred from filenames, inline payload ceilings are checked before dispatch, and an adapter that cannot represent a part fails explicitly instead of dropping it; see multimodal inputs for provider coverage. Compare Fixed Targets with an Arena anyinfer run "Classify this" --schema schema.json \ --arena openai:gpt-5-mini,anthropic:claude-haiku-4-5 \ --arena-strategy consensus --stats --dry-run reports the arena call ceiling and summed cost range while making zero provider calls. Named policies from the shared configuration use --arena-name. See arena runs for selection and tool-loop semantics. Declaring Tools --tool takes a JSON file declaring one tool, and is repeatable: { "name": "get_weather", "description": "Look up the current weather for a city.", "parameters": { "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] } } anyinfer run "What is the weather in Boston?" \ --config anyinfer.json --tool get_weather.json run never executes tools. It reports what the model asked for (on stderr, or in the tool_calls array under --json) and leaves the calling to you; for an automated call-and-respond cycle, use the tool loop in a script. --tool-choice requires or forbids tool use (auto, none, required). Routing and Fallback --route names an ordered fallback chain and is repeatable. The first target that succeeds wins, so a local-first setup with a hosted backstop is one line: anyinfer run "Draft a commit message." --config anyinfer.json \ --route ollama:qwen3:8b --route anthropic:claude-sonnet-4-5 --route overrides --target, since naming an ordered list is the more specific instruction. Sampling and Limits anyinfer run "Write a haiku about latency." --config anyinfer.json \ --temperature 0.9 --max-tokens 60 --stop "---" --timeout 30 --reasoning (minimal, low, medium, high) sets reasoning effort on models that expose it. Parameters a provider or model does not support are dropped rather than rejected, since a parameter that does nothing is the failure mode that looks exactly like success. Every drop is reported as a warning. Costing a Request Before You Send It --dry-run reports what a request would spend and whether it fits, using the same budget calculator the client holds the real request to: cat report.md | anyinfer run "Summarize:" --config anyinfer.json \ --target openai:gpt-4.1 --dry-run target openai:gpt-4.1 input estimate 18432 tokens (floor 6912) messages 18401 schema 31 context window 128000 (catalog) output reserve 4096 input allowance 115712 remaining 97280 fits yes estimated cost 0.0138-0.0697 USD Nothing is sent, and an unknown figure prints unknown, never a plausible default. --json emits the same information for scripts. Embedding and Reranking anyinfer embed and anyinfer rerank are the operation counterparts of run: $ anyinfer embed "how does retry backoff work" --target cohere:embed-v4.0 --json $ anyinfer embed --file corpus.txt --target ollama:nomic-embed-text --out vectors.json $ anyinfer rerank "which doc covers backoff" --file docs.txt --top-n 3 --target cohere:rerank-v3.5 Plain output prints a one-line summary to stderr (vector counts, never thousands of floats); the full vectors only appear with --json or --out. Inputs come from a positional argument, --file (newline-delimited), --jsonl, or stdin. Both commands accept --trace / --trace-json for the run manifest: $ anyinfer embed "hello" --target cohere:embed-v4.0 --trace-json | jq .operation "embedding" Requests larger than the target's verified batch limit are split and re-assembled by the core. A configured operation_routes block supplies the default target when --target is omitted (see the configuration reference). Checking a Target Actually Works anyinfer verify sends one tiny real request and reports what came back: the thing a health check cannot tell you, since a credential can be valid for a model listing and useless for inference. It is the CLI face of verify(). anyinfer verify ollama:qwen3:8b --config anyinfer.json anyinfer verify cohere:embed-v4.0 --operation embedding --config anyinfer.json --operation embedding (or rerank) proves the operation the same way, judged on the vector or ranking that came back instead of a chat reply. ok ollama:qwen3:8b 412 ms, schema via grammar With no target it checks every target in the configured route, exiting non-zero if any failed, so it works as a setup gate: anyinfer verify --config anyinfer.json || { echo "fix your config first"; exit 1; } Failures distinguish unreachable from reachable but wrong, which need different fixes: FAILED openai:gpt-5 401 unauthorized (check the api_key for this provider) answered ollama:qwen3:0.6b the provider answered, but not in the requested shape: response was not JSON --json emits the same information for scripts, including anything the provider reported about its own runtime. A target known to reason gets a larger probe budget than the ordinary 64 tokens: a thinking model spends a small budget on reasoning before it says anything, and the truncated result would read as an empty answer — a connection failure you do not have. Fitting a Directory into a Prompt anyinfer context collects files, reduces them to a budget, and prints the envelope. Walking a filesystem and deciding what is safe to send is an application's job; the library only reduces what it is handed. anyinfer context src/ --query "how does credential resolution work?" --max-tokens 8000 The envelope goes to stdout so it can be piped; the account of what was dropped goes to stderr: tiered: 46 of 340 document(s); ~7900 of 8000 tokens; 12 collapsed; 282 omitted; limited by tokens Vendored, generated, binary, and oversized files are skipped; pass --include-generated to offer them anyway, and --pin PATH to force a file in ahead of the ranked candidates. Give the budget with --max-tokens, or with --target to take it from that model's context window. An unknown window is refused rather than guessed: the context window of 'openai-compat:mystery' is unknown, so there is no budget to reduce against; pass --max-tokens to choose one yourself --plan runs every deterministic strategy against the corpus and reports what each would produce, spending no inference; plan before you commit walks through reading its table. Tuning Every advanced setting has a flag, and they read the context block of --config as their baseline: anyinfer context src/ --query "…" --max-tokens 8000 --preset recommended anyinfer context src/ --query "…" --max-tokens 8000 \ --context-selection-order density --context-diversity 0.3 --context-query-expansion Precedence is config file, then --preset, then individual flags; boolean settings take a --no- form to turn off what the file or preset turned on. --json prints the machine-readable record instead of the envelope, for both modes. Exit Codes Code Meaning 0 The request succeeded. 1 The request failed; the error and its hint are on stderr. For verify, at least one target did not pass. 2 The command was used incorrectly; no prompt, no providers, bad flags. 130 Interrupted with Ctrl-C. Key Takeaways anyinfer init writes a configuration from discovered evidence, with keys as env:// references, so the generated file is safe to commit. run composes: reply text on stdout, --stats on stderr, --json for scripts, and --dry-run to price a request without sending it. anyinfer verify spends one tiny real request per target, distinguishes unreachable from reachable-but-wrong, and exits non-zero on failure. run reports tool calls but never executes them; automated cycles belong to the tool loop in a script. See Also The sidecar: the long-running OpenAI-compatible service Shared configuration Enforce a JSON schema Run the tool loop Fit a corpus to a context budget --- # Integrate / OpenAI-Compatible Sidecar Source: https://anyinfer.dev/serve/ OpenAI-Compatible Sidecar An OpenAI-compatible projection of the same configured hybrid runtime used by the Python SDK and command-line tool. It allows existing clients to use hosted, hub, and local routes without creating a second routing or configuration system. sequenceDiagram participant O as OpenAI client participant C as Codec participant R as Router participant A as Adapter O->>C: chat.completion C->>R: GenerationRequest R->>A: route A-->>R: StreamEvent R-->>C: StreamEvent C-->>O: chat.completion.chunk Python installationStandalone bundle pip install "anyinfer[serve]" anyinfer serve --config anyinfer.json Download the platform's bundle from Downloads, unzip it, then run: anyinfer-serve --config anyinfer.json The standalone build includes the frontend and built-in dependency-free adapters. Use the Python installation when a provider requires an optional SDK or authentication extra, such as GitHub Copilot or Azure Entra authentication. from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused") client.chat.completions.create( model="ollama:qwen3:8b", # or "medium", or "anthropic:..." messages=[{"role": "user", "content": "hi"}], ) Since the internal primitive is a normalized event stream that adapters already project provider dialects into, the frontend is only the inverse projection at the edge: a wire codec plus an ASGI app around a normal AsyncClient. No routing, validation, telemetry, credential, or local-inference code is duplicated, and an architecture test enforces that the frontend stays a codec. What It Serves Endpoint Behavior POST /v1/chat/completions Streaming and non-streaming. model is parsed as a target. GET /v1/models Catalog aliases plus any explicitly exposed targets. GET /health Liveness. Requires no authentication. Anything else under /v1 404 with a clear explanation. Embeddings and generated image/audio outputs are out of scope. Typed image, document, and audio inputs are accepted in OpenAI content arrays and capability-gated before dispatch; see multimodal inputs. Model Strings Are Targets Every target spelling works in the model field, which is what makes federation free: {"model": "medium"} {"model": "anthropic:claude-sonnet-4-5"} {"model": "ollama:qwen3:8b"} {"model": "llama-cpp:qwen2.5-7b-instruct-q4-k-m"} A round-trip test enforces that no target spelling can carry structure an OpenAI model field cannot. What Survives the Wire, and What Does Not Survives: text and multimodal message parts, tools, tool_choice, response_format.json_schema, temperature, top-p, max tokens, stop sequences, the stream flag, usage, and finish reasons. Unrecognized extra-body fields reach provider_options, so the escape hatch survives too. Does not in the stock shape: timing marks and attempt records. They have no chat.completion.chunk representation. An AnyInfer-aware caller can request the complete run manifest without changing what a stock OpenAI client receives: { "model": "medium", "messages": [{"role": "user", "content": "hi"}], "anyinfer_manifest": true } For a buffered response, anyinfer_manifest is a top-level response property. For a stream, it is one terminal SSE frame immediately before [DONE]. Absence of the request field means absence of both response forms. See run manifests. Security Binds 127.0.0.1 by default. A non-loopback bind requires both --allow-remote-exposure and a bearer token. An unauthenticated LLM gateway on a network is a credential laundering service. Backend credentials never transit: the frontend authenticates clients to itself. Standard redaction applies to logs; payload retention is off by default. There are no configuration or execution endpoints of any kind, so a captured token can spend inference, but it cannot rewrite routing or run code. BashPowerShell export ANYINFER_SERVE_TOKEN=$(openssl rand -hex 32) anyinfer serve --host 0.0.0.0 --allow-remote-exposure $env:ANYINFER_SERVE_TOKEN = -join ((48..57)+(97..102)|Get-Random -Count 64|%{[char]$_}) anyinfer serve --host 0.0.0.0 --allow-remote-exposure Keeping It Running anyinfer serve install writes the systemd unit, launchd agent, or scheduled task that keeps the sidecar up across logins and reboots, after first showing the exact file and commands. See running as a service. Embedding It create_app returns a plain ASGI app, mountable in an existing Starlette or FastAPI stack: from anyinfer.serve import create_app app = create_app(async_client, auth_token=token, expose_targets=("anthropic:claude-sonnet-4-5",)) Behind a Proxy The app sets X-Accel-Buffering: no on streaming responses. Without it, reverse proxies buffer the whole response and streaming silently stops being streaming (a failure that only appears in deployment, never in local testing). Oversized Conversations The sidecar applies whatever context policy its client was built with, since it is a codec over a normal client rather than a second core. Give the shared config a history block and a conversation that outgrows its target's window is compacted instead of refused, with the same rules and the same ContextReduced telemetry an SDK caller gets; context reduction covers the rules themselves. A caller with a different tolerance can say so per request. The request body is a documented superset of OpenAI chat completions, and this is what that superset is for: { "model": "openai:gpt-4o", "messages": [...], "anyinfer_history": {"mode": "proactive", "keep_recent": 2} } false refuses compaction for that request and returns the overflow error instead of a shortened conversation; true accepts the defaults. A malformed value is a 400 rather than a silent fallback to the gateway's setting. Reducing an Explicit Corpus An application can also send an explicit, caller-approved corpus with a request: { "model": "openai:gpt-5-mini", "messages": [{"role": "user", "content": "Where is token refresh handled?"}], "anyinfer_context": { "documents": [ {"path": "src/auth.py", "content": "...", "pinned": true}, {"path": "src/session.py", "content": "..."} ], "query": "token refresh", "strategy": "ranked", "max_tokens": 6000, "placement": "system" } } anyinfer_context is stateless: every request carries the documents, the sidecar stores none of them and never collects files, and reduction is delegated to the normal core client. Default ceilings hold the envelope to 1,000 documents and 5 MiB. The inference-spending distill strategy is refused here, and a request needs either a trusted target context window or an explicit max_tokens budget, so the reducer always has a real ceiling to fit against. The response reports selected and omitted counts without echoing paths or content; a stream carries the same summary in its terminal extension frame. Since uploading material the server then omits wastes bandwidth, a large local corpus is better served by anyinfer context or anyinfer run --context-dir running beside the files. Context reduction explains the strategies; fit a corpus to a context budget covers choosing one. Fixed-Target Arena Requests An AnyInfer-aware caller can add anyinfer_arena with the complete ArenaPolicy field set, or use a configured arena name as model. The response stays a valid single-choice OpenAI completion with candidate evidence added under the same extension name; streams buffer the branches and emit only the winner, so candidate events never interleave. Strategies, spend reservation, and tool-loop behavior are covered in arena runs. Configuration The sidecar, CLI, and Python SDK use the same shared configuration file. Key Takeaways The sidecar is a wire codec over the same client the SDK uses; routing, credentials, telemetry, and context policy come from the shared configuration, not a second system. Any target spelling works as the model field, so hosted, hub, and local routes are all reachable from a stock OpenAI client. The AnyInfer extensions (anyinfer_manifest, anyinfer_history, anyinfer_context, anyinfer_arena) are additive: a client that does not send them receives a plain OpenAI completion. A non-loopback bind requires both --allow-remote-exposure and a bearer token, and backend credentials never transit the frontend. See Also Integrate AnyInfer Run a prompt from the shell: the same config file, one prompt, no server Shared configuration Running as a service: surviving a reboot --- # Integrate / Running the Sidecar as a Service Source: https://anyinfer.dev/serve/running-as-a-service/ Running the Sidecar as a Service An application pointing at http://127.0.0.1:8080/v1 needs that endpoint to exist at boot, not just while somebody keeps a terminal window open. anyinfer serve install writes the systemd unit, launchd agent, or scheduled task that arranges it, and shows the file first. The service runs from the same shared configuration file as the CLI and SDK. anyinfer serve install --print --config anyinfer.json # see it, write nothing anyinfer serve install --config anyinfer.json # write it and register it anyinfer serve status # read-only anyinfer serve uninstall The standalone download works the same way: anyinfer-serve install. Its archive also ships an INSTALL.txt rendered from the same templates, so the download and the command cannot describe different definitions. What It Will and Will Not Do It prints before it writes. Every path shows the exact file and the exact commands, and asks for confirmation unless --yes is passed. User scope by default. systemctl --user, a LaunchAgent, and a per-user scheduled task all install without privileges. --system generates the system-wide definition and prints the commands to run as root; it will not elevate on its own. It never overwrites silently. An existing definition stops the command; --force replaces it. uninstall removes what install wrote: the definition and, where one exists, the private environment file. status is read-only. It reports whether a definition exists and what the platform's manager says about it. It never starts, stops, or restarts anything: that is the manager's job, and a command that issued control verbs would have become the process supervisor AnyInfer is not. After a successful install the command runs anyinfer verify against the configured route, unless --no-verify is passed. A service that starts cleanly and then fails every request at 3am because a credential reference is wrong is the failure this catches. A failure is reported; nothing is uninstalled. What Gets Generated Linux (systemd)macOS (launchd)Windows (scheduled task) ~/.config/systemd/user/anyinfer-serve.service, or /etc/systemd/system/ with --system. [Service] Type=simple ExecStart=/usr/local/bin/anyinfer serve --host 127.0.0.1 --port 8080 --config /srv/anyinfer.json EnvironmentFile=-/home/you/.config/anyinfer/serve.env Restart=on-failure NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=read-only RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX The hardening directives are on rather than offered. This process reads a configuration file and speaks HTTP; it has no business writing to the filesystem, gaining privileges, or opening anything but IP sockets, and a default that must be turned on is a default nobody has. Logs go to the journal: journalctl --user -u anyinfer-serve.service. ~/Library/LaunchAgents/dev.anyinfer.serve.plist, with RunAtLoad, KeepAlive, and a ten-second ThrottleInterval so a crash loop does not spin. When a bearer token is configured the agent runs through /bin/sh to source the private environment file. launchd has no EnvironmentFile, and its EnvironmentVariables dictionary lives in the plist itself, which is exactly where a token must not go. %LOCALAPPDATA%\AnyInfer\anyinfer-serve.xml, registered with schtasks to run at logon. A logon task, not a Windows Service: the sidecar is a console executable, and a service shim would be a second supervisor. For boot-time start, use a third-party service wrapper; that wrapper is then the operator's to maintain. Tokens and Exposure A loopback service needs no token. Everything below applies only when one is exposed. The token never enters the definition. On Linux and macOS it goes to a mode-0600 environment file beside the unit, read as ANYINFER_SERVE_TOKEN. The file is created at that mode rather than tightened afterwards. On Windows no token file is written at all. A POSIX file mode means little there, so the command says to set the variable where the OS already guards it: setx ANYINFER_SERVE_TOKEN . The task inherits it at logon. A non-loopback definition cannot be generated without both --allow-remote-exposure and a token. The running server enforces that already; generation enforces it too, so a unit that would survive reboots as an unauthenticated gateway cannot be produced at all. See the security guidance for what exposure means. Printed output passes through redaction, so a token cannot reach the terminal or its scrollback. Logs Nothing is redirected by default: systemd has the journal and launchd has os_log, and duplicating them into a file helps nobody. The Windows task has no sink, so --log-file writes one, and AnyInfer does not rotate it. The generated definition states that, and so does this page, so the file's growth never comes as a surprise. When the Executable Moves The generated definition names an absolute path: the console script, or the standalone executable under a frozen build. If the path lies inside a temporary or extraction directory the command refuses and explains: once that directory is cleaned up, the service fails at the next boot. Unpack the download somewhere permanent and rerun. After upgrading or relocating, regenerate: anyinfer serve install --force. Key Takeaways anyinfer serve install shows the exact definition before writing it, installs at user scope by default, and runs anyinfer verify against the configured route afterwards. status reports and never controls; starting and stopping belong to the platform's service manager. Exposure rules survive into the definition: no non-loopback unit can be generated without both --allow-remote-exposure and a bearer token, and the token lives in a mode-0600 environment file (or the OS environment on Windows), never in the definition itself. Regenerate with --force after the executable moves or is upgraded. --- # Integrate / Stream Typed Events Source: https://anyinfer.dev/guides/streaming/ Stream to a Terminal A stream yields typed events while the request runs: text deltas, reasoning, timing marks, and attempt failures. This page shows the patterns a terminal frontend needs; the full event vocabulary is in the API reference. SyncAsync import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("ollama")]) with client.stream("Explain TCP slow start.", target="ollama:qwen3:8b") as stream: for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) result = stream.result print( f"\n\n{result.usage.output_tokens} tokens, " f"first token in {result.timing.first_token_ms:.0f} ms" ) import anyinfer as ai async with ai.AsyncClient([ai.ProviderSettings.of("ollama")]) as client: async with client.stream("Explain TCP slow start.", target="ollama:qwen3:8b") as stream: async for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) result = stream.result print( f"\n\n{result.usage.output_tokens} tokens, " f"first token in {result.timing.first_token_ms:.0f} ms" ) Use the Context Manager Leaving the block early cancels the in-flight request. Without it, an abandoned stream keeps generating — and, on a hosted provider, keeps billing: with client.stream(prompt, target=target) as stream: for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) if user_pressed_escape(): break # the request is cancelled on the way out Show Thinking Separately Reasoning models emit a separate channel, excluded from the answer text: for event in stream: match event: case ai.ReasoningDelta(text=t): print(dim(t), end="", flush=True) case ai.TextDelta(text=t): print(t, end="", flush=True) Measure Time to First Token for event in stream: if isinstance(event, ai.TimingMark) and event.name == "first_token": print(f"[{event.at_ms:.0f} ms] ", end="", flush=True) TTFT is measured by the core against time.monotonic(), identically for every provider, so numbers from different backends are directly comparable. To export timings to a metrics system rather than print them, see observability. Show Fallback as It Happens When a fallback chain is in play, AttemptFailed events allow you to show the switch as it happens rather than leaving the user staring at a stalled cursor: for event in stream: match event: case ai.AttemptFailed(record=record): print(f"[{record.target} failed: {record.error.type_name}]") case ai.TextDelta(text=t): print(t, end="", flush=True) Key Takeaways Use the context-manager form; leaving the block early cancels the in-flight request instead of letting a hosted provider keep generating and billing. Reasoning text arrives as ReasoningDelta, a channel separate from the answer, so you can dim it or drop it without parsing anything. TTFT is measured by the core, not the provider, so numbers from different backends are directly comparable. AttemptFailed events surface retries and fallback while they happen, not after. See Also The event stream: the ordering guarantees you can rely on. Add a fallback chain: the routing behind AttemptFailed. Observe requests: exporting these events instead of printing them. API reference: every event type and its fields. --- # Integrate / Enforce a JSON Schema Source: https://anyinfer.dev/guides/structured-output/ Enforce a JSON Schema Pass a schema and result.structured comes back already validated against it, whichever mechanism the target supports: import anyinfer as ai REVIEW = { "type": "object", "properties": { "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]}, "score": {"type": "integer", "minimum": 1, "maximum": 5}, "themes": {"type": "array", "items": {"type": "string"}}, }, "required": ["sentiment", "score", "themes"], "additionalProperties": False, } result = client.generate( "Analyze this review:\n" + review_text, target="medium", schema=REVIEW, repair=ai.Repair(max_attempts=1), ) analysis = result.structured # already validated against REVIEW print(analysis["sentiment"], analysis["score"]) repair=ai.Repair(max_attempts=1) allows one corrective round trip against the same model before the call fails; see repair for what that costs and why it never falls back to another provider. Handling Failure try: result = client.generate(prompt, target="medium", schema=REVIEW) except ai.SchemaViolationError as error: log.warning("model produced: %s", error.raw_text) for message in error.errors: log.warning(" %s", message) You get the bounded raw output, specific validation errors, and any delimiter-confirmed complete top-level members in error.partial, so your application can inspect the response or tighten the prompt. Fallback never fires here: the model answered, just in the wrong shape. Pydantic Models Work No pydantic dependency is added; the model is duck-typed through model_json_schema(): from pydantic import BaseModel class Review(BaseModel): sentiment: str score: int result = client.generate(prompt, target="medium", schema=Review) parsed = Review.model_validate(result.structured) Knowing What Happened result.structured_mechanism # "grammar" | "json_schema" | "json_mode" | "prompt" result.repair_attempts # 0 if the model got it right first time Both are worth logging in aggregate. A model that frequently needs repair is usually a prompt problem; a target that unexpectedly reports "prompt" may not be the model you thought you configured. Key Takeaways result.structured is validated client-side against your original schema, whatever mechanism the provider used to produce it. A SchemaViolationError carries the raw text, the specific validation errors, and any recoverable partial members (enough to debug the prompt, not just the failure). Repair is opt-in and costs an extra request per attempt; budget for it on latency-sensitive paths. Pydantic models are accepted directly, with no pydantic dependency in the library. See Also Structured output: how the mechanism is chosen, and how to write schemas that work on grammar-based engines. Test your application offline: proving a repair budget converges with a scripted provider. --- # Integrate / Add a Fallback Chain Source: https://anyinfer.dev/guides/fallback/ Add a Fallback Chain A Route names the targets to try in order and the retry policy for each; the full semantics live in routing. import anyinfer as ai route = ai.Route( targets=( "anthropic:claude-sonnet-4-5", # preferred "openai:gpt-5", # if Anthropic is unavailable "ollama:qwen3:8b", # last resort, always local ), retry=ai.Retry(max_attempts=2), ) result = client.generate(prompt, route=route) print("served by", result.target) Set a Default Route Once client = ai.Client(providers, route=ai.Route(targets=("medium", "small"))) result = client.generate(prompt) # no target= needed Inspect What Happened for attempt in result.attempts: print(f"{attempt.target}: {attempt.outcome}") if attempt.error: print(f" {attempt.error.type_name}: {attempt.error.detail}") anthropic:claude-sonnet-4-5: failed ProviderUnavailableError: provider returned HTTP 503 openai:gpt-5: ok The trail is also visible while a request runs: a stream emits an AttemptFailed event at the moment a target fails, so a UI can show the switch instead of a stalled cursor. Handle Total Failure try: result = client.generate(prompt, route=route) except ai.AllTargetsFailedError as error: for attempt in error.attempts: alert(f"{attempt.target}: {attempt.error and attempt.error.detail}") Since a chain only proves itself when things fail, the test kit can script those failures offline: a provider that 503s once and then answers exercises this exact path in CI. Route by Failure Class A context overflow needs a bigger model, not another same-sized one: route = ai.Route( targets=("openai:gpt-5-mini", "openai:gpt-5"), context_window_targets=("anthropic:claude-sonnet-4-5",), ) When a ContextLengthError occurs, the router switches to that chain instead of continuing down the general one. Tune Retry Behavior ai.Retry( max_attempts=3, backoff_base_s=0.5, # 0.5s, then 1s, then 2s... backoff_max_s=30.0, # ...capped here ) A server's Retry-After is honored when it is longer than the computed backoff, but the sleep is still capped at backoff_max_s. By default, auth failures and context overflows are not retried: repeating them cannot succeed, and the budget is better spent on the next target. Override when you know your provider better: ai.Retry(retry_on=lambda error: error.retryable or error.http_status == 409) Health Gating A target that recently failed with a transport or availability error is skipped for health_ttl_s seconds, so one dead endpoint does not cost every subsequent request its full timeout. Skipped attempts appear in the trail as skipped_unhealthy. Turn it off with health_gate=False when you want every target attempted regardless. A Good Default Shape Put a local model last. When every hosted provider is unreachable (an outage, a captive portal, an expired card), a local fallback is the difference between degraded and down: ai.Route(targets=("medium", "llama-cpp:qwen2.5-3b-instruct-q4-k-m")) Key Takeaways Targets are tried strictly in order, and result.attempts records what happened to every one, so a slow or rerouted request is explainable after the fact. Auth failures and context overflows are not retried by default, because repeating them cannot succeed; retry_on= overrides the predicate when you know better. context_window_targets sends an overflow to a bigger model instead of down the general chain, where a same-sized target would fail identically. A local model at the end of the chain keeps your application degraded rather than down when every hosted provider is unreachable. See Also Routing and rate limits: the full route semantics, including what per-call target= does and does not override. Stream to a terminal: rendering AttemptFailed as it happens. Test your application offline: scripting failures to prove the chain works. --- # Integrate / Run the Tool Loop Source: https://anyinfer.dev/guides/tool-loop/ Run the Tool Loop run_tools runs the generate, call tools, feed results back loop for you, with bounded rounds and normalized errors: import anyinfer as ai from pathlib import Path @ai.tool def read_file(path: str) -> str: """Read a project file.""" return Path(path).read_text(encoding="utf-8") @ai.tool def list_files(directory: str = ".") -> list: """List files in a directory.""" return [p.name for p in Path(directory).iterdir()] result = client.run_tools( "What does README.md say about installation?", tools=[read_file, list_files], target="anthropic:claude-sonnet-4-5", max_rounds=8, ) print(result.text) The decorator derives the JSON schema from your signature and the description from your docstring, so a tool is declared once rather than kept in sync with a hand-written schema. Supported Parameter Types str, int, float, bool, list, dict, their parameterized forms (list[str]), and Optional[T]. Anything else raises ToolLoopError when the tool is declared, before a schema that misdescribes the tool ever reaches a model. Parameters with defaults are optional; the rest are required. Errors Reach the Model, Not You A tool that raises becomes an error-flagged result the model can react to: @ai.tool def fetch(url: str) -> str: """Fetch a URL.""" return httpx2.get(url).text # may raise The model sees ConnectError: connection refused as a tool result and can apologize, try a different URL, or give up — all ordinary conversation. Only loop-level faults raise to you: an unknown tool, or an exhausted round budget. The Round Bound max_rounds (default 8) bounds the loop, because a model that keeps calling tools would otherwise never terminate: try: result = client.run_tools(prompt, tools=tools, target=target, max_rounds=4) except ai.ToolLoopError as error: log.warning("%s (%s)", error.detail, error.hint) Execution Is Sequential v1 dispatches tool calls one at a time, in the order the model requested. Parallel execution is deferred: it raises cancellation and ordering questions that no current consumer needs answered. Naming and Overrides @ai.tool(name="search_docs", description="Search the documentation index.") def search(query: str, limit: int = 10) -> list: ... Plain functions work too: run_tools(tools=[my_function]) wraps them automatically. Async result = await async_client.run_tools(prompt, tools=[read_file], target=target) Safety The loop executes whatever the model asks for, within the tools you provide. Treat tool implementations as a security boundary: validate paths, bound sizes and durations, and do not expose a tool that runs arbitrary commands unless that is genuinely your intent. Tools from an MCP Server Model Context Protocol servers distribute tools (filesystems, databases, internal APIs) behind one protocol, and AnyInfer can use one as a source of tools for the loop above: pip install "anyinfer[mcp]" from anyinfer.mcp import MCPServer, MCPToolset async with await MCPToolset.connect( MCPServer(name="fs", command=("mcp-server-filesystem", "./docs")), ) as toolset: result = await client.run_tools( "Which guide explains fallback?", tools=toolset.tools, target="anthropic:claude-sonnet-4-5", ) toolset.tools are ordinary AnyInfer tools: same bounded rounds, same sequential dispatch, same rule that a failing tool becomes a result the model can recover from. Names are namespaced by server (fs__read_file), so two servers offering search do not collide. Describing Servers in Configuration { "format_version": 1, "providers": [{"id": "anthropic"}], "mcp": [ { "name": "fs", "command": ["mcp-server-filesystem", "./docs"], "deny_tools": ["write_file"] } ] } Loading a configuration file never starts a server; the entries are inert until you connect them. Both stdio env values and HTTP headers values accept env:// and credential:// references, which resolve and register for redaction only on connect. Inspect what a server offers without running anything: anyinfer mcp list --config anyinfer.json fs__read_file Read a file from the allowed directory [read-only — server's claim] fs__list_dir List entries in a directory [read-only — server's claim] What Is Not Supported Sampling: a server asking the client to run a generation. Honoring that would let a remote server drive inference through your credentials, so it is not implemented. Prompts, resources, and roots: out of scope; this integration is a tool source. AnyInfer as an MCP server: non-Python clients reach your models through the OpenAI-compatible sidecar instead. Trust Tool results enter the model's context, and the server decides what they say. That is the prompt-injection surface every tool has; connecting a server you do not control widens it, and allow_tools/deny_tools narrow what a server may expose. Annotations such as "read-only" are the server's claims, not guarantees: AnyInfer captures them on ToolSpec.annotations so your code can reason about them, and never grants access, skips a step, or auto-approves anything because a server said it was safe. Testing It The test kit's in-process fake MCP server makes an MCP-fed tool loop testable without a subprocess: from anyinfer.testing import FakeMCPServer, FakeMCPTool fake = FakeMCPServer([FakeMCPTool("read_file", result="file contents")]) toolset = await MCPToolset.connect( MCPServer(name="fs", url="http://fake.invalid/mcp"), transport_factory=lambda _: fake.transport(), ) Key Takeaways A tool is declared once: the schema comes from the signature, the description from the docstring, and an unsupported parameter type fails at declaration rather than misdescribing the tool to a model. A raising tool becomes a result the model reacts to in conversation; only an unknown tool or an exhausted max_rounds raises to your code. MCP servers plug in as tool sources with identical loop semantics, and configuration entries stay inert until you connect them. Tool results are a prompt-injection surface and server annotations are unverified claims: narrow exposure with allow_tools/deny_tools and validate inside your tool implementations. See Also A local tool agent: the loop end to end against a local model. Test your application offline: scripted providers and the fake MCP server. Configuration: the mcp block the CLI and sidecar share. --- # Integrate / Fit a Corpus to a Budget Source: https://anyinfer.dev/guides/fitting-context/ Fit a Corpus to a Context Budget You have a pile of documents and a model with a finite window. This is the four-step pattern: build documents, ask what fits, reduce, and place the result. Reduction lives in anyinfer.context, an optional dependency-free subpackage. Your application decides what exists and what is safe to send; the library decides what fits. See context reduction for why that line is drawn there. 1. Build Documents from anyinfer import context documents = [ context.ContextDocument.of("src/auth/credentials.py", credentials_source), context.ContextDocument.of("src/auth/tokens.py", tokens_source), context.ContextDocument.of("README.md", readme_text, pinned=True), ] of() computes the digest, detects the language from the path, and derives a structural extract (the signatures-and-imports view the tiered strategy falls back to). Pass extract="" to skip extraction, or language= to override detection. pinned=True means the user explicitly chose this file: it sorts ahead of everything and is never chunked. 2. Ask What Fits Build the request you would send without the corpus, and ask what is left over: import anyinfer as ai messages = [ai.user("How does credential resolution work?")] budget = client.budget(messages, target="anthropic:claude-sonnet-4-5") max_tokens = budget.remaining_tokens if max_tokens is None: # The window is unknown; the library will not guess one for you. max_tokens = 8_000 When a target's context window is unknown, remaining_tokens is None; the fallback is your explicit choice, made where you can see it. See token estimation and context budgets for where the number comes from. 3. Reduce reduction = context.select( documents, query="how does credential resolution work?", max_tokens=max_tokens, ) The default auto strategy sends everything when it fits and falls back to tiered when it does not. Name a strategy when you want a specific shape: context.select(documents, query, max_tokens=max_tokens, strategy="ranked") # whole files context.select(documents, query, max_tokens=max_tokens, strategy="tiered") # full coverage context.select(documents, query, max_tokens=max_tokens, strategy="packed") # chunk-level The five strategies and their tradeoffs are documented on the concept page. 4. Place the Result and Check What Happened messages.insert(0, ai.user(reduction.text)) result = client.generate(messages, target="anthropic:claude-sonnet-4-5") The envelope goes in your message; the library never modifies the request. Then check what it cost you: if not reduction.complete: log.info("context reduced: %s", reduction.summary()) # ranked: 12 of 340 document(s); ~7900 of 8000 tokens; 328 omitted; limited by tokens summary() is content-free: counts and ceilings, never paths or content. metadata() gives the full machine-readable record for a debug pane. Observe Reductions like Any Other Event class ContextWatcher: def on_event(self, event): if isinstance(event, ai.ContextReduced): metrics.gauge("context.omitted", event.omitted_count) reduction = context.select(documents, query, max_tokens=max_tokens, observer=ContextWatcher()) The ContextReduced event carries counts and ceilings only, like every telemetry event. Generating Module Digests The tiered strategy renders module digests you supply, but never generates them: summarizing spends inference, and that is your decision rather than a side effect of packing. Generate them once and cache them: surfaces = context.module_surfaces(documents, depth=2) digests = {} for module, surface in surfaces.items(): key = hashlib.sha256(surface.encode()).hexdigest() if (cached := digest_cache.get(key)) is not None: digests[module] = cached continue summary = await client.generate( f"Describe what this module does in two sentences:\n\n{surface}", target="anthropic:claude-haiku-4-5", ) digests[module] = summary.text digest_cache[key] = summary.text reduction = context.select( documents, query, max_tokens=max_tokens, strategy="tiered", module_digests=digests, ) module_surfaces() is deterministic, so the digest cache key is stable across runs. The cache itself stays app-side. Going Further Each refinement is one section of the concept page: plan() prices every deterministic strategy for free before you commit; see plan before you commit. Duplicates collapse losslessly, and a file that just misses the budget can be shortened instead of dropped; see losing less than you drop. Carrying over the previous turn's selection keeps the prompt prefix stable, so provider caches keep hitting; see turn two: send the same thing. Every algorithmic choice is a ContextTuning field, and recommended() is the set worth having for source code; see tuning. compact_history() and HistoryPolicy apply the same discipline to the conversation itself; see conversations are context too. When no fidelity reduction is enough, the answer is more requests rather than fewer tokens; see distill a corpus. Key Takeaways Ask client.budget() what is left over before reducing; an unknown window comes back as None, and the fallback number is yours to choose in the open. The default auto strategy sends everything when it fits and degrades to tiered only when it must, so small corpora pay nothing. The envelope lands in your own message; the library never edits GenerationRequest.messages. reduction.complete and summary() report exactly what the budget cost, in content-free form that is safe to log. Module digests for tiered are generated by your code, keyed on the deterministic module_surfaces() output, so a cache survives across runs. See Also Context reduction: the strategies and their tradeoffs. Distill a corpus: the map/reduce cookbook. Token estimation and context budgets: where the budget comes from. --- # Integrate / Test Your Application Offline Source: https://anyinfer.dev/guides/testing-your-app/ Test Your Application Offline Your application's inference code has behavior worth testing: it falls back when a provider is down, it repairs a malformed structured answer, it reduces a corpus to fit a budget. Testing that normally means either mocking the library, which tests your mocks, or calling a real provider from CI, which is slow, costs money, and fails for reasons that have nothing to do with your change. AnyInfer ships the third option. anyinfer.testing gives you a provider whose behavior you declare, and pytest fixtures that wire it to a real client. Everything runs in-process: no sockets, no credentials, no network, and the same result on every machine. pip install anyinfer # the fixtures come with it — no extra to install Declare a Provider, Get a Real Client The fixtures are available as soon as anyinfer is installed. anyinfer_scripted builds a provider; anyinfer_client builds a client wired to it. from anyinfer.testing import ScriptedModel def test_summarizer_returns_the_models_answer(anyinfer_client, anyinfer_scripted): provider = anyinfer_scripted([ScriptedModel("small", text="A one-sentence summary.")]) client = anyinfer_client(provider) result = client.generate("Summarize this", target=provider.target("small")) assert result.text == "A one-sentence summary." Everything between your call and that assertion is the real library: the router resolved the target, the adapter spoke the wire dialect, the core measured the timings. Prove Your Fallback Chain Works A scripted model can be told to fail. Failures are consumed in order, then the model answers normally, so "fails once, then succeeds" is one line, enough to exercise the fallback chain you configured. from anyinfer.testing import ScriptedFailure, ScriptedModel def test_retries_a_transient_failure(anyinfer_client, anyinfer_scripted): provider = anyinfer_scripted( [ScriptedModel("flaky", failures=(ScriptedFailure(status=503, retry_after_s=0.0),))] ) client = anyinfer_client(provider) result = client.generate("hi", target=provider.target("flaky")) assert [attempt.outcome for attempt in result.attempts] == ["retried", "ok"] retry_after_s=0.0 advertises the header without making your suite wait for it. Prove Your Repair Budget Converges malformed-json answers with something that will not validate, so the repair loop runs for real: from anyinfer.testing import ScriptedFailure, ScriptedModel SCHEMA = { "type": "object", "properties": {"answer": {"type": "string"}}, "required": ["answer"], } def test_repairs_an_invalid_structured_answer(anyinfer_client, anyinfer_scripted): provider = anyinfer_scripted( [ ScriptedModel( "structured", structured={"answer": "valid on the second try"}, failures=(ScriptedFailure(kind="malformed-json"),), ) ] ) client = anyinfer_client(provider) result = client.generate( "extract", target=provider.target("structured"), schema=SCHEMA, repair={"max_attempts": 1} ) assert result.structured == {"answer": "valid on the second try"} assert result.repair_attempts == 1 The Failures You Can Script kind What the provider does What it lets you test status Returns an HTTP error, optionally with Retry-After Retry, backoff, the attempt trail truncate Cuts the stream mid-event Partial-response handling, teardown malformed-json Answers with something that fails validation Schema validation and the repair loop timeout Raises a read timeout Your timeout handling, without waiting refusal Finishes with content_filter Your content-policy fallback Assert on Telemetry anyinfer_events collects the same typed event stream your observers consume in production. It is payload-free: it never captures prompt or response text, so adding it to a suite cannot start logging user content. from anyinfer.events.telemetry import RetryScheduled def test_emits_a_retry_event(anyinfer_client, anyinfer_scripted, anyinfer_events): provider = anyinfer_scripted( [ScriptedModel("flaky", failures=(ScriptedFailure(retry_after_s=0.0),))] ) anyinfer_client(provider).generate("hi", target=provider.target("flaky")) assert anyinfer_events.of_type(RetryScheduled) Model Capabilities Are Declarable Too A scripted model states what it supports. Declaring a model without JSON support is how you test what your code does on the weakest structured-output mechanism: from anyinfer.types.capabilities import Feature, ModelCapabilities, Sourced ScriptedModel( "plain", structured={"answer": "ok"}, capabilities=ModelCapabilities( context_window=Sourced(8_192, "catalog"), features=Sourced(Feature.STREAMING | Feature.SYSTEM_PROMPT, "catalog"), ), ) The result still validates (client-side validation is always authoritative), but result.structured_mechanism reports prompt instead of json_schema, which is what your production code will see against a model that cannot do better. Regression-Test Inference Behavior When the route, repair budget, cache placement, or context policy is the contract you care about, compare a golden run manifest instead of asserting on model prose or unstable timing: def test_answer_path(anyinfer_golden_manifest): result = application.answer("hi") anyinfer_golden_manifest(result.manifest, "answer-path") The fixture removes request IDs and timings before comparing manifests/answer-path.json beside your test. Run pytest --update-manifests only after an intentional behavior change, then review the JSON diff. The complete fallback-and-repair example runs offline in AnyInfer's own suite. The Fixtures Fixture What it gives you anyinfer_scripted Factory for scripted providers, registered for this test only anyinfer_client Factory for sync clients, closed automatically anyinfer_async_client The same, asynchronous anyinfer_events A payload-free telemetry collector anyinfer_registry The per-test provider registry, if you need it directly anyinfer_cassette Resolves a cassette stored beside your test file anyinfer_recording Whether this run is recording cassettes anyinfer_golden_manifest Compare a normalized run manifest with a checked-in golden Each test gets its own provider registry, so two tests may register the same provider id without depending on execution order. Every fixture and scripted type is specified in the testing API reference. Recording Real Traffic In order to test against what a provider actually sent, record it once and replay it forever: def test_against_recorded_traffic(anyinfer_cassette, anyinfer_recording): cassette = anyinfer_cassette("summarize") ... Run your suite with ANYINFER_RECORD_CASSETTES=1 to record; unset it to replay. Recorded bodies pass through the redaction registry before reaching disk, so a cassette you commit alongside a test cannot carry a registered credential. Key Takeaways A scripted provider exercises the real router, adapter layer, and core, so the suite tests the library's behavior rather than your mocks. Failures are consumed in order before the model answers normally, which makes "fails once, then succeeds" a one-line declaration. anyinfer_events is payload-free by construction; adding telemetry assertions cannot start logging user content. Golden run manifests pin routing and policy decisions without asserting on prose; refresh them with pytest --update-manifests only after an intentional change. Recorded cassettes pass through redaction before reaching disk, so committing one cannot leak a registered credential. See Also Run manifests: what a manifest records and why it makes a stable golden file. Golden manifest example: fallback and repair pinned end to end. Testing API reference: every fixture and scripted type. --- # Integrate / Compare Targets Without Spending Source: https://anyinfer.dev/guides/comparing-targets/ Compare Targets Without Spending compare() resolves a concrete request (the exact messages, schema, tools, sampling controls, and cache policy) against every target you name and reports what it would become on each: whether it fits, which structured-output mechanism would enforce the schema, what would be dropped, and what it would cost. No provider is contacted, target order is preserved, and nothing is ranked. If an application wants a cost-first or local-first policy, make that selection visibly in application code and pass the resulting order to Route. A Deterministic, Offline Example This complete shape uses a scripted provider with two different capability profiles, so it is deterministic, offline, and shows a degradation rather than two identical records: from anyinfer.testing import ScriptedModel, ScriptedProvider from anyinfer.types.capabilities import Feature, ModelCapabilities, Sourced import anyinfer as ai provider = ScriptedProvider( "offline", [ # Full-featured: a wide window and native JSON Schema support. ScriptedModel( "full", capabilities=ModelCapabilities( context_window=Sourced(32_768, "default"), features=Sourced(Feature.STREAMING | Feature.JSON_SCHEMA | Feature.SYSTEM_PROMPT, "default"), ), ), # Narrow: a small window and no schema mechanism beyond prompt injection. ScriptedModel( "narrow", capabilities=ModelCapabilities( context_window=Sourced(64, "default"), features=Sourced(Feature.STREAMING | Feature.SYSTEM_PROMPT, "default"), ), ), ], ) registry = provider.register(ai.ProviderRegistry(load_builtins=False, load_entry_points=False)) with ai.Client([provider.settings()], registry=registry, use_default_catalog=False) as client: results = client.compare( "Return an object describing this sentence in detail, with several fields. " * 5, targets=[provider.target("full"), provider.target("narrow")], schema={"type": "object"}, ) for item in results: print(item.requested, item.budget.fits if item.budget else None, item.structured_mechanism) # offline:full True json_schema # offline:narrow False prompt assert [item.resolvable for item in results] == [True, True] assert provider.requests == [] The identical prompt fits full's 32,768-token window and does not fit narrow's 64-token one; item.budget.fits reports each target independently, in the order the targets were given. The schema degrades rather than failing: full would enforce it natively (json_schema), while narrow has no schema mechanism, so its record reports the prompt-injection fallback (prompt). And provider.requests is still empty afterward, because compare() never constructs an adapter or sends a request, no matter how many targets it evaluates. Unknowns stay unknown: an unknown or untrusted context window produces a None fit and an unpriced target produces cost=None, never a plausible number. A target that does not exist or is not configured comes back as a resolvable=False record explaining why, rather than an exception, so one bad candidate does not erase the comparison. refresh=True may contact providers to refresh their model listings (the one exception to the no-contact rule), but still generates no text. Since compare() cannot predict undocumented provider refusals, use verify() when you need one bounded real request as proof. The command-line equivalent accepts repeatable targets: anyinfer compare "Summarize this" --target medium --target ollama:qwen3:8b --json The sidecar exposes the same records at POST /v1/anyinfer/compare, with an OpenAI-shaped request plus a targets array. From One Comparison to a Diff compare() answers "what does this request become on this target, right now." The portability diff tool (anyinfer.compare_diff) answers two follow-on questions: did that answer just change (regression detection), and what exactly changes in a move from A to B (a portability report). It reuses compare()'s no-dispatch guarantee: no provider is called, and no target is ranked or recommended. For regression detection, define a fixture set (the requests you care about staying stable) as a small JSON file: { "schema_version": 1, "fixtures": [ { "id": "structured-summary", "request": {"messages": [{"role": "user", "text": "Summarize this document."}]}, "targets": ["anthropic:claude-sonnet-4-5", "openai:gpt-5"] } ] } from anyinfer import Client, compare_diff client = Client(providers) fixtures = compare_diff.load_fixtures("fixtures.json") current = compare_diff.snapshot(fixtures, client=client) client.close() Check current against a baseline committed to your repository, and fail CI when they differ: import json baseline = json.load(open("baseline.snapshot.json")) report = compare_diff.diff(baseline, current) if not report.is_empty: print(compare_diff.render_text(report)) raise SystemExit(1) That catches the case where a code change (an adapter update, a dependency bump, a provider preset change) silently alters what a fixed request becomes on a fixed target. The portability report needs no baseline file; it is the "should I move from A to B" answer for one request, live: report = compare_diff.diff_targets( fixtures[0], "anthropic:claude-sonnet-4-5", "openai:gpt-5", client=client ) print(compare_diff.render_text(report)) The fixture format is public and versioned (compare_diff.FIXTURE_SCHEMA_VERSION). Define fixtures against your own request shapes, since your regression risk is your own requests; the API reference has every signature. When Resolution Is Not the Question When the question is answer quality rather than request resolution, and you are willing to spend real generations, run an arena. A golden manifest answers "did this run's behavior change", while compare_diff answers "did this request's resolution change". Key Takeaways compare() resolves a request against every named target without dispatching, ranking, or choosing; order is preserved and unknowns stay None. Read the fit verdict from item.budget.fits; an unresolvable target is a resolvable=False record, never an exception. compare_diff.diff() turns snapshots into a CI regression gate, and diff_targets() produces a live A-to-B portability report, both without a provider call. Spending real generations to judge answers is an arena's job, not compare()'s. See Also Arena runs: judging answers by spending. Regression-test fallback and repair: pinning a run's behavior with a golden manifest. Capabilities and provenance: the data comparison reads. compare_diff API: full signatures and the fixture schema. --- # Integrate / Add Your Own Provider Source: https://anyinfer.dev/guides/custom-providers/ Add Your Own Provider A provider AnyInfer does not ship is not a fork: a provider is a small installable package, and once it is installed, yourprovider:model targets resolve everywhere: the Python API, the command line, the sidecar, and any config UI built on the setup spec. Do You Need an Adapter at All? If your endpoint speaks /chat/completions and differs only by URL, authentication spelling, and a few quirks, you do not. Point the built-in OpenAI-compatible provider at it: client = ai.Client( [ ai.ProviderSettings.of( "openai-compat", base_url="https://llm.internal.example/v1", api_key="env://INTERNAL_LLM_KEY", ) ] ) Write an adapter when there is real protocol translation to do: a different request shape, a different streaming framing, or discovery that reports something the OpenAI listing cannot express. Scaffold It anyinfer conform acme --scaffold ./acme-anyinfer That writes a package that already imports, registers, and resolves: acme-anyinfer/ acme_anyinfer/__init__.py the descriptor and its entry point acme_anyinfer/adapter.py the four methods to fill in contracts/acme.md the protocol snapshot to record what you depend on tests/test_conformance.py certification, ready to point at your endpoint pyproject.toml the entry point and your capability declarations README.md Write the Four Methods An adapter exposes exactly list_models, health, generate, and aclose. generate yields normalized events; everything else is the core's job: Retry, fallback, health gating, schema validation and repair, first-token timing, usage normalization, cost, telemetry, and redaction live in AnyInfer's core. If you find yourself adding control flow to an adapter, it belongs in the core instead, and it is probably already there. That constraint is what makes an adapter small. It is also what makes your provider behave identically to every built-in one without you implementing any of it. The adapter walkthrough covers each method's contract in detail, and what your models support flows through the same capabilities system the built-ins use; declare what you know and leave the rest unknown. Declare What Your Provider Needs The descriptor's ProviderSetupSpec is what allows a configuration UI to render your provider without knowing which provider it is: which fields to prompt for, which have sensible defaults, which are credentials, and which environment variable each conventionally comes from. Fill it in and every AnyInfer-based application can configure your provider. Declare SetupField.env_var on any field with a conventional variable: the bare name, "ACME_API_KEY", not the env:// reference form. It is the machine-readable half of what placeholder says in prose, and it is what allows anyinfer init to find your provider already usable on somebody's machine, and a config UI to say "we found this in your environment", without either of them parsing an example sentence. Certify It anyinfer conform acme --model acme-large acme ✅ list_models ✅ health ✅ non_streaming ✅ streaming ❌ usage usage must report output tokens ➖ reasoning declared unsupported This is the same conformance suite the built-in adapters run, and their results are published as the conformance matrix. Cases your provider genuinely cannot support are declared in your pyproject.toml, where they show as ➖ rather than as failures: [tool.anyinfer.conformance] reasoning = false # no reasoning channel on this API retry_after = false # rate limiting cannot be provoked on demand Declaring them in the project file rather than on the command line keeps the claim reviewable: "what we do not support" is checked in, not typed once on a bad day. The command exits non-zero on any failure, so your own CI can gate on it. Add --markdown-row for a pasteable conformance-matrix row, or --json for a machine-readable report. Record What You Depend On contracts/acme.md is the snapshot of exactly which upstream details your adapter relies on: endpoints, auth headers, version pins, fields sent and read, streaming framing, and error-mapping inputs. It exists so that when the provider changes something, you can tell — by comparing the snapshot against their current documentation rather than by waiting for a production failure. The in-tree adapters keep the same snapshots under contracts/, written by the procedure in contracts/NEW-PROVIDER.md. Install and Use It pip install -e ./acme-anyinfer client = ai.Client([ai.ProviderSettings.of("acme", api_key="env://ACME_API_KEY")]) client.generate("hello", target="acme:acme-large") Nothing in the application changed. If your package fails to load (a bad import, an id that collides with a built-in), anyinfer doctor says so by name, rather than leaving your provider mysteriously absent. Key Takeaways An endpoint that speaks /chat/completions needs no adapter; openai-compat with a base_url covers it. An adapter is four methods that translate protocol. Retry, validation, timing, cost, telemetry, and redaction stay in the core, which is why your provider behaves like every built-in one. Conformance cases you cannot support are declared in pyproject.toml, so the claim is checked in and reviewable, and CI can gate on the suite's exit code. The contract snapshot records the upstream details you depend on, so provider drift is detectable by comparison instead of by production failure. See Also Writing an adapter: the four methods in full. Conformance: what the certification suite checks and why. Conformance matrix: where the built-in adapters stand. OpenAI-compatible provider: the no-adapter path. --- # Integrate / Embed, Store, and Query a Small Corpus Source: https://anyinfer.dev/guides/vector-store/ Embed, Store, and Query Without a Database anyinfer-store is an embedded vector store: one SQLite file that persists the vectors client.embed() returns and answers similarity queries in-process, with no server. It serves the same audience local inference already serves: prototypes, personal tools, notebooks, small internal apps, single-tenant desktop and local-first applications. pip install -e src/anyinfer-store # from a repository checkout, until a first PyPI release The core package installs as usual; the store is a separate sub-project. Embed, Store, Query import anyinfer as ai from anyinfer_store import VectorStore client = ai.Client([ai.ProviderSettings.of("ollama")]) store = VectorStore.open("corpus.db") for doc_id, text in documents.items(): result = client.embed([text], target="ollama:nomic-embed-text", input_type="document") store.add(doc_id, result.vectors[0], space=result.space, text=text) query = client.embed(["what did the release notes say about pricing?"], target="ollama:nomic-embed-text", input_type="query") matches = store.query(query.vectors[0], space=query.space, top_k=5) for match in matches: print(match.entry.id, match.score, match.entry.text) store.close() The scale boundary The store targets corpora of thousands to low hundreds of thousands of vectors, at typical embedding dimensions (384–3072), in one process on one machine with one writer. Search is brute-force cosine similarity, and VectorStore.add warns past SIZE_WARNING_THRESHOLD (200,000 entries) that brute force may stop being comfortably fast. When you outgrow that (millions of vectors, multi-region reads, a managed control plane), point a real vector database (pgvector, Qdrant, Weaviate, Pinecone, ...) at anyinfer.embed()/anyinfer.rerank() directly: both consume the same public EmbeddingResult/RerankResult types this package does, so only where the vectors end up changes. VectorStore.query refuses a query whose embedding space doesn't match the store's bound space. That is, the identical cross-space safety rule AnyInfer's own routing applies to a fallback target (see the embedding-space safety rule) is applied to persistence: a wrong-but-plausible vector comparison fails loudly instead of returning a confident-looking, meaningless result. Second-Stage Reranking Coarse vector search, then a real rerank pass over its top-k candidates: from anyinfer_store import query_and_rerank items = await query_and_rerank( store, query.vectors[0], "what did the release notes say about pricing?", space=query.space, client=async_client, rerank_target="cohere:rerank-v3.5", candidate_k=20, top_n=5, ) Reranking needs text: an entry stored without text= cannot be reranked, and query_and_rerank refuses rather than inventing a document from nothing. The semantic search example puts embedding, storage, and reranking together end to end. Lifecycle store.remove(doc_id) # delete one entry store.compact() # reclaim disk space after deletions (VACUUM) store.export_jsonl("out.jsonl") # portable interchange format another_store.import_jsonl("out.jsonl") A store is one SQLite file; copying it is a valid backup or migration strategy on its own, and export_jsonl/import_jsonl exist for moving data between formats or into a real vector database, not as the only way to move a store. Scope The store is a library with no network service of its own, matching AnyInfer core's no-daemon posture, and it collects nothing: your application decides what gets embedded and stored, the same line anyinfer.context draws. Concurrent writers get SQLite's own file-locking and nothing more. DESIGN.md §29 in the repository is the full design record. Key Takeaways Every entry carries its embedding space, and a query from a different space is refused rather than compared (the same safety rule routing applies to embedding fallback). Reranking needs stored text; an entry added without text= cannot be reranked. A store is one SQLite file, so copying it is a complete backup; export_jsonl/import_jsonl cover interchange. Outgrowing the store changes where vectors land, not how you call AnyInfer; real vector databases consume the same EmbeddingResult/RerankResult types. See Also Embeddings and reranking: the types and the cross-space safety rule. Semantic search example: the full pipeline in one script. Installation: extras and sub-projects. --- # Integrate / Run a Model Locally Source: https://anyinfer.dev/guides/local-inference/ Run a Model Locally From a bare machine to a generated answer without installing or operating a separate model daemon. AnyInfer shows what this machine can run, downloads and hash-verifies the weights, fetches a pinned llama-server runtime after an explicit install command, and supervises the server from the application process. 1. See What This Machine Can Run The model catalog classifies every entry against detected hardware: import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("llama-cpp")]) view = client.local_catalog("llama-cpp", best_at="coding") for entry in view.runnable: size = entry.model.est_file_bytes / 1024**3 print(f"{entry.model.id:<32} {size:5.1f} GB {entry.fit.level}") qwen2.5-coder-7b-instruct 4.4 GB gpu qwen2.5-coder-14b-instruct 8.4 GB tight qwen2.5-coder-32b-instruct 18.5 GB cpu devstral-small 13.3 GB cpu Entries come back best-fit-first, and every one carries its reasoning: entry = view.entries[0] print(entry.fit.reasons[0]) # needs 5.3 GiB of VRAM; 15.6 GiB is budgeted at the balanced posture view.entries includes models that will not fit, classified no. Use view.runnable to show only the plausible ones, and view.entries when a user asked to see everything. The same thing from a terminal: $ anyinfer models list --best-at coding $ anyinfer models list --all --json anyinfer doctor prints the detected hardware and a recommended tier without writing anything. Hardware detection never raises; anything it could not determine stays unknown rather than becoming a guess. When the engine runs on a different machine, detection describes the wrong computer; see when the model runs somewhere else for supplying that host's specifications. 2. Check the Cost Before Committing Large models are large. Ask what a download would take before starting it: plan = client.acquire_model("qwen2.5-coder-14b-instruct", dry_run=True).plan print(f"{plan.quantization}: {plan.total_bytes / 1024**3:.1f} GB") print(f"already on disk: {plan.already_have_bytes / 1024**3:.1f} GB") Nothing is written. This is what a confirmation dialog should be built on. The quantization is a result, not an input: the highest-quality rung that fits this machine's memory budget. The catalog gives the rule and how to override it. 3. Download It def show(progress): if progress.fraction is not None: rate = progress.bytes_per_second or 0 print( f"\r{progress.fraction:5.0%} {rate / 1024**2:5.1f} MiB/s " f"[{progress.file_index}/{progress.file_count}] {progress.filename}", end="", ) report = client.acquire_model("qwen2.5-coder-14b-instruct", progress=show) print(f"\n{report.plan.quantization} at {report.entry.handle}") The percentage is correct from the first callback, counts anything already on disk, and never goes backwards across shards. Interrupt it and nothing is lost: partial transfers are kept, and running the same call again resumes rather than restarting. $ anyinfer models add qwen2.5-coder-14b-instruct $ anyinfer models add qwen2.5-coder-14b-instruct --dry-run $ anyinfer models add qwen3-32b --variant qwen3-32b-q6-k 4. Install a Runtime $ anyinfer runtime list $ anyinfer runtime install The default is the small CPU, Metal, or Vulkan variant appropriate for this machine. Archives are pinned and hash-verified, and no background service is installed; the step is explicit because runtime downloads can be large. CUDA is a separate, much larger opt-in that AnyInfer never installs on its own: $ anyinfer runtime install cuda which refuses, before downloading anything, if the driver or GPU is too old for the pinned build. When more than one runtime is installed, llama.cpp selects the best usable backend by default; pin one for a provider instance with options={"runtime": "cuda"}, or point at your own build with options={"binary": "/custom/llama-server"}. 5. Generate result = client.generate( "Write a Python function that reverses a linked list.", target="llama-cpp:qwen2.5-coder-14b-instruct-q4-k-m", ) That single call resolves the artifact, downloads and verifies it if step 3 did not, tunes a server plan for the hardware, starts llama-server on loopback, waits for readiness, and answers. Later calls reuse the running server. How the plan is tuned (postures, memory budgets, the KV cache) belongs to the local subsystem; pass options={"posture": "conservative"} on the provider settings to change it. client.models("llama-cpp") lists only models registered in the local store, and client.locate_model() finds a downloaded file again without network I/O; see finding it again. A tier alias such as target="medium" resolves through the same catalog, and a user's pick can become that default; see using a pick as your default tier. Ollama Instead If you already run Ollama, it needs no supervision at all: client = ai.Client([ai.ProviderSettings.of("ollama")]) result = client.generate(prompt, target="ollama:qwen3:8b") Ollama gives you grammar-enforced structured output and per-phase timings; llama-cpp gives you control over tuning and the exact model file. Either way it is one target string. Troubleshooting could not find llama-server on PATH: run anyinfer runtime install, install a llama.cpp build on PATH, or pass options={"binary": "/path/to/llama-server"}. needs about 40.0 GiB but only 8.0 GiB of VRAM is uncommitted: admission control refused before spawning, so nothing crashed. Choose a smaller tier or a more conservative posture. llama-server exited with code 3 while loading: the error includes the server's own log tail, which usually names the real cause (an incompatible quantization, a corrupt file, or genuine memory exhaustion). A model unloads while you are still reading its output: it should not; the idle timer keys on active streams, not on when the last request arrived. If you see this, please report it. Key Takeaways local_catalog() classifies every catalog entry against this machine, best fit first, with the reasons attached. A dry_run=True acquisition prices the download without writing anything, and a real one resumes after interruption. Runtime installs are explicit, pinned, and hash-verified; CUDA is a separate opt-in that is refused when the driver or GPU is too old. One generate() against a llama-cpp: target acquires, tunes, supervises, and answers; later calls reuse the running server. See Also The local subsystem: detection, tuning, and supervision. The model catalog: fit levels, verification, and the store. Quickstart: the first working call, local or hosted. --- # Integrate / Observability Source: https://anyinfer.dev/guides/observability/ Observe Requests, and Bridge to OpenTelemetry Every request emits typed telemetry events as it runs. This page shows how to consume them in-process and how to export them to OpenTelemetry; the event stream page covers the per-request events a stream consumer sees. In-Process Observers import anyinfer as ai class Metrics: def on_event(self, event: ai.TelemetryEvent) -> None: match event: case ai.FirstToken(at_ms=ms, target=target): histogram("ttft_ms", ms, provider=target.provider_id) case ai.AttemptCompleted(usage=usage, target=target): counter("tokens_out", usage.output_tokens or 0, provider=target.provider_id) case ai.RetryScheduled(error=error): counter("retries", 1, error=error.type_name) case ai.RequestFailed(error=error): counter("failures", 1, error=error.type_name) client = ai.Client(providers, observers=[Metrics()]) Keep on_event fast: it runs inline on the request path, so queue anything slow. An observer that raises is isolated and warned about once: a broken telemetry sink must never fail a generation. Catch Silent Degradation Two events exist specifically to make otherwise-invisible problems visible: class DegradationWatch: def on_event(self, event): match event: case ai.ParameterDropped(parameter=p, target=t, reason=why): log.warning("%s ignored %s: %s", t, p, why) case ai.UsageEstimated(field_name=field, method=how): log.info("usage.%s was estimated via %s", field, how) ParameterDropped fires when a provider accepts a parameter and discards it: the failure mode where temperature=0 silently does nothing and looks exactly like success. RateLimitWaited belongs to the same family. A request held back by client-side pacing is indistinguishable from a slow provider unless something says so, which is why the wait also lands in result.timing.phases["queued_ms"]. Payload Privacy Prompt and response text are None unless an observer opts in, and stripping happens per observer: client.subscribe(metrics) # never sees text client.subscribe(audit_trail, payloads=True) # sees prompt and response Everything still passes redaction first, so a resolved credential cannot appear even in a payload-carrying event. A JSONL Trail An audit trail needs no special support: consume the events directly and have your observer serialize each one to a JSONL file. OpenTelemetry from anyinfer import otel otel.install(client) # payload-free otel.install(client, record_payloads=True) # include prompt/response text Needs the [otel] extra; nothing OTel-related is imported otherwise. You get one span per request with attempts as span events, plus gen_ai.client.token.usage, gen_ai.client.operation.duration, and gen_ai.server.time_to_first_token, using GenAI semantic-convention attribute names so standard tooling reads them. Every event in the contract crosses the bridge. Events carrying a request_id become span events on that request's span, while the three that belong to no single request (ContextReduced, ServerLifecycle, DownloadProgress) become standalone spans, since attaching them to an arbitrary in-flight request would misattribute the work. The full mapping is in the OpenTelemetry bridge; the bridge is a consumer of the event contract, so consuming events directly and exporting to OTel are both first-class, and you can do both. Cost if result.usage.cost_usd is not None: ledger.record(result.usage.cost_usd) else: ledger.record_unknown(result.target) # do NOT record this as zero None means unknown, not free. Treating the two the same turns a reporting gap into a silent financial error. See cost is tri-state. Key Takeaways Observers run inline on the request path: keep them fast, and know that one which raises is isolated rather than allowed to fail a generation. ParameterDropped, UsageEstimated, and RateLimitWaited make degradation visible that would otherwise look exactly like success. Payloads are opt-in per observer, and redaction runs before any event is delivered. The OTel bridge consumes the same event contract as your observers, so the two paths never disagree and can run side by side. Never record an unknown cost as zero; cost_usd is None when the price is not trusted. See Also Telemetry and observers: the full event contract and the OTel mapping. The event stream: per-request events and their ordering guarantees. Cost and spending: the ledger and spend ceilings behind cost_usd. --- # Integrate / Confidentiality Tiers Source: https://anyinfer.dev/guides/confidentiality-tiers/ Confidentiality Tiers BYOK (bring-your-own-key) inference already answers one confidentiality question: your application's calls go straight from your process to the provider you configured, AnyInfer is never a proxy, and redaction keeps secrets out of logs. That protects your customer's data from AnyInfer and from you. It does not protect your own prompt IP (templates, orchestration, few-shot curation) from a customer running your client software on infrastructure they own. No purely client-side technique can: the customer owns the machine, the OS, and the network stack. This page is a ladder from "raises the cost of extraction" (Tiers 1–2) up to the one point where a real cryptographic guarantee becomes possible (Tier 3, hardware-attested local execution), plus a verification layer on top of it (Tier 4). Each tier's guarantee, cost, and limits are stated once, in the table: Tier What it guarantees What it costs Ships in 0 (BYOK) Your customer's prompt data never passes through AnyInfer or you Nothing; this is the default anyinfer core 1 (SealedTemplate) Your template plaintext resists static extraction from the shipped bundle No protection against a live debugger or memory inspection anyinfer-confidential 2 (AnyInfer Relay) Your orchestration logic never ships to the client at all You're back in the customer's data path for that call anyinfer-confidential 3 (Attested local execution) Not even root on the host can read the prompt in transit to or during local inference Requires specific TEE hardware (SEV-SNP/TDX today) anyinfer core 4 (Model provenance) The model weights that ran are exactly what you signed, verified inside Tier 3's boundary Only a Tier 4 claim when Tier 3 also holds anyinfer core The full design record, including exclusions and open questions, is DESIGN.md §30 in the repository. Tier 1 (Sealed Templates) A template is authored as plaintext, sealed at build time with AES-256-GCM, and shipped as an opaque asset. At runtime, TemplateVault decrypts a template into memory only immediately before rendering and best-effort-zeroes the buffer afterward. Decryption is gated on a signed, time-boxed license blob, so an install without a valid license cannot render a single prompt (which doubles as a licensing mechanism). pip install "anyinfer-confidential[relay]" # relay is optional; Tier 1 alone needs no extra from anyinfer_confidential import ( KeyRing, TemplateVault, generate_key, generate_signing_keypair, issue_license, seal_template, ) # Build time, run once, keep the private outputs out of the client bundle: key = generate_key() private_key, public_key = generate_signing_keypair() # public_key ships with clients template = seal_template( "Summarize this for {audience}: {document}", key=key, template_id="summarize", key_id="k1", ) # Deployment time, per customer install: license_blob = issue_license("customer-42", private_key=private_key, valid_days=30) # Runtime, inside the shipped client: vault = TemplateVault( key_ring=KeyRing({"k1": key}), license_public_key=public_key, license_blob=license_blob, ) prompt = vault.render(template, audience="engineers", document="the release notes") Every sealed template carries a key_id and KeyRing holds as many keys as you provision, so rotation is re-sealing under a new id; a compromised historical key stops decrypting once dropped from the ring. Entitlement is offline by default: the license blob validates entirely locally, so an air-gapped deployment works. Online revocation is opt-in (revocation_checker), failing open by default since offline operation is the baseline; set revocation_fail_closed=True when guaranteed revocation matters more than availability. The anyinfer-confidential CLI mirrors the library one-for-one (keygen, seal, issue-license) for build pipelines that are not Python. Tier 2 (the Relay) Tier 1 protects template text. The Relay protects the orchestration pipeline itself (which templates fire in what order, routing logic, few-shot selection) by never shipping it to the client. The cost is symmetric: for that call you are back in the customer's data path, which trades against the Tier 0 posture. The Relay sees the assembled request transiently and persists nothing, by design. Relay accepts non-proprietary slot-fill inputs and a routing key, resolves them against a server-side RelayRoute, and either returns the assembled prompt for the client to send itself (mode="assemble"; no credential ever touches the Relay) or forwards it using a credential supplied fresh on every call (mode="forward", never persisted). Tenant isolation is structural: a request scoped to one tenant_id cannot resolve another tenant's routes. from anyinfer_confidential.relay import Relay, RelayRegistry, RelayRoute registry = RelayRegistry() registry.register( "customer-42", RelayRoute(routing_key="summarize", template=template, target="ollama:qwen3:8b"), ) relay = Relay(vault=vault, registry=registry) result = await relay.handle( tenant_id="customer-42", routing_key="summarize", slots={"audience": "engineers", "document": "the release notes"}, mode="assemble", ) print(result.assembled_prompt) anyinfer_confidential.app.build_app(relay) serves it over ASGI with the relay extra. Self-hosted and hosted deployments run the identical Relay class; AnyInfer does not currently operate a hosted instance. Tier 3 (Attested Local Execution) The one tier with a real cryptographic guarantee, because it targets AnyInfer's own local adapters instead of a cloud call. When the host supports a trusted execution environment (AMD SEV-SNP or Intel TDX today), the local runtime can run inside it, and confidential_execution_status() reports whether the guarantee holds right now, on this box: from anyinfer.local import confidential_execution_status, available_backends backend = available_backends()[0] status = confidential_execution_status(backend=backend) if status.end_to_end: print(f"attested: {status.detail}") else: print(f"not attested: {status.detail}") # render this to the caller, don't guess ConfidentialExecutionAdapter wraps a local adapter and enforces the same check as a precondition: it refuses to generate() unless end_to_end is True, raising ConfidentialExecutionError instead of falling back to unattested execution. Enforcement and pre-flight call the identical function, so they cannot drift apart. from anyinfer.providers.confidential_execution import ConfidentialExecutionAdapter adapter = ConfidentialExecutionAdapter(inner_llama_cpp_adapter, backend=backend, model=model) # adapter.generate(req) now fails closed instead of silently running unattested end_to_end=True means precisely: the CPU package this process runs in is inside an attestable TEE, and, if the selected model offloads any layers to a GPU, that GPU is confidential-computing-capable and has CC mode enabled, closing the PCIe bridge. ConfidentialExecutionStatus carries every intermediate fact (cpu_tee, gpu_cc_capable, gpu_cc_enabled, gpu_offload_required) so an application can render a specific reason rather than a bare False. Detection Versus Cryptographic Attestation What is implemented today is detection: the check reads the TEE guest device nodes (/dev/sev-guest, /dev/tdx_guest) and NVIDIA's nvidia-smi conf-compute surface, which tells you the guest kernel believes it is inside a TEE. It is not yet the stronger claim of cryptographic attestation: generating and verifying a signed hardware quote against AMD's or Intel's root of trust, which is what rules out a lying hypervisor. That verification step is scoped (an attest-extra addition) but not built. Do not read end_to_end=True today as "a cryptographic quote was checked"; this section will change when that lands. Deployment Scope, Today CPU-only (SEV-SNP or TDX): broadly available as GA lift-and-shift confidential VMs on AWS, Azure, and GCP, no application changes. GPU-offload (NVIDIA H100 CC): confirmed GA only on Azure (with SEV-SNP) and Google Cloud (with TDX), and only for H100. Treat this as the newer, narrower claim. AWS Nitro Enclaves and Intel SGX are detected and reported in cpu_tee, but are not part of the end_to_end claim in this release. Tier 4 (Model Provenance) Tier 3 proves where a prompt ran; Tier 4 proves what ran inside it, with a signed manifest and a hash check. This is verification-only software: AnyInfer never signs anything and never touches a private key. You sign your own model manifests with your own keys; AnyInfer ships the verifier. from anyinfer.local import ModelManifest, hash_model_weights, verify_model_manifest # At sign time, on infrastructure you control (never AnyInfer's): weight_hash = hash_model_weights(model_path) # ... sign {model_id, weight_hash, vendor_key_id, signed_at} with your Ed25519 key ... # At verify time, inside a Tier 3-attested process: ok = verify_model_manifest(manifest, weights_path=model_path, vendor_public_key=public_key) confidential_execution_status() accepts manifest=/vendor_public_key= and populates model_verified on the status. Verification is never cached, so a swapped file is caught on the next call. Treat model_verified is True as meaningful only when end_to_end is True too; a hash-and-signature check on an unattested host is a weaker, different guarantee. Appendix: SOC 2 Control Mapping An auditor evaluating a vendor built on AnyInfer needs the tiers restated in Trust Services Criteria vocabulary. Each row cites the typed fact it rests on, so the claim can be re-verified against running code. This table is a starting point for your own auditor conversation, not a substitute for one: your organization holds the SOC 2 report, and your auditor decides how a control is worded for your environment. SOC 2 control area Confidentiality tier The typed fact Caveat Confidentiality of data in transit Tier 0 (BYOK) AnyInfer's adapters call the provider directly; no proxy hop exists in the call graph Applies to customer data, not vendor prompt IP; that's what Tiers 1-4 answer Confidentiality of data at rest Tier 1 (SealedTemplate) EncryptedTemplate.ciphertext (AES-256-GCM); plaintext never touches the on-disk asset Protects against static extraction only, not a live-memory or debugger control Access control / authentication Tier 1 entitlement TemplateVault refuses to decrypt without a signature-verified, unexpired LicenseBlob Offline by default; online revocation is opt-in per deployment Data retention / minimization Tier 2 (AnyInfer Relay) Relay.handle() writes no request or response body to any durable store; zero retention is structural You operate the Relay process, so its hosting environment's logging discipline is yours Logical access / tenant isolation Tier 2 multi-tenant RelayRegistry.resolve(tenant_id, routing_key) cannot resolve another tenant's routes Relevant only for a shared Relay serving more than one downstream vendor Confidentiality of data in use Tier 3 (attested local execution) ConfidentialExecutionStatus.end_to_end: root on the host cannot read the prompt during local inference, when True Requires TEE hardware (SEV-SNP/TDX today); False reported plainly otherwise Change management / fail-safe defaults Tier 3 enforcement ConfidentialExecutionAdapter raises and never calls the inner adapter when attestation is unavailable Enforcement and pre-flight share one function, so they cannot drift apart Integrity of processing Tier 4 (model provenance) ConfidentialExecutionStatus.model_verified: weights hash-match a vendor-signed manifest, checked fresh on every call Only a Tier 4 claim in combination with end_to_end Three scope notes an auditor will ask about: none of the tiers are availability controls, and none should be cited as one; the GPU-attestation claim is scoped to the deployment pairings above; and Tier 3 today is detection, not cryptographic attestation; read that section before the follow-up question arrives. Key Takeaways Tiers 1–2 raise the cost of extracting prompt IP; only Tier 3 (TEE-attested local execution) carries a cryptographic guarantee, and Tier 4 is meaningful only inside it. Everything fails closed: no valid license, no rendered prompt; no attestation, no generation; no matching hash, model_verified=False. Tier 3 today detects TEE presence rather than verifying a signed hardware quote; cite it accordingly. The SOC 2 mapping restates the same typed facts in auditor vocabulary; it does not add guarantees. See Also The local subsystem and its API reference: Tiers 3–4's machinery. Credentials and redaction: the Tier 0 posture. Why and when to use AnyInfer: where confidentiality fits the larger case. --- # Integrate / Coding Agents Source: https://anyinfer.dev/guides/coding-agents/ Coding Agents The most common way a library gets integrated in 2026 is that a coding agent writes the integration. That path has a specific failure mode here, and this page is about closing it. An agent working from pre-training and a web search will confidently produce an OpenAI clone: client.chat.completions.create(...), a model= keyword, response_format={"type": "json_object"}, because that is what almost every other library in this space looks like. AnyInfer is not that, so the most predictable guess is also the most likely to be wrong. Worse, the wrong version usually runs: a hand-written retry loop around a call that already retries produces four attempts where you asked for two, and nothing fails loudly enough to notice. Three artifacts exist to correct that, and every one of them is derived from something canonical rather than written twice. anyinfer agents-md — Instructions for Your Repository anyinfer agents-md >> AGENTS.md anyinfer agents-md --format claude > CLAUDE.md anyinfer agents-md --format copilot > .github/copilot-instructions.md A short fragment: the call shape, the traps worth pre-empting, and the list of things not to hand-roll. It is rendered from live introspection: the provider counts come from the registry, the extras from installed distribution metadata, and the version from the package. A fragment generated by one release describes that release and stamps which one it was, so a stale copy is noticeable rather than merely wrong. Point it at your configuration and it also names what you configured, so an agent working in that repository writes targets that exist there: anyinfer agents-md --config anyinfer.json >> AGENTS.md The command prints and writes nothing. Installing instructions into somebody's repository uninvited is exactly the kind of helpfulness that gets a tool distrusted; the redirect is yours to make, and it is also your review step. llms.txt — the Documentation, Machine-Readable Built with the site and published at its root: https://anyinfer.dev/llms.txt: the index: a one-line summary of every page, grouped the way the navigation is. Each section heading links a full-text bundle of just that section (llms/concepts.txt, llms/providers.txt, …), each sized to fit one context window. https://anyinfer.dev/llms-full.txt: the full text of every page in one file, navigation chrome stripped, including the generated API reference. It has outgrown a single context window; fetch a section bundle unless you truly want the whole corpus. All of them are generated from the navigation and the built pages, so a page added to the docs appears without a hand edit and a deleted one cannot linger. The Integration Skill docs/agents/INTEGRATION.md is the canonical procedure: establish the version, read what the application already configured, write the call the way the Python SDK guide does, do not re-implement the core, interpret results accurately, prove it offline with the test kit, then check the work. Three thin entry points in the repository invoke it: .claude/skills/anyinfer-integration/, .agents/skills/anyinfer-integration/, and .github/prompts/anyinfer-integration.prompt.md. To use it in your own project, copy the canonical file and whichever shim your tool reads: mkdir -p docs/agents .claude/skills/anyinfer-integration curl -o docs/agents/INTEGRATION.md \ https://raw.githubusercontent.com/anthturner/AnyInfer/main/docs/agents/INTEGRATION.md curl -o .claude/skills/anyinfer-integration/SKILL.md \ https://raw.githubusercontent.com/anthturner/AnyInfer/main/.claude/skills/anyinfer-integration/SKILL.md The library never writes into your .claude/, .agents/, or .github/ directory, and the wheel carries no skill payload. Adjust the relative link in the shim to wherever you put the procedure. One Caveat A generated fragment and a copied procedure describe the release they came from, so regenerate them when you upgrade: anyinfer agents-md > /tmp/anyinfer-agents.md && diff /tmp/anyinfer-agents.md AGENTS.md What does not go stale is the library itself. anyinfer providers, anyinfer verify, and anyinfer run --dry-run answer from the installed code, which is why the procedure tells an agent to run them rather than to trust anything it remembers, including this page. Key Takeaways The predictable failure is an OpenAI clone that runs: hand-rolled retries around a call that already retries, and no loud failure to flag it. anyinfer agents-md renders from live introspection and stamps its release; it prints only, so installing the fragment stays your decision and your review step. llms.txt and llms-full.txt are generated from the built site, so they cannot drift from the documentation. The canonical procedure lives in docs/agents/INTEGRATION.md; the .claude/, .agents/, and .github/ entries are thin shims that invoke it. See Also The integration procedure: what an agent is told to do, step by step. Integrate the Python SDK: the call shapes an agent should produce. Test your application offline: how "prove it offline" works. --- # Integrate / Integration Procedure Source: https://anyinfer.dev/agents/INTEGRATION/ Integrating AnyInfer — the procedure The canonical, tool-neutral procedure for a coding agent adding or changing AnyInfer code in an application. The Codex skill, the Claude Code skill, and the Copilot prompt are thin entry points that read this file; nothing in them repeats what is written here. Read it in full before writing code. It exists because the most predictable guess about this library — that it is an OpenAI client with more providers — is wrong in ways that compile, run, and produce plausible output while quietly duplicating work the core already does. Step 0 — establish which version you are working against python -c "import anyinfer; print(anyinfer.__version__)" anyinfer agents-md # the short instruction fragment for this exact version anyinfer agents-md prints the call shape, the traps, and the live provider and extras lists. Read its output before anything you remember about this library. Append it to the repository's own instructions if it is not there yet: anyinfer agents-md >> AGENTS.md It writes nothing itself — the redirect is yours to make. Step 1 — find out what the application already configured anyinfer providers # every registered provider and the fields it needs cat anyinfer.json # the shared configuration, if the repo has one anyinfer agents-md --config anyinfer.json If there is no configuration file, anyinfer init writes one from what the machine can already reach. Do not invent provider ids, model names, or targets: a target that does not resolve fails at dispatch, and the purpose of the registry is that the answer is lookupable. Step 2 — write the call One primitive: a request becomes a typed event stream, and the non-streaming call is that stream drained. import anyinfer as ai with ai.Client([ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")]) as client: result = client.generate(prompt, target="anthropic:claude-sonnet-4-5") Four things to get right, because each has an obvious wrong version: target=, not model=. A target is provider:model, split on the first colon only, so ollama:qwen3:8b names the model qwen3:8b. A bare string with no colon is a catalog alias (small, medium, large). schema=, not response_format=. Pass a JSON Schema, a mapping, or a pydantic-style model. The strongest mechanism the target supports is chosen for you and the reply is validated before it is returned. Add repair=ai.Repair(max_attempts=1) if a retry on a malformed answer is worth one more call. route=ai.Route(...), not a retry loop. Retries, backoff, Retry-After, fallback between targets, and health gating all belong to the router. Every attempt lands on result.attempts afterwards. async first. AsyncClient is the same surface with await. The sync Client wraps it on a background loop; do not build a second synchronous path. Step 3 — do not re-implement the core If the change you are about to make is on this list, the library already does it, tests it, and reports what it did. Duplicating it produces double retries, double token counting, or a second answer to a question that already has one. Tempting to write Already exists a retry/backoff wrapper ai.Route, ai.Retry a JSON-repair loop schema= plus repair=ai.Repair(...) if provider == "ollama": ... registry descriptors; every provider fact is a field a token counter, a price table client.budget(...), result.usage prompt trimming anyinfer.context, ai.HistoryPolicy a secret loader api_key="env://VAR", resolved once and redacted a spend guard ai.SpendPolicy on the client client-side rate limiting RateLimits on the provider instance Step 4: interpret results accurately result.usage.cost_usd is a Decimal or None. None means the price is unknown, not zero. Never coerce it to 0. Capability values are Sourced[T]: .value plus a .provenance of catalog, discovered, probed, or default. Do not present a defaulted context window as a measured one. result.warnings and the telemetry stream carry degradations: a dropped parameter, a weaker structured-output mechanism, a compacted conversation. If the application shows results to a user, show these too. Step 5 — prove it, offline The library ships its own test kit, so an integration's fallback, repair, and reduction paths get real tests with no credentials and no network: from anyinfer.registry import ProviderRegistry from anyinfer.testing import ScriptedFailure, ScriptedModel, ScriptedProvider registry = ProviderRegistry(load_builtins=True, load_entry_points=False) provider = ScriptedProvider( "acme", [ScriptedModel("flaky", failures=(ScriptedFailure(status=503, retry_after_s=0.0),))], ) provider.register(registry) Then verify the real targets answer, which a health check cannot tell you — a credential can be valid for a model listing and useless for inference: anyinfer verify --config anyinfer.json Step 6 — check the work before reporting it Every target you wrote resolves: anyinfer verify . No credential value appears in any file you changed; only env:// or credential://system/... references. No retry, validation, cost, or provider-branch logic was added outside the library. The application's own tests cover the failure path, not only the success path. Where to look next https://anyinfer.dev/llms.txt: the documentation index, and https://anyinfer.dev/llms-full.txt for the full text. anyinfer run "..." --dry-run: what a request would cost and whether it fits. The guide behind this procedure: coding agents. --- # Integrate / Reference Application Source: https://anyinfer.dev/guides/demo-app/ The Pack-In Demo Application AnyInfer ships a PySide6 reference application in src/demo_app/. It is a worked example of integration, not part of the library's public API; nothing in anyinfer imports it, and nothing in it is importable from anyinfer. It runs with no credentials and no network: the default configuration talks to an in-process fake provider built on anyinfer.testing.fakes, so every subsystem it exercises is demonstrable offline. Running It pip install -e ".[demo]" anyinfer-demo Or, from a checkout, via the task runner or the module directly: python workspace.py demo python -m demo_app Flag Effect --config PATH Use a specific settings file instead of the per-user default. --reset Ignore saved settings and start from the offline defaults. The Integration Pattern Worth Copying Qt owns the main thread; the Client owns a background loop thread. demo_app/engine.py keeps them apart: Every call runs on a QThreadPool worker; never on the GUI thread. Results cross back as Qt signals, which Qt marshals to the GUI thread. No widget touches AnyInfer directly, and the engine touches no widget. Calling client.generate() from a button handler instead would freeze the UI for the length of the request. Equally important is what the demo does not contain: no retry loop, no fallback logic, no schema validation, and no timing measurement. Those belong to the library, and the demo's job is to show how little an application needs to add on top of it. What It Demonstrates Each surface is a small, inspectable use of one public subsystem: Provider setup renders one panel per registered provider from its declared ProviderSetupSpec, with no per-provider code; third-party adapters registered through the anyinfer.providers entry-point group appear automatically. See configuration and providers. Streaming writes the transcript from TextDelta events as they arrive, with ReasoningDelta text in a separate collapsible region; two tabs can stream from two providers at once. See stream to a terminal. Telemetry re-emits each typed event as a Qt signal through a plain observer registered without payloads=True, so prompt and response text are labeled "withheld" rather than shown empty. See telemetry and observers. Structured output sends a JSON Schema and reports which mechanism the core selected and how many repair rounds it took. See structured output. Routing and fallback are reproducible without a real outage through four offline fake models (reliable, flaky, slow, tools). See routing and rate limits and add a fallback chain. The target inspector runs resolve(), verify, probe, and a paired warm/cold benchmark against the selected target, with price tags on the buttons that spend real tokens. See proving a target works. The tool loop hands two @tool functions to Client.run_tools() and lists the functions that executed. See run the tool loop. Local inference (Tools → Local Inference…) turns hardware detection, the model catalog, benchmarking, and runtime installs into one dialog. See the local subsystem and the model catalog. Every major surface carries a chip that opens the SDK story behind it: the public calls involved, a copyable plain-Python snippet, and where in the demo source it is wired. A test resolves every named symbol against the real package, so the help cannot drift from the API. Tests The demo is covered by tests/demo_app/, which runs headless (QT_QPA_PLATFORM=offscreen) and drives real generations (streaming, retry, fallback, structured output) through the offline provider: pytest tests/demo_app # or the full suite: python workspace.py check --only=test Key Takeaways The demo runs offline against fake providers, with no credentials and no network. Copy the threading pattern in demo_app/engine.py: AnyInfer calls on worker threads, results back to the GUI thread as Qt signals, no widget touching the SDK. The demo contains no retry, fallback, validation, or timing logic of its own; those belong to the library. Its tests run headless and drive real generations through the offline provider. See Also Integrate AnyInfer Stream to a terminal: the same event stream, without Qt. Observe requests Add a fallback chain --- # Concepts / Concepts Source: https://anyinfer.dev/concepts/ Concepts Eighteen ideas. Read them once and the rest of the API follows from them. They build on each other roughly in this order, but each page stands alone. The Request Path Page The idea in one line Targets and aliases Where a request goes: two spellings that resolve to one thing. The event stream A generation is an ordered stream of typed events; everything else is a projection of it. Routing and rate limits Retries, fallback chains, health gating, and opt-in pacing; deterministic and fully traceable. Structured output A schema is a contract: strongest native mechanism, always client-side validated, optional bounded repair. Sessions Letting a provider keep what it already knows, without changing any answer. Embeddings and reranking Typed, routed inference operations with a fallback safety rule generation does not need. Multimodal inputs Images, documents, and audio enter as typed payloads without fictional token estimates. Cost and Context Page The idea in one line Capabilities and provenance Every capability value records where it came from, so a developer knows how much to trust it. Token estimation and context budgets How many tokens a request will spend, whether it fits, and when to refuse before dispatch. Cost and spending Unknown cost stays unknown, while trusted usage and prices support real ceilings. Prompt caching Reuse provider-side prompt work without confusing cache hints with guarantees. Context reduction Fitting more material than the window holds, and reporting exactly what was dropped. Local Execution Page The idea in one line The local subsystem Hardware detection through supervised llama-server, so local models are one target string. The model catalog What a machine could run locally, whether this one can run it, and verified acquisition. Operations Page The idea in one line Credentials and redaction Secrets are referenced, not embedded, and can never reach a log. Telemetry and observers Typed in-process events, payload-free by default. Run manifests One serializable, diffable explanation of a call's decisions. Arena runs Compare a fixed target set concurrently, select deterministically, and retain every candidate. The One Rule Underneath All of Them Adapters only translate. The core orchestrates. Retry, fallback, health gating, schema validation, repair, TTFT measurement, usage normalization, cost computation, telemetry, and redaction all live in the core, implemented once, behaving identically no matter which provider served the request. A provider adapter does exactly four things: list models, report health, translate a request into its wire format and its responses back into events, and close. When a developer changes target= from a hosted model to a local one, the behavior the application depends on does not change with it. --- # Concepts / Targets and Aliases Source: https://anyinfer.dev/concepts/targets/ Targets and Aliases A target says where a request goes. There are two spellings and one resolution path. flowchart LR A["'provider:model'"] --> C[Resolved target] B["'alias' e.g. medium"] --> D[First configured provider that offers it] D --> C provider:model client.generate(prompt, target="anthropic:claude-sonnet-4-5") client.generate(prompt, target="ollama:qwen3:8b") The string is split on the first colon only, because model names legitimately contain colons: "ollama:qwen3:8b" is the provider ollama and the model qwen3:8b. Provider names normalize (lowercased, trimmed, _ → -) and honor aliases, so "Claude:claude-sonnet-4-5" and "anthropic:claude-sonnet-4-5" are the same target. Aliases An alias is a tier name that resolves to a concrete model per provider: client.generate(prompt, target="medium") The bundled catalog ships small, medium, and large. Which provider serves an alias is determined by the order providers were configured: client = ai.Client( [ ai.ProviderSettings.of("ollama"), # tried first ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) client.generate(prompt, target="medium") # -> ollama:qwen3:4b Reverse that list and medium resolves to Anthropic instead. The rule is deliberately boring: first configured provider that offers the alias wins. Nothing is scored, ranked, or chosen on the application's behalf, so the same code makes the same choice every time. Why Aliases Exist They allow an application to offer "pick a size" instead of "pick a model id", and allow a developer to change which model backs a tier without touching application code. Catalog entries are data; application code refers to tiers. Resolution Is Total Resolution either produces a concrete target or raises a ConfigError telling the caller what to do instead. It never silently substitutes a different model: client.generate(prompt, target="gpt-5") # ConfigError: unknown target 'gpt-5' # (hint: use 'provider:model' (e.g. 'anthropic:claude-sonnet-4-5'), # or one of these aliases: large, medium, small) A target can be resolved without issuing a request: resolved = client.resolve("medium") print(resolved.provider_id, resolved.model, resolved.via_alias) # ollama qwen3:4b medium Proving a Target Actually Works Resolving a target says it is spelled correctly, nothing more. verify() spends one tiny request to prove the credential, model, and deployment actually serve, and probe() measures features on compatibility endpoints; both are covered in proving a target works. The CLI wraps the first as anyinfer verify. Overriding the Catalog Applications overlay their own catalog; app entries win: from anyinfer.catalog import Catalog overlay = Catalog.from_mapping( { "format_version": 1, "aliases": { "medium": { "description": "our pinned medium tier", "targets": {"anthropic": {"model": "claude-sonnet-4-5"}}, } }, } ) client = ai.Client(providers, catalog=ai.load_default_catalog().overlay(overlay)) An overridden alias replaces the bundled one wholesale rather than merging its target map, so a provider can be removed from a tier that should not be used. Pinning a catalog is also how an application insulates itself from bundled-catalog churn. Targets Are OpenAI Model Strings Every target spelling fits in an OpenAI model field, which is what makes the sidecar able to federate without inventing a routing syntax: curl localhost:8080/v1/chat/completions \ -d '{"model": "ollama:qwen3:8b", "messages": [{"role": "user", "content": "hi"}]}' This is enforced by a round-trip test, not just intended. Key Takeaways A target is either provider:model or a catalog alias; both resolve through the same path, and resolution either succeeds concretely or raises ConfigError. Aliases resolve to the first configured provider that offers them, every time; nothing is scored or chosen on the application's behalf. Every target spelling is a valid OpenAI model string, which is what lets the serve frontend federate without inventing a routing syntax. See Also Routing and rate limits: a Route is an ordered list of targets plus policy. Capabilities: what is known about a resolved model, and how to verify or probe it. The model catalog: where aliases and their tier data live. --- # Concepts / The Event Stream Source: https://anyinfer.dev/concepts/events/ The Event Stream A generation is an ordered stream of typed events. Everything else (the non-streaming generate(), the OpenAI chunk format, a progress bar) is a projection of that one primitive. This page covers the request-scoped stream a caller consumes; telemetry is the separate observer-facing channel for what happened around the request. flowchart LR A[GenerationRequest] --> B{stream?} B -->|yes| C[TextDelta] B -->|yes| D[ReasoningDelta] B -->|yes| E[ToolCallDelta] C --> F[StreamEnded] D --> F E --> F B -->|no| G[Generation] The Events Event Meaning TextDelta(text) A fragment of the visible answer. ReasoningDelta(text) A fragment of thinking, excluded from the answer. ToolCallDelta(index, call_id, name, arguments_fragment) Part of a tool call; correlate by index. UsageUpdate(usage) A usage report. May arrive mid-stream, and more than once. TimingMark(name, at_ms) "attempt_start" or "first_token", measured by the core. AttemptFailed(record) A target attempt failed; a retry or fallback may follow. StreamEnded(result) Terminal. Carries the assembled Generation. The Ordering Guarantees These are a binding contract, verified by the conformance suite for every adapter: Zero or more AttemptFailed may precede any content (failed targets, retries). Within one attempt, TimingMark("attempt_start") comes first, and TimingMark("first_token") appears exactly once, immediately before the first content delta. StreamEnded is always the final event, exactly once. An unrecoverable failure raises instead of yielding it. Within one attempt, concatenating every TextDelta.text equals StreamEnded.result.text. Guarantee 4 is what allows a consumer to render deltas as they arrive and still trust the final result. Guarantees 2 and 4 are scoped per attempt: when a schema violation triggers the opt-in repair loop, the repair re-runs the target inside the same stream, announced by a fresh TimingMark("attempt_start"). Treat each attempt_start as "clear and start over": after it, the delta sequence restarts and result.text reflects the final attempt only. Consuming It Print deltas as they arrive: with client.stream(messages, target="ollama:qwen3:8b") as stream: for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) Or watch for specific events, then take the authoritative result: async with client.stream(messages, target=target) as stream: async for event in stream: if isinstance(event, ai.TimingMark) and event.name == "first_token": record_ttft(event.at_ms) record(stream.result.usage, stream.result.timing) Plain generate() is the same machinery with the stream drained internally; ignoring events costs nothing and changes nothing. Non-Streaming Providers Still Stream An adapter for a provider with no streaming API emits one TextDelta and a final event. Consumer code does not change, which is the point. The contract is the library's, not the provider's. Usage Is a Late-Arriving, Optional Event Usage often arrives after the finish reason, in a trailing chunk. Two consequences: The parser drains to the protocol's terminal sentinel, never stopping at finish_reason; stopping early undercounts tokens. Providers that report usage only on their terminal object (Ollama) still produce a UsageUpdate event, because the core synthesizes one. Consumers see one behavior. Finish Reasons Are an Open Enum FinishReason is "stop" | "length" | "tool_calls" | "content_filter" | "other". A value a provider invents tomorrow normalizes to "other" rather than crashing the reassembler. Timing Is Measured by the Core first_token_ms, total_ms, and output_tokens_per_s are measured centrally against time.monotonic(), so they mean the same thing across every provider and are comparable. Throughput is measured over the decode window (first token → completion), because including queue and prefill time would understate a model's generation rate. Provider-reported sub-timings, when available, land in timing.phases: result.timing.phases # {"model_load_ms": 300.0, "prefill_ms": 200.0, "decode_ms": 1000.0} Early Exit Cancels Leaving a stream's context manager before draining it cancels the underlying request: with client.stream(prompt, target=target) as stream: for event in stream: if enough(event): break # the provider request is cancelled here Key Takeaways A generation is one ordered stream of typed events; generate() is just the drained stream, not a separate code path. Four ordering guarantees are enforced for every adapter by the conformance suite: concatenated TextDeltas always equal the final text. Usage and timing are measured centrally, so they mean the same thing across providers. See Also Routing and rate limits: where AttemptFailed comes from. Telemetry: the separate, observer-facing event channel. Stream typed events: the task-oriented walkthrough. OpenAI-compatible sidecar: the OpenAI chunk projection. --- # Concepts / Routing and Rate Limits Source: https://anyinfer.dev/concepts/routing/ Routing and Rate Limits A route decides where a request goes and what happens when an attempt fails: retries, fallback chains, and health gating, all deterministic and fully traceable afterward. Rate limiting is the other half of the same concern: pacing dispatch so a predictable 429 never arrives. Both live here. flowchart TD A[Attempt target] --> B{Success?} B -->|no, retryable| C[Retry] C --> A B -->|no, exhausted| D[Fallback target] D --> A B -->|context overflow| E[context_window_targets] E --> A B -->|content filter| F[content_policy_targets] F --> A A Route Is a Policy Object route = ai.Route( targets=("anthropic:claude-sonnet-4-5", "openai:gpt-5", "ollama:qwen3:8b"), retry=ai.Retry(max_attempts=3, backoff_base_s=0.5, backoff_max_s=30.0), health_gate=True, health_ttl_s=30.0, ) result = client.generate(prompt, route=route) Targets are tried in order. Each gets up to max_attempts tries before the router moves on. There is no scoring, load balancing, or adaptive selection; Route is a policy object precisely so smarter selection could be added later without changing any client method. Naming a Target Does Not Discard Your Policy A route configured on the client governs calls that do not name a route of their own, and it keeps governing them when a call redirects itself with target=: client = ai.AsyncClient(providers, route=ai.Route( targets=("anthropic:claude-sonnet-4-5",), retry=ai.Retry(max_attempts=5), )) # Still five attempts. `target=` changed where the call goes, not how it is governed. result = await client.generate(prompt, target="openai:gpt-5") The same holds for target-shaped spellings of route= (a single string, or a sequence of them) and for a session's target: they name targets and say nothing about policy, so the policy in force carries. To depart from the client's defaults, pass a fully constructed Route; that is a complete statement of policy, honored exactly as written. The specialized chains are the exception: context_window_targets and content_policy_targets name other providers, and quietly redirecting to a target the caller did not ask for would be the same surprise pointing the other way. They are never inherited by a call that names its own target. What Gets Retried The default predicate declines failures that repetition cannot fix, since retrying a deterministic failure burns budget a transient one might have needed: Failure Retried? RateLimitError (429) Yes, honoring Retry-After TransportError (timeout, connection) Yes ProviderUnavailableError (5xx) Yes AuthError (401/403) No; the same key will fail the same way ContextLengthError No; the same prompt is the same size ModelNotFoundError (404) No Override it when you know better: ai.Retry(retry_on=lambda error: error.http_status == 503) Backoff is exponential from backoff_base_s, raised to the server's Retry-After when that is longer, and capped by backoff_max_s. The error catalog records the retry semantics of every error type. Failure-Specific Fallback Chains The right next target depends on why the last one failed. A context overflow needs a larger model, not another same-sized one, and a content-policy refusal needs a differently-governed provider, not a retry: route = ai.Route( targets=("openai:gpt-5-mini",), context_window_targets=("anthropic:claude-sonnet-4-5",), # bigger context content_policy_targets=("ollama:qwen3:8b",), # different governance ) On a ContextLengthError, the router switches to context_window_targets instead of continuing down the general chain. When a generation finishes with finish_reason == "content_filter", the router discards the refusal and redirects to content_policy_targets, at most once per request, and never after streamed text from the refusing attempt has reached the consumer, since a silent restart would contradict what was already rendered. The redirected attempt is recorded with outcome "redirected". If the chain refuses too, that refusal surfaces normally. Health Gating A target that recently failed with a transport or availability error is skipped for health_ttl_s seconds (the health gate) rather than costing every subsequent request its full timeout: result.attempts # [AttemptRecord(target=..., outcome="skipped_unhealthy"), # AttemptRecord(target=..., outcome="ok")] The TTL is short on purpose: a stale "unhealthy" verdict costs more than one extra failed attempt. Health state is keyed per provider:model, so one bad model does not gate a provider's others. Disable it with health_gate=False when every target should be attempted. The Attempt Trail Every result carries its complete routing history: for attempt in result.attempts: print(attempt.target, attempt.outcome) if attempt.error: print(" ", attempt.error.type_name, attempt.error.detail) Outcomes are "ok", "retried", "failed", "skipped_unhealthy", or "redirected". This is what makes "why was that request slow?" answerable in production. When everything fails: try: result = client.generate(prompt, route=route) except ai.AllTargetsFailedError as error: for attempt in error.attempts: log.warning("%s: %s", attempt.target, attempt.error and attempt.error.detail) What Is Not a Routing Failure A schema violation raises SchemaViolationError directly and does not trigger fallback. The request reached the model and the model answered; it just answered the wrong shape, and sending it to a different provider addresses the wrong problem. Use repair for that. Note that embedding and rerank routes fall back under a stricter rule, because two models' vectors are not interchangeable. Similarly, a mid-stream protocol error after content has been emitted is raised rather than retried: the consumer has already seen text, and a silent restart would duplicate or contradict it. Pacing Before the Limit Everything above reacts to failure. Rate limiting anticipates one kind: an asyncio.gather over a hundred requests would otherwise send a hundred requests, take a wall of 429s, and only then back off. Client-side pacing is opt-in: with no limits configured, requests dispatch exactly as before. Limits belong to a provider instance, not to the application, because a rate limit is a property of an account at a provider. Two instances on two keys have two independent allowances: client = ai.Client( [ ai.ProviderSettings.of( "openai", api_key="env://OPENAI_API_KEY", limits=ai.RateLimits(max_concurrent=8, requests_per_minute=300), ), ] ) The same limits block appears in the shared configuration file, so the CLI and sidecar pace identically. Field Default What it does max_concurrent unbounded Most requests in flight at once. The permit is held for the whole exchange, streaming included requests_per_minute unset Sustained rate, enforced as a token bucket, so a small burst is allowed and then paced min_interval_s 0 Smallest gap between two dispatches, for providers that object to bursts regardless of rate respect_headers true Slow down when the provider's own headers say its window is nearly spent reserve_fraction 0 Fraction of the provider's stated allowance to leave untouched reserve_fraction matters whenever this process is not the only consumer of the key: spending down to the last request in a window means whichever other consumer arrives next is the one that gets throttled. Pacing is bounded to one process. There is no shared state across workers or hosts, no quota enforcement beyond what is configured, and no routing around a busy provider; choosing a different target because one is throttled would be load balancing, which AnyInfer does not do. Learning from the Provider Providers publish their remaining allowance in response headers. Which headers a provider uses is declared on its descriptor and recorded in its contract snapshot (OpenAI uses durations like 6m0s; Anthropic uses RFC 3339 instants). Every derived wait is clamped, so a skewed clock costs a bounded pause rather than a hang. A provider that declares no header dialect is paced by the configured bounds alone. Asking for respect_headers where it cannot work produces a ParameterDropped event saying so. Seeing the Wait A paced request looks slow, so the wait is reported in the result and in the event stream: result.timing.phases.get("queued_ms") # present only when this request waited def observer(event): if isinstance(event, ai.RateLimitWaited): print(f"{event.provider_id} held a request {event.waited_s:.2f}s ({event.reason})") client.events.subscribe(observer) reason is one of concurrency, interval, or provider-headers, so a slow fan-out can be attributed to the bound that caused it. anyinfer doctor prints the configured limits for the same reason. One interaction worth knowing: queue time counts against the request's own timeout_s. Aggressive pacing and a tight timeout will fight each other, so raise timeout_s when pacing hard. Key Takeaways Targets are tried in order with no scoring or load balancing; only failures that repetition can plausibly fix are retried. Per-call target= changes where a request goes, not how it is governed; the client route's retry and health policy still apply. Every result carries its full attempt trail, and every pacing wait appears as queued_ms and a typed event, so slowness is attributable after the fact. Rate limits are opt-in, per provider instance, and pace one process; they never reroute a request or invent a quota the provider did not state. See Also The event stream: AttemptFailed and RateLimitWaited during a request. Error catalog: every error and its retry semantics. Cost and spending: the other ceiling a client can carry. --- # Concepts / Structured Output Source: https://anyinfer.dev/concepts/structured-output/ Structured Output A schema is a contract, not a hint. Passing one returns a value that satisfies it, or an error explaining why not; never a "mostly right" string to re-parse. flowchart TD A[SchemaSpec] --> B{native mechanism} B -->|grammar| C[llama.cpp GBNF] B -->|json_schema| D[OpenAI text.format] B -->|json_mode| E[plain JSON] B -->|prompt| F[instruction text] C --> G{valid?} D --> G E --> G F --> G G -->|no| H[repair, retry] H --> A PERSON = { "type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"], } result = client.generate(prompt, target="medium", schema=PERSON) result.structured # {"name": "Ada", "age": 36} — already validated Pydantic models work too, via duck typing (AnyInfer takes no pydantic dependency): result = client.generate(prompt, target="medium", schema=MyPydanticModel) How It Works Three steps, and the third is the one that matters. 1. Pick the strongest mechanism the model supports. grammar > json_schema > json_mode > prompt Mechanism What it does Who has it grammar Constrains decoding so invalid tokens cannot be produced llama.cpp, Ollama json_schema Provider validates against the schema OpenAI, Azure, Anthropic (emulated) json_mode Provider guarantees some valid JSON Several prompt The schema is described in the system prompt Everywhere Unknown capabilities fall to prompt, which works everywhere; an unrecognized model degrades to something that still produces a validated result rather than failing outright. The chosen mechanism is recorded on the result: result.structured_mechanism # "grammar" 2. Project the schema for that provider. Grammar-based engines choke on constructs that are cheap for a validator but expensive for a grammar: minLength/maxLength on strings, and minItems/maxItems of 2000 or more, are stripped for the wire only. If a local model keeps failing a length constraint, that is why: the constraint never reached the engine, and clearer prompt wording will help more than a tighter bound. Two things improve results under every mechanism: prefer enum over free-form strings, and keep nesting shallow. 3. Validate the response against the original schema. Always. Regardless of mechanism, regardless of what the provider claimed. This is non-negotiable for a specific reason: backends have shipped bugs where structured-output enforcement is silently disabled under certain conditions (thinking modes, in particular), producing unconstrained output with no error. A provider's claim that it constrained the output is not evidence that it did. Stripping a constraint in step 2 therefore never weakens what comes back: the original schema is what decides. Grammar Mode Still Describes the Schema in the Prompt A GBNF grammar guarantees well-formed JSON, not meaningful JSON. A model that was never shown the schema will happily emit schema-shaped nonsense that satisfies the grammar and fails the caller. So for engines whose grammar mode does not condition the model (llama.cpp and Ollama), AnyInfer injects the schema into the prompt as well as compiling the grammar. This is a descriptor property, not a blanket rule: providers whose json_schema mode already conditions the model do not need it. Repair Opt in to letting the model correct itself; see repair in the glossary: result = client.generate( prompt, target="medium", schema=PERSON, repair=ai.Repair(max_attempts=1), ) result.repair_attempts # 0 if it got it right the first time On a violation, AnyInfer re-prompts the same model with the validation errors appended. It does not fall back to a different provider: a schema violation says something about the model's output, not the endpoint's health, and fallback would spend the routing budget on a problem fallback cannot fix. Each attempt is a full additional request. A repair budget is therefore also a latency budget, worth accounting for on latency-sensitive paths. The budget is bounded. When it is exhausted: try: result = client.generate(prompt, schema=PERSON, target="medium") except ai.SchemaViolationError as error: print(error.errors) # ("age: 'age' is a required property",) print(error.raw_text) # what the model actually said The error carries the validation errors, bounded raw output, and (when a truncated top-level JSON object contains delimiter-confirmed complete members) error.partial plus error.missing_fields. Partial members are evidence, not a valid result: AnyInfer never guesses a cut-off scalar, asks another provider to continue it, or treats recovered fields as schema-validated. The budget is the caller's to set for almost every provider. A few declare a ceiling (Microsoft 365 Copilot allows one repair attempt, because each is a full Graph round trip), and the clamp is never silent: asking for more emits a ParameterDropped event naming repair.max_attempts. Extraction Is Forgiving; Validation Is Not Models wrap JSON in code fences and prose even when told not to. Extraction handles that: it tries the whole string, then scans for the first balanced {...} or [...], respecting string literals so a brace inside a string does not end the scan. Validation, once a value is extracted, is strict. Key Takeaways A schema is a contract: the caller always gets a client-side-validated value, regardless of which mechanism produced it. The strongest native mechanism is chosen automatically: grammar beats json_schema beats json_mode beats prompt, and the choice is recorded on the result. Repair is opt-in, bounded, and re-prompts the same model; it never triggers fallback. See Also Capabilities: how mechanism selection knows what a model supports. How-to: enforce a JSON schema --- # Concepts / Sessions Source: https://anyinfer.dev/concepts/sessions/ Sessions Every request is independent by default, which keeps results reproducible and fallback safe. A conversation is the case where that default wastes work: the provider often already has everything the next turn needs. A session is how a caller says these requests belong together, without having to know what any particular provider does about it. with client.session("copilot:auto") as chat: client.generate("Summarize this report.", session=chat) client.generate("Now list the risks.", session=chat) What Each Provider Actually Saves The providers that can carry state between turns save completely different things, which is why the handle is opaque rather than a conversation object: Provider What an open session keeps What that saves GitHub Copilot The conversation, server-side Prior turns are not re-sent at all: fewer tokens billed, and no duplicated history llama.cpp The supervised server, pinned No model load between turns, and the KV cache the next turn reuses survives Ollama The model, resident (keep_alive) No reload of several gigabytes of weights mid-conversation Everything else treats a session as inert. A Session Never Changes an Answer It is a performance and cost optimization, and holding that line is what makes it safe to pass one everywhere. Opening a session against a provider that cannot keep state is allowed and does nothing: session = client.session("openai:gpt-5") session.supported # False session.reuse # 'unsupported', and every request behaves exactly as it would have Reuse Is Reported, Not Assumed reuse says what happened on the last turn, not what was hoped for: Value Meaning fresh The provider started new state: the first turn, or one it had already expired. resumed The provider continued state it already held. unsupported Nothing was reused: this provider cannot, or that turn went somewhere else. State Is Bound to One Target Provider state is not portable, so a session names the target it belongs to and applies only there. If a route falls back to a different provider, or a different model on the same one, that turn simply runs without it and reports unsupported: result = client.generate( "and the risks?", route=ai.Route(targets=("ollama:qwen3:8b", "openai:gpt-5")), session=chat, # chat belongs to ollama:qwen3:8b ) chat.reuse # 'unsupported' if the fallback answered Because a session already names a target, a caller can leave the target off entirely and it stands in; it never overrides an explicit target or route. Closing Is Local close() stops the handle being used; it does not reach out to the provider. Server-side state expires on the provider's own schedule (Ollama's keep_alive timer, Copilot's service-side session lifetime). Closing the client itself does release what an adapter holds open locally, such as a Copilot SDK session. Key Takeaways A session is an opaque handle, not a conversation: the library never interprets what a provider stores in it. It never changes an answer, so passing one to a stateless provider is safe and inert. reuse reports what the provider actually did, including when a fallback meant the session did not apply. See Also Routing and rate limits: what happens to a session when a route falls back. The local subsystem: why pinning a supervised server matters. Capabilities and provenance: supports_sessions on the descriptor. --- # Concepts / Embeddings and Reranking Source: https://anyinfer.dev/concepts/embeddings/ Embeddings and Reranking Embedding and reranking are stateless inference operations, typed and routed the same way generation is, but they are not generation. EmbeddingRequest and RerankRequest are their own types; nothing here is ever added as a field on GenerationRequest. import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("ollama")]) result = client.embed( ["What is the capital of France?", "Paris is the capital of France."], target="ollama:nomic-embed-text", ) print(len(result.vectors), result.space.dimensions) ranked = client.rerank( "What is the capital of France?", ["Paris is the capital of France.", "Berlin is the capital of Germany."], target="ollama:some-rerank-model", ) for item in ranked.items: print(item.document_id, item.score) Both accept a single target, a fallback chain, or a route the same way generate() does. AnyInfer produces vectors and relevance rankings; it does not persist them, build a search index, or crawl a corpus. An application brings its own store and feeds it with these results, or uses the small optional anyinfer-store add-on, which draws the boundary in detail. The Embedding-Space Safety Rule Two embedding vectors are only meaningfully comparable when they came from the same model, the same revision, and (for models that distinguish it) the same input-intent handling. A query re-embedded by a fallback model produces numbers that look exactly as plausible as the primary model's, and will fail to match anything in an index built against the primary. Nothing in the response says it happened, which makes this failure worse than an ordinary provider error. AnyInfer's answer is EmbeddingSpace, carried on every EmbeddingResult: print(result.space.provider_id, result.space.model, result.space.dimensions) By default, embedding routes retry on the same resolved target only: no cross-provider or cross-model fallback. A chain that reaches a different provider:model is refused before any request is sent, with an actionable ConfigError, because AnyInfer never guesses that two spaces are equivalent. A caller that genuinely wants vectors that may not be comparable passes allow_incompatible_fallback=True; the result then always carries a warning naming both targets. A caller can also assert the expected space up front: result = client.embed( ["hello"], target="ollama:nomic-embed-text", expected_space=my_stored_index_space, ) A successful response from a target that does not match expected_space is rejected rather than returned. Input Intent Some embedding models produce measurably better retrieval when a query and the documents it will be compared against are embedded with different instructions: query_vec = client.embed(["capital of France"], target="ollama:nomic-embed-text", input_type="query") doc_vecs = client.embed(docs, target="ollama:nomic-embed-text", input_type="document") A provider that does not distinguish input intent ignores the field. A model that requires it but received none degrades per its own documented default, and that degradation is recorded as a warning. Reranking and Document Identity RerankDocument ids are caller-owned and opaque; AnyInfer never generates or interprets them. Every RankedItem carries back the original index and the document id it was given, so a caller can always map a ranked result back to its source, and a malformed provider response (an out-of-range or duplicate index) is rejected rather than guessed at. Scores are meaningful only within one result from one target. They are never comparable across providers or models, and AnyInfer never merges or averages scores from separate rerank attempts. Batching Providers disagree on how many inputs, documents, tokens, or bytes one request may carry. Splitting an oversized request is core policy, never an adapter's own decision, and it only happens against a verified limit: An oversized embedding request against a target with a verified batch limit is split into ordered chunks, dispatched concurrently, and re-assembled in input order. A batch failure is all-or-error; a caller never gets back an EmbeddingResult silently missing vectors. When no verified limit exists, a request up to a bounded default goes out as a single call, and anything larger is refused with an actionable error rather than a guessed provider maximum. Reranking is not automatically split, because scores from separate document batches are not globally comparable unless a provider documents otherwise. The batch= parameter takes an anyinfer.BatchPolicy: max_concurrency bounds parallel chunks, allow_split=False refuses splitting outright, max_items_override supplies a limit the caller has verified, and rerank_cross_batch=True is the explicit opt-in for chunk-local rerank rankings, with a warning that the scores are not one global ordering. Frontends The sidecar exposes POST /v1/embeddings as an OpenAI-compatible codec and POST /v1/anyinfer/rerank as an AnyInfer-native route (there is no established OpenAI-shaped rerank dialect to emulate). The CLI exposes anyinfer embed and anyinfer rerank. All three surfaces are projections over the same AsyncClient calls. Key Takeaways Embedding and rerank requests are their own typed operations with their own routing rule: same resolved target only, unless incompatible fallback is explicitly allowed. Every result carries its EmbeddingSpace, and expected_space= turns a stored index's assumptions into an enforced precondition. Batch splitting happens only against verified limits and is all-or-error; rerank scores are never merged across batches. Storage and search stay outside the core; anyinfer-store is the optional add-on for having them without a database. See Also Semantic search over a small corpus: the runnable example. Embed, store, and query: the optional persistence add-on. Text Embeddings Inference · Voyage AI and Jina AI: the retrieval-only providers. --- # Concepts / Multimodal Inputs Source: https://anyinfer.dev/concepts/multimodal-inputs/ Multimodal Inputs AnyInfer generation requests may contain images, documents, and audio alongside text. The output is still text and tool calls; this does not add image generation, speech output, transcription, or another inference API. from pathlib import Path import anyinfer as ai message = ai.Message( role="user", content=( ai.Text("Explain this diagram and summarize the report."), ai.ImagePart(data=Path("diagram.png").read_bytes(), media_type="image/png"), ai.DocumentPart( data=Path("report.pdf").read_bytes(), media_type="application/pdf", filename="report.pdf", ), ), ) result = client.generate(message, target="openai:gpt-5.6") ImagePart and DocumentPart accept either inline bytes or a remote url, never both. AudioPart accepts inline bytes. Bytes stay unencoded in the domain model; each adapter base64-encodes them only when its wire protocol requires it. The default ceilings are 20 MiB for one inline part and 50 MiB for the whole request. generate() and stream() accept max_input_part_bytes and max_input_bytes for a tighter bound. A violation is rejected before a provider call. Capabilities and Conservative Budgets The capability flags are VISION, DOCUMENT, and AUDIO_IN. A trusted capability record that lacks a required flag refuses the request before dispatch. Unknown or defaulted capability data does not pretend to be an authoritative "no"; the adapter still either projects the part or raises an explicit unsupported-input error. Image and document token costs depend on provider formulas, resolution, page count, and model. When no catalog formula is available, budget.estimate.unpriced_parts reports the gap and both budget.fits and budget.estimated_cost are None; the context gate lets the request through instead of inventing a token count. Frontends The CLI reads attachments in the frontend and passes bytes to the same typed request: anyinfer run "What is important here?" --image diagram.png --document report.pdf The sidecar preserves OpenAI message content arrays containing image_url, input_audio, and file parts, subject to the same request caps. No attachment content appears in telemetry, manifests, or error messages. Which Providers Accept What Support varies by part type, provider, and model. The conformance matrix records per-provider multimodal support from actual test runs, and each provider page states its own quirks: for example, Ollama takes inline images only, and the supervised llama.cpp path needs a catalog artifact with a pinned projector companion. Provider contract snapshots record the wire spellings and their verification sources. Key Takeaways Parts are typed (ImagePart, DocumentPart, AudioPart) and size-capped before any provider call; adapters handle wire encoding. A trusted capability record gates multimodal requests before dispatch; an unknown one defers to the adapter's explicit accept-or-raise. Unpriceable parts make the budget honest: fits and estimated_cost go None instead of guessing. Per-provider support lives in the conformance matrix, not in a hand-maintained list here. See Also Conformance matrix: per-provider multimodal support, from test runs. Capabilities and provenance: the flags that gate dispatch. Token estimation and context budgets: how unpriceable parts are reported. --- # Concepts / Capabilities and Provenance Source: https://anyinfer.dev/concepts/capabilities/ Capabilities and Provenance Every capability value records where it came from: its provenance. Providers omit, misreport, and change these numbers, so before routing, budgeting, or billing against one, a consumer needs to know how much to trust it. flowchart LR A[default] --> B[catalog] B --> C[discovered] C --> D[probed] D --> E[override] E --> F[assembled capability] caps.context_window # Sourced(value=128000, provenance='discovered') The Five Provenances Weakest to strongest: Provenance Meaning default A descriptor-level fallback. A placeholder, not a fact. catalog From bundled static data the project maintains, including the pricing table. discovered Reported by the provider's own model listing. probed Measured by an opt-in probe that spent a real request. override Set by the integrating application. Outranks everything. Assembly layers them in that order, field by field. A weaker value never displaces a stronger one, and unknown stays None rather than becoming a guess: caps = ModelCapabilities(context_window=Sourced(8192, "catalog")) caps = caps.overlay(ModelCapabilities(context_window=Sourced(32768, "discovered"))) caps.context_window # Sourced(32768, 'discovered'); discovery wins Some providers report rich model listings. OpenRouter includes per-model pricing, Nebius reports context and quantization, and xAI reports feature support; those values arrive at discovered provenance and beat the catalog. Where a listing is unavailable, assembly degrades to the weaker layers rather than failing. What Capabilities Drive Structured-output mechanism selection. The feature flags decide grammar vs json_schema vs JSON mode vs prompt; see structured output. Cost computation. Only trusted-provenance pricing produces money; see cost and spending. Pre-dispatch gating. A request that provably cannot fit a known context window fails fast instead of paying a round trip. Only trusted-provenance windows gate; see context budgets. Probe sizing. A target known to reason gets a larger budget for the verify() probe, since a thinking model spends the ordinary one before it says anything. default_temperature and default_top_p record what "provider default" concretely means for a model, populated only from the provider's own documentation via its contract snapshot. Almost every provider answers None, and that is the finished state, not a gap: inventing a plausible number would defeat the point of tagging where numbers come from. Overriding Capabilities capability_overrides applies the application's own numbers at override provenance, the strongest layer, so a deliberate correction never loses to data the library merely collected: client = ai.Client( providers, capability_overrides={ "azure-foundry:my-gpt5-deployment": ai.ModelCapabilities( pricing=ai.Sourced(ai.Pricing(Decimal("1.10"), Decimal("9"))), context_window=ai.Sourced(400_000), ), }, ) Provenance on the supplied fields is stamped automatically; supplying them is the provenance. The auto Sentinel Some providers pick the model at request time (GitHub Copilot's "auto"). The only safe capability claim is then the conjunction across every model the provider might choose: the minimum of each numeric bound, the intersection of feature flags. caps = conjunction([gpt_5_caps, gpt_41_caps]) caps.context_window # the smaller of the two caps.features # only features both support If any candidate's bound is unknown, the conjunction is unknown; a minimum cannot be promised without knowing every value. Three States, Not Two A capability is natively supported, emulated by the core (a schema prompt-injected for a provider with no structured-output mode, say), or explicitly unavailable. The fourth state (a parameter accepted, discarded, and reported as success) is the one AnyInfer refuses to have: temperature=0 that had no effect looks exactly like temperature=0 that worked. Known drops are declared on the descriptor and reported as ParameterDropped telemetry instead of sent. The same rule applies per model. A descriptor knows how a provider spells reasoning effort; it does not know which of that provider's models have one. A request carrying reasoning="high" to a model whose capabilities lack Feature.REASONING withholds the field and reports it. Both only happen on a known absence. A default-provenance feature set is a guess, and the library does not drop a caller's parameter on a guess (the same rule as the pre-dispatch gate). Proving a Target Works Three mechanisms answer "will this target actually serve my request?", from cheapest to most thorough. resolve() proves the spelling. It maps a target string or alias to a concrete provider and model, or raises with a hint; see targets and aliases. No network traffic. verify() proves one round trip. Resolution says nothing about whether the credential can generate, the model id exists at that endpoint, or the deployment has capacity; a health probe does not either, since everything a health probe touches can be fine while inference still fails. verify() spends one tiny request and reports rather than raises: result = client.verify("openai:gpt-5") result.ok # answered, in the shape asked for, with the expected content result.reached # answered at all result.detail # what went wrong, when something did result.target # which model actually served it — meaningful for "auto" The two booleans are separate because the fixes are different: reached ok What it means False False Nothing answered. Wrong endpoint, bad credential, no capacity. True False The connection is fine; the model could not hold the requested shape. True True Good. The CLI wraps the same call as anyinfer verify. probe() measures features. On the compatibility surface, every preset endpoint and self-hosted server starts from an educated guess, and a server that accepts response_format while ignoring it is indistinguishable from one that honors it — until a schema stops being enforced. probe() settles it by trying, one tiny request per feature: report = client.probe("openai-compat:m") # four requests by default report.summary # 'openai-compat:m: supports JSON_MODE, STREAMING; does not support JSON_SCHEMA' Findings record at probed provenance, so the next request stops guessing. Pass record=False to look without committing. Outcomes are three-state: supported, unsupported, and inconclusive (the provider accepted the request and answered something else). Inconclusive results are not recorded, because one reply cannot separate a weak model from an ignored parameter. Probing costs requests Four round trips for the default feature set, billed like any other. Run it once when an application first configures an endpoint, not on every start. Runtime Diagnostics A capability says what a model can do, not what state the engine is in right now. The worst local-inference surprise lives in that gap: the request succeeded, the answer is correct, and it took ninety seconds because the model no longer fits in VRAM and half of it ran on the CPU. No health probe catches that; the server is perfectly reachable. Providers that can inspect their own runtime report it: for note in client.diagnostics("ollama"): print(note.code, note.message) # ollama.gpu-spill qwen3:8b is only 45% resident in VRAM; the rest runs on the CPU ... The same notes arrive on every result that hit the condition, and as ProviderDiagnostic telemetry: result = client.generate(prompt, target="ollama:qwen3:8b") result.warnings # ("qwen3:8b is only 45% resident in VRAM; ...",) Which providers can answer is declared on the descriptor (reports_diagnostics). Today that is Ollama (VRAM spill) and llama.cpp (a GPU machine serving on the CPU). Diagnostics are advisory: they never fail a request and never gate routing. Inspecting Capabilities for model in client.models("openrouter"): caps = model.capabilities if caps and caps.context_window: print(model.id, caps.context_window.value, f"({caps.context_window.provenance})") Key Takeaways Every capability value carries provenance (default, catalog, discovered, probed, or override), and assembly never lets a weaker source displace a stronger one. Unknown stays None. Nothing is upgraded from "assumed" to "known" without a listing, a probe, or an override. resolve() checks spelling for free, verify() spends one request to prove a round trip, and probe() spends ~4 to measure features on compatibility endpoints. Parameters are only withheld on a known absence, and every withholding is reported as ParameterDropped. See Also Cost and spending: tri-state cost and where prices come from. Structured output: the mechanism ladder the feature flags drive. Token estimation and context budgets: what the context window gates. Telemetry: ParameterDropped and ProviderDiagnostic events. --- # Concepts / Token Estimation and Context Budgets Source: https://anyinfer.dev/concepts/budgeting/ Token Estimation and Context Budgets How many input tokens will this request spend, and does it fit the model it is going to? Every app that assembles large prompts ends up hand-rolling this arithmetic per provider. AnyInfer answers it once, against the same provenance-tagged capability data that drives routing and cost. The result is not isolated bookkeeping: it feeds the context reducer, the pre-dispatch gate, cost planning, and the router's context-overflow chain, so changing the target changes all four from the same capability record. flowchart LR A[messages] --> B[estimate: tokens, floor] B --> C{trusted window known?} C -->|no| D[fits = None] C -->|yes| E{floor > window?} E -->|yes| F[ContextLengthError] E -->|no| G[fits = True] The Calculator budget() computes a preflight budget without sending anything; no request is issued, no network is touched: budget = client.budget(messages, target="openai:gpt-4.1") budget.input_allowance_tokens # window − output reserve − safety headroom budget.estimate.tokens # estimated input spend, by component budget.remaining_tokens # how much more material still fits budget.fits # True / False / None The allowance is the context window minus two deductions: Output reserve: room for the response. Derived, not flat: a request that sets max_output_tokens reserves exactly that; otherwise the 4,096-token default applies, capped by the model's known maximum output. Safety headroom: 5% of the window, clamped to [256, 8192], held back against estimation error. An app packing context reads remaining_tokens and keeps adding material while it stays positive. That is the whole loop. Unknown Stays Unknown The verdict is tri-state, exactly like cost. When no trustworthy context window is known, fits and the allowances are None, never a guessed default window presented as a bound: budget.fits # None; unknown, distinguishable from both True and False Estimates Are Two Numbers No tokenizer ships in the core. The default estimator is a byte heuristic, and it is explicit about being an estimate by carrying two figures with opposite biases: Figure Bias Used for tokens High (ceil(bytes/3)) Planning; deciding how much more fits. floor Low (bytes//8) The pre-dispatch gate; refusing a request. The two consumers need opposite errors: when packing, overestimating keeps the application safe; when refusing, only an underestimate justifies the refusal. Anything more accurate plugs in through the TokenEstimator protocol; tiktoken, a provider's count-tokens endpoint, llama-server's /tokenize. An exact tokenizer returns floor == tokens, which gives the gate full force: class TiktokenEstimator: def estimate(self, text: str) -> ai.TokenEstimate: count = len(encoding.encode(text)) return ai.TokenEstimate(count, count) client = ai.Client(providers, estimator=TiktokenEstimator()) When the Provider Bills for More Than You Sent Some providers wrap your messages in a harness of their own (an agent preamble, built-in tool declarations, workspace framing), then bill and window-check the inflated total. Estimating such a provider from message bytes alone under-counts every request, so the provider declares its own correction and the budget reports it as a separate component: budget = client.budget(messages, target="copilot:auto") budget.estimate.messages.tokens # what you sent budget.estimate.envelope.tokens # what the provider wraps around it GitHub Copilot is the case in the shipped registry. The correction moves the planning figure only, never the floor: a lower bound may only claim tokens the provider certainly charges, so a calibrated provider packs more conservatively without ever refusing a request it might have served. Estimated Cost When trustworthy pricing exists for the target (see where prices come from), the budget also carries a preflight cost range: budget.estimated_cost # CostEstimate(low=..., high=..., currency="USD") or None low prices the estimate's floor with zero output: the least the request can cost. high prices the planning estimate plus the full output reserve: a ceiling under the budget's own assumptions. It is a range on purpose: the input estimate is two-sided and the output spend is unknown until the model stops, so one number would be false precision. And it is tri-state like everything else; no trusted pricing means None, never $0.00. Estimated money and reported money never mix: result.usage.cost_usd is only ever computed from provider-reported usage, and estimated_cost only ever from the estimate. The Pre-Dispatch Gate A request that provably cannot fit its target's context window fails before the round trip. The gate is conservative in what it claims: Only trusted-provenance windows gate (catalog, discovered, probed, override). A default window is a placeholder, and a placeholder never blocks a request. Only the estimate's floor gates, compared against the whole window: no reserve, no headroom. A heuristic overestimate can never refuse a request that might have fit. A gated target raises ContextLengthError, the same class a provider would return, so the route's overflow chain redirects identically either way, minus the latency: route = ai.Route( targets=("openai:gpt-4.1-mini",), context_window_targets=("openai:gpt-4.1",), # where overflow goes instead ) The gate is on by default and can be disabled per client with Client(..., context_gate=False). Key Takeaways budget() never touches the network: it is a preflight calculation against the same provenance-tagged capability data that drives routing and cost. The verdict is tri-state: fits is None, not a guess, when no trusted context window exists. Only the estimate's conservative floor, compared against a trusted window, can gate a request before dispatch. See Also Capabilities and provenance: where the context window comes from and why its provenance decides whether it may gate. Cost and spending: the ceiling the preflight range is checked against. Routing and rate limits: context_window_targets and the rest of the fallback model. Context reduction: packing material against remaining_tokens. --- # Concepts / Cost and Spending Source: https://anyinfer.dev/concepts/cost/ Cost and Spending AnyInfer computes what each call cost, keeps a per-client spend ledger, and can refuse a request before it crosses a ceiling. One rule underlies all three: an unknown cost is reported as unknown, never rendered as zero. Cost Is Tri-State State Meaning How it renders A number Computed from trusted pricing and provider-reported usage 0.004125 None Unknown: no pricing, or pricing that is not trusted unknown, never $0.00 Decimal(0) Genuinely free, as local inference is 0.000000 Rendering an unknown cost as $0.00 turns a reporting gap into a silent accounting error. Since AnyInfer will not do that, every total below carries a count of the calls it could not price alongside the ones it could. Cost is computed centrally, from pricing whose provenance is trusted (catalog, discovered, probed, or override), so every provider reports it identically. A descriptor-level fallback price is a placeholder and never produces money. Where Prices Come From A bundled pricing table supplies the catalog layer for hosted models. Each entry records when it was last verified and against what source; a weekly repository check watches for drift, and fetch_pricing() pulls the maintained file for numbers newer than the installed release. Prices are keyed by provider and model, because the same model served by a different engine may cost differently. On top of the table: OpenRouter reports real per-token pricing in its model listing, so its costs carry discovered provenance and beat the table. Local engines (Ollama, llama.cpp) get a genuine Pricing(0, 0): free inference is a real zero, not an unknown. Azure AI Foundry and the Copilots ship no table entries: Foundry pricing is region- and deployment-specific, and Copilot bills by subscription rather than per token. Their costs stay None unless overridden. How the bundled table is checked for drift is a contributor concern; see the scheduled repository checks. What One Call Cost result = client.generate(prompt, target="anthropic:claude-sonnet-4-5") result.usage.cost_usd # Decimal, or None result.usage.input_tokens result.usage.cache_read_tokens # served from the provider's prompt cache Check cost_usd for None before formatting it. The cache_read_tokens field is how prompt caching shows up in the bill. What This Client Has Spent ledger = ai.SpendLedger() client = ai.Client(providers, ledger=ledger) ... totals = client.spend() print(totals.cost, totals.requests, totals.unknown_requests) Read cost together with unknown_requests: a total that omits the calls it could not price understates spend while looking authoritative. if not totals.complete: print(f"{totals.unknown_requests} of {totals.requests} calls could not be priced") Break spending down by target, or by the application's own labels: client.generate(prompt, target=..., metadata={"tenant": "acme", "feature": "summarize"}) ledger.by_target() # {"anthropic:claude-sonnet-4-5": SpendTotals(...)} ledger.by_label("tenant") # {"acme": SpendTotals(...), "globex": SpendTotals(...)} The library never interprets the labels. Tenant, feature, job id: that vocabulary is the application's, carried through untouched. There is no process-wide ledger. Two clients that should share a total are given the same SpendLedger object; a global would make totals depend on import order and would merge the accounting of unrelated libraries sharing a process. Stopping Before You Spend Too Much client = ai.Client( providers, spend=ai.SpendPolicy(max_total_usd=Decimal("25"), max_request_usd=Decimal("0.50")), ) A ceiling is checked before dispatch, beside the context gate, so a refusal costs nothing. Crossing it raises SpendLimitError, which carries the ceiling, what had already been spent, and the estimate that tripped it, so the arithmetic is visible and not just the verdict. The estimate is the high end of the preflight cost range: the pessimistic number, since a guard built on the optimistic one would admit requests it was meant to stop. When the Cost Cannot Be Known ai.SpendPolicy(max_total_usd=Decimal("25"), on_unknown="refuse") allow is the default: a target with no trusted pricing is sent, which preserves the behavior that existed before ceilings did. refuse is for callers who would rather fail than spend blind. There is no option that treats unknown as zero, because a guard that does that enforces nothing while appearing to. A refusal is not a routing signal. It leaves the router entirely rather than falling back to a cheaper target: a ceiling is client-wide, so a different target does not satisfy it. Keeping a Total Across Restarts The library writes nothing on its own. For durability, own the file: store = ai.SpendStore("~/.myapp/spend.json") store.accumulate(ledger) # atomic; folds today's ledger into the stored total store.load()["total"].cost Reads are total: a missing, truncated, or foreign file yields nothing rather than raising. Scope This is accounting for one client in one process. It cannot see other processes or other consumers of the same API key. Organization-wide quotas and fleet-level spend controls belong to a deployment around AnyInfer, not inside it. Key Takeaways Cost is tri-state: a number from trusted pricing, a genuine Decimal(0) for local inference, or None for unknown. None is never rendered as $0.00. Read SpendTotals.cost together with unknown_requests; a total is only honest with both. A SpendPolicy ceiling refuses before dispatch, using the high end of the preflight estimate. Unknown-cost targets pass by default; set on_unknown="refuse" to fail instead. Durable totals are opt-in via SpendStore; the library writes nothing on its own. See Also Capabilities and provenance: where pricing trust comes from. Token estimation and context budgets: the preflight estimate a ceiling checks against. Arena runs: comparing targets when you are willing to spend; the portability diff compares without spending. --- # Concepts / Prompt Caching Source: https://anyinfer.dev/concepts/caching/ Prompt Caching Most providers can hold on to a prefix of the prompt and charge less the next time they see it. What "hold on to it" means differs: some want to be told exactly where the reusable part ends, others work it out themselves and want the prefix left undisturbed. AnyInfer treats that difference the way it treats structured output: the caller states an intent (cache what is worth caching) and the core picks the strongest mechanism the target offers, reporting when a weaker one is all that is available. This is not a response cache Prompt caching caches the prompt a request sends, on the provider's side, for the provider's retention window. It never skips a call, never reuses an answer, and never makes a repeated question free. AnyInfer stores nothing. It Is Off Unless You Ask result = client.generate(prompt, target="anthropic:claude-sonnet-4-5") # no caching result = client.generate(prompt, target="...", cache=ai.CachePolicy()) # caching Caching changes what a provider bills and how long it keeps a copy of the prompt, and neither is a decision the library makes on the developer's behalf. Set the policy once on the client (ai.Client(providers, cache=ai.CachePolicy())) or in the shared configuration file, where the CLI and sidecar pick it up too. The Two Mechanisms Explicit: the provider accepts per-segment marks. AnyInfer decides which segments are worth marking, largest first, bounded by the provider's own ceiling; the adapter spells each mark in that provider's wire format. Anthropic works this way. Implicit: the provider caches a stable prefix on its own. There is nothing to send, so AnyInfer's job is to leave the prefix undisturbed and to report when the request itself is defeating it. OpenAI and DeepSeek work this way. When a target offers neither, the policy is reported as dropped via a ParameterDropped event rather than silently ignored. What Gets Marked Three kinds of segment, in the order they sit on the wire: Segment Why it is a good candidate Tool declarations Identical on every turn of a conversation, and often large The system block Stable by construction The conversation prefix Everything before the current turn; grows as the chat does Segments smaller than the provider's floor are skipped: below the floor a mark is billed as a cache write that no later read ever pays back. The policy allows a developer to narrow what is eligible: ai.CachePolicy(include_tools=True, include_system=False, min_segment_tokens=2048, max_marks=2) Seeing What Happened The result reports which mechanism was engaged: result.cache_mechanism # "explicit", "implicit", or None result.usage.cache_read_tokens # what the provider says it served from cache result.usage.cache_write_tokens # what it says it stored cache_mechanism is what was asked for; the usage figures are what the provider reported. Cost is computed only from the reported numbers (an intention is never billed as an outcome), and when a provider does not report cache accounting, the figures stay None rather than becoming zero. Subscribers see a CachePlanned telemetry event carrying the mechanism, the mark count, and the estimated cacheable size. Both events are content-free: positions and counts, never text. Making Caching Actually Work An implicit-caching provider only helps if the prefix is byte-identical between turns. The usual mistakes: a timestamp or request id in the system prompt tools serialized in a different order each time context blocks assembled from a set rather than a list If cache_read_tokens stays at zero while hits are expected, that is where to look. Context reduction renders in path order by default for exactly this reason. Key Takeaways Caching is opt-in, and a policy on a target that supports neither mechanism is reported as dropped, never silently ignored. Intent and outcome are separate fields: cache_mechanism says what was planned, usage.cache_read_tokens says what the provider actually served. Marks below the provider's minimum segment size cost money instead of saving it, so AnyInfer skips them. Implicit caching lives or dies on a byte-identical prefix; look for timestamps and unstable serialization order when hits stay at zero. See Also Cost and spending: how cache reads show up in the bill. Context reduction: stable rendering that keeps prefixes identical. Telemetry: CachePlanned and ParameterDropped events. --- # Concepts / Context Reduction Source: https://anyinfer.dev/concepts/context-reduction/ Context Reduction You have more material than the model's window holds. anyinfer.context decides what to send and tells you exactly what it dropped. Together with client.budget(), this makes context preparation part of the inference contract: the selected target supplies the limit, the reducer stays within it, and the result carries a machine-readable account of lost fidelity. flowchart LR A[your corpus] --> B{fits the budget?} B -->|yes| C[whole] B -->|no| D[ranked / tiered / packed] D --> E[envelope + what was omitted] C --> E You Collect, the Library Reduces Your application collects: walking the filesystem, applying ignore rules, excluding secrets, asking the user what to share. That stays yours, because it is where the security policy lives and where every application differs. The library ranks, selects, and represents what you hand it; this subpackage never opens a file, never touches the network, and adds no dependencies. import anyinfer as ai from anyinfer import context # You collected and approved these. docs = [context.ContextDocument.of(path, text) for path, text in my_approved_files] budget = client.budget(messages, target="anthropic:claude-sonnet-4-5") reduction = context.select( docs, query="how does credential resolution work?", max_tokens=budget.remaining_tokens or 8_000, ) messages.insert(0, ai.user(reduction.text)) print(reduction.summary()) # ranked: 12 of 340 document(s); ~7900 of 8000 tokens; 328 omitted; limited by tokens One question comes up before any of this: can't the material just go in several messages? No. Every message in a request shares one context window, so splitting the same material across ten messages sends exactly as many tokens as one. What works is either less fidelity in one request (ranked, tiered, packed) or more requests (distill). The Five Strategies Strategy Sends Use when whole Everything The corpus fits. Nothing to decide. ranked The most relevant whole documents You want full files, and partial ones would confuse tiered Every document, at decreasing fidelity The model should know the whole corpus exists packed The most relevant chunks The answer is one function in a large file distill A summary written by the model The corpus will never fit at any fidelity auto, the default, sends everything when it fits and falls back to tiered. Not sure which? plan() costs all four and tells you. tiered answers "how do I say something about every file?" with three tiers, each cheaper per document: a module rollup (one entry per directory, with a reserved budget share so it cannot be crowded out), structural extracts for the highest-ranked documents, and verbatim files for whatever budget remains. A model that knows src/auth/ exists can ask about it; one that never saw it cannot. packed splits documents at paragraph boundaries, ranks every chunk, and packs the best. Adjacent chunks are coalesced when rendered, so a contiguous run appears as one block. Pinned documents are never chunked: pinning means "the user chose this file", and sending a piece of it answers a question they did not ask. distill is the only strategy that issues generation calls, so it is a separate function rather than a select() strategy: result = await context.distill( corpus, "what changed in the release?", client=client, target="anthropic:claude-sonnet-4-5", ) print(result.calls, "calls") # the multiplier, made visible print(result.usage.cost_usd) # what it actually cost It maps each chunk to notes, then reduces the notes to an answer, going hierarchical when the notes exceed the window. A deterministic reducer= replaces the reduce call entirely. See the distill example. Losing Less Than You Drop Real corpora repeat themselves. Byte-identical documents collapse by default: one copy is rendered, the rest become pointers, and nothing is lost. Near-identical documents collapse only on request (near_duplicate_threshold), because the near-duplicate's differences are not sent; that makes reduction.complete false. Pinned documents are never collapsed. Similarity uses MinHash over banded signatures, so thousands of documents cost a linear pass and group identically on every run. A file that just misses the budget can be shortened instead of dropped: compact_fallback retries it with comments, docstrings, license headers, and blank runs removed (a 25–40% saving on real source). Only lines that are entirely a comment are removed, because stripping a trailing // correctly would need a parser this subpackage does not have. Plan Before You Commit plan() runs every deterministic strategy, measures what each would render, and throws the text away. It spends no inference and touches no network, so the numbers are exact: outcome = context.plan(docs, query, max_tokens=8_000) print(outcome.summary()) # 340 document(s) against 8000 tokens; best tiered (46 kept, 0 omitted, ~7900 tokens); # distill would spend 141+ call(s) outcome.best() is a recommendation (most of the corpus at the highest fidelity), not a decision. An app that would rather have twelve whole files than four hundred summarized ones should read options and pick for itself. Turn Two: Send the Same Thing Handing back the previous reduction's state keeps the selection stable, so the prompt prefix doesn't churn when the corpus barely moved: second = context.select(docs, query, max_tokens=8_000, tuning=tuning, previous=first.state()) print(second.carried_over) # documents kept because the last turn had them Unchanged documents get carry_over_bonus added to their score; a document whose content changed is excluded, since carrying it over would move the prefix anyway. This pairs with stable rendering: selected documents render in path order by default, so two turns that select the same documents produce byte-identical text and provider prompt caches keep hitting. Pass render_order="rank" if you would rather have relevance ordering than cache stability. Ranking Is Lexical The built-in ranker is BM25-style (term frequency, saturated and length-normalized, weighted by inverse document frequency), plus two code-corpus signals: a query term in the path outweighs the same term in the body, and anchor files (README, pyproject.toml) get a small bonus. It has no embeddings, which keeps the default path free of a model dependency and an index to invalidate. Three ContextTuning settings close part of the gap without either: split_identifiers tokenizes resolveCredentials as the compound and its parts; query_expansion ranks once, harvests the terms that make the top documents distinctive, and ranks again; salience_weight blends in import-graph centrality, which is query-independent and so still orders a corpus when the query is weak or absent. For semantic retrieval, anyinfer.semantic_ranker() wraps your own embed()/rerank() or any embedding index into the same Ranker protocol select() expects; see the API reference. Tuning Every algorithmic choice is a field on ContextTuning. Pass it to select(), put it in the context block of the configuration file, or set it with a --context-* flag on anyinfer context; the three name the same things. Every setting that changes what gets sent is off by default, so a reduction never changes shape between releases; the single exception is exact-duplicate collapse, which is lossless. ContextTuning.recommended() is the set worth having for a source-code corpus. The full field table, with the reasoning behind density ordering and the diversity penalty, is in the API reference. Conversations Are Context Too select() reduces material you collected. compact_history() reduces material you produced, which in an agentic loop is where the window actually goes: compaction = context.compact_history(messages, max_tokens=budget.remaining_tokens or 8_000) result = await client.generate(list(compaction.messages), target=target) # history: 14 of 42 message(s); ~7600 of 8000 tokens; 12 dropped; 9 payload(s) elided Three passes over the middle of the conversation, cheapest loss first: tool-result payloads are elided, then text payloads, then plain messages are dropped, stopping the moment it fits. System messages and the recent window are never touched, tool-call pairing is never broken (a message carrying a ToolCall or ToolResult is emptied, never dropped), and every elision is visible as [elided N characters]. If the protected messages alone exceed the budget you get fits=False and the conversation back unchanged. To apply the same rules automatically on the request path, hand the client a policy: client = ai.Client(providers, history=ai.HistoryPolicy()) last_resort, the default mode, compacts only after every target (including Route.context_window_targets) is exhausted, preferring a larger-window model to losing history. proactive compacts before dispatch instead. The policy is off unless you configure it, never compacts against an unknown window, and every compaction emits a ContextReduced event, so a shortened conversation is never a silent one. Because the policy lives on the client, the Python API, the CLI, the tool loop, and the sidecar all inherit it identically. Every Reduction Announces Itself Reduction emulates a larger context window, and a truncated corpus produces answers that look just like a complete one's. So the result records everything: reduction.omitted_count # 328; not represented at all reduction.collapsed_exact # 12; sent once under another path, losslessly reduction.collapsed_near # 3; a similar file was sent; the differences were not reduction.compacted_count # 5; sent without their commentary reduction.partial_count # 4; only some spans of the file were sent reduction.complete # False reduction.summary() # content-free, safe to log or show a user complete means every offered document reached the model at full fidelity. Pass an observer= to receive a ContextReduced telemetry event; it carries counts and ceilings only, never paths or content, because a path name can itself be sensitive. Budgets follow the same rule as capabilities: when the target's window is unknown, budget.remaining_tokens is None and the library will not invent one; you choose the fallback in the open. Byte and document ceilings apply independently of tokens (max_bytes defaults to 4 MiB, max_documents to 200). The Envelope Format Reduced output is a mechanical data envelope (neutral tags, HTML-escaped attributes, no prose): …content… …span… …code… You place reduction.text in your own message; the library never touches GenerationRequest.messages. The format is stable enough to parse back out of stored transcripts, so changing it is a documented breaking change: format is bumped when an existing element's meaning changes, not when one is added. The render functions are in the API reference. Key Takeaways Your application collects and approves material; the library only ranks, selects, and renders it, with no file or network access of its own. Five strategies cover the fidelity ladder, and plan() prices all the deterministic ones exactly, for free, before you commit. Every reduction reports what was omitted, collapsed, compacted, or partial; complete is the one-flag summary. compact_history() and HistoryPolicy apply the same discipline to conversations, without ever breaking tool-call pairing or touching the system prompt. Path-ordered rendering keeps consecutive turns byte-identical, which is what keeps prompt caches hitting. See Also Fitting a corpus to a budget: the task-oriented walkthrough. Token estimation and context budgets: where max_tokens comes from. Context reduction API: every type, tuning field, and render function. --- # Concepts / The Local Subsystem Source: https://anyinfer.dev/concepts/local/ The Local Subsystem Running a model on the local machine should be one target string, with the same guarantees as a hosted call. This page covers the machinery behind that string: hardware detection, server tuning, supervision, and measurement. flowchart LR A[Hardware profile] --> B[Backend] B --> C[Download GGUF] C --> D[llama-server] D --> E[OpenAI-compat target] result = client.generate(prompt, target="llama-cpp:qwen2.5-7b-instruct-q4-k-m") Behind that call: resolve the artifact from the catalog, download and verify it, detect the hardware, tune a server for it, start it on loopback, speak the OpenAI dialect, answer. Six components, composed once so applications do not compose them. Hardware Detection Is Advisory from anyinfer import local profile = local.detect() profile.total_ram_bytes # 136_365_211_648 profile.primary_accelerator # Accelerator(kind='cuda', total_vram_bytes=..., ...) profile.warnings # everything that could not be determined, and why Detection proposes; callers decide. Every probe is best-effort: a missing tool, a permission error, or unparseable output produces a warning and a None field, never an exception. A wrong number here would mis-tune a server, so unknown is always preferred to guessed. Results are disk-cached, keyed by a signature of the probe executables themselves, so installing a GPU driver invalidates the cache without the user having to know. Override with ANYINFER_HARDWARE_CACHE_BYPASS or ANYINFER_HARDWARE_CACHE_REFRESH. Tuning Explains Itself plan = local.plan_server( profile, local.TuningInputs(artifact_size_bytes=4_680_000_000, parameter_size="7B"), posture="balanced", ) plan.context_size # 32768 for line in plan.rationale: print(line) # offloading all layers to the cuda device # 24.0 GiB of VRAM: budgeting 11.2 GiB for the KV cache after 4.4 GiB of weights # using 4 CPU threads Postures (conservative, balanced, aggressive) control how much of the machine to commit; aggressive additionally enables a q8_0 KV cache and two concurrent slots. Two subtleties the tuner accounts for, both real failure causes when missed. The KV cache scales with concurrency: llama.cpp spreads --ctx-size across --parallel slots, so the real footprint is context × parallel, and budgeting one slot then serving two runs out of VRAM. And weights are resident too: the KV budget is what remains after the model, not the whole device. Model files themselves are pinned, hash-verified, atomic, and resumable; that machinery belongs to the catalog; see what gets verified. Supervision The supervised llama-server runs as a child process with rules that cover the ways local multiplexing goes wrong: Swaps are serialized. Two requests for two unloaded models do not race: the first loads, the second waits. Racing means two full model loads competing for the same VRAM, and both lose. Requests block until ready. A caller never gets a 503 because a model happened to be loading; the wait is bounded by a health-check timeout. "Loading" and "failed" are distinguished. The child's output is captured, so a broken model reports why, instead of looking like a slow one for the full timeout. The idle timer keys on active streams, not last-request time, so a long generation with no new requests is not killed mid-flight. VRAM admission is checked before spawning. A model that provably will not fit is refused with a clear message rather than crashing the child with an OOM. Reaping is verified. A process is not gone because it was asked to stop; on Windows a launcher can exit while the server it spawned keeps the port and the GPU. Servers bind 127.0.0.1 only. A non-loopback bind requires allow_remote_exposure=True. What Is Already Usable Here Before running anything, there is a cheaper question: what can this machine use right now? from anyinfer import default_registry, local found = await local.discover(default_registry) # (DiscoveredProvider(provider_id='ollama', evidence='endpoint', detail='4 models', …), # DiscoveredProvider(provider_id='anthropic', evidence='environment', # credential_ref='env://ANTHROPIC_API_KEY', …)) Discovery contacts only loopback addresses that a provider descriptor declares as its defaults, reports an endpoint only when it answers with at least one model, and never reads a secret: environment evidence records the variable's name as env://NAME and the value stays where it was. The OS keyring is a third source, off by default because reading a vault can prompt the user to unlock it. anyinfer init is this composed with the config writer; see the CLI guide. Tier Recommendation recommendation = local.recommend_alias(profile, ai.load_default_catalog()) recommendation.alias # "large" recommendation.reason # "24 GiB of VRAM comfortably fits the large tier" recommendation.confident # False when memory could not be determined Requirements live in the catalog as data, so updating a recommendation is a catalog change rather than a code change. Unknowns never inflate the recommendation. Measuring What a Model Actually Does Here A tier recommendation predicts; it does not measure. On the same GPU the same weights can differ by an order of magnitude depending on what else is resident, so an application choosing a default (or explaining a slow session) needs a number from this machine, not from a table: measurement = client.benchmark("llama-cpp:qwen3-8b-q4-k-m") measurement.prefill_tokens_per_s # compute-bound: sets time to first token measurement.decode_tokens_per_s # bandwidth-bound: sets the rest of the wait measurement.summary # 'llama-cpp:qwen3-8b-q4-k-m: prefill 1840 tok/s, ttft 1120 ms, decode 38.4 tok/s' The two rates are separate because a machine can be fast at one and slow at the other. prefill_tokens_per_s is None unless the provider timed its own prefill phase; deriving it from time-to-first-token would fold queueing and network latency into a figure labeled compute. measurement.model_load_ms distinguishes cold from warm: a duration when this run paid a cold start, 0.0 when the model was already resident, None when the engine does not report loads. Without it, every first measurement looks like a bad one. Nothing is written unless asked. MeasurementStore persists results keyed by a fingerprint over provider, model, endpoint, machine, and runtime, so a measurement taken somewhere else never masquerades as a fresher version of this one. The CLI wraps the same call as anyinfer benchmark. What Is Not Here No bundled binaries or weights: llama-server runtimes and GGUF files are runtime-fetched by design, which keeps wheels small and the GPU build matrix out of the dependency tree. Key Takeaways Hardware detection is advisory: every probe is best-effort, and unknown is always preferred to a guessed number that could mis-tune a server. The tuner budgets KV cache per concurrent slot and after resident weights: the two arithmetic mistakes that make a "fits comfortably" plan fail. Supervision serializes model swaps, blocks requests until ready, checks VRAM before spawning, and binds loopback only. benchmark() measures prefill and decode separately and reports whether the run paid a cold start. See Also The model catalog: what fits this machine, and acquiring it. Run a model locally: the end-to-end walkthrough. llama.cpp provider · Ollama provider --- # Concepts / The Model Catalog Source: https://anyinfer.dev/concepts/catalog/ The Model Catalog The catalog answers three questions in sequence: what local models exist, whether this machine can run them, and how a pick becomes verified bytes an engine can load. It serves two shapes of caller over the same data: "Just give me a good default": the tier ladder (small, medium, large). "Let me see what I could run": the model catalog, forty-odd curated local models, each annotated with whether this machine can run it. They are bridged, not merged: a user's pick from the catalog flows into the ladder, so an application can offer both without maintaining two code paths. What Is in It models.json ships with AnyInfer and holds one row per logical model: from anyinfer import load_default_catalog catalog = load_default_catalog() entry = catalog.model("qwen2.5-7b-instruct") entry.display_name # 'Qwen2.5 7B Instruct' entry.parameter_size # '7B' entry.license # 'apache-2.0' entry.best_at # ('general-chat', 'multilingual', 'tool-use') entry.channels # ('llama-cpp', 'ollama') entry.est_file_bytes # 4683073632 Each entry carries a quantization ladder (the same model at Q8_0, Q6_K, Q5_K_M, and Q4_K_M), because "which model" and "at what quality" are different questions, and the second depends on the hardware: for variant in entry.variants_for("llama.cpp"): print(variant.quantization, variant.est_file_bytes) # Q8_0 8.10 GB # Q6_K 6.25 GB # Q5_K_M 5.44 GB # Q4_K_M 4.68 GB Most models are published through two channels, recorded per entry. A GGUF file set (a repository, an immutable commit, named files, a SHA-256 per file) is what AnyInfer downloads and verifies itself for the supervised llama.cpp path. An Ollama registry tag is what gets recommended when the Ollama daemon owns the store; its manifest digest is recorded so a moved tag is detectable. Every row also states its kind: "generation" (the default) or "embedding". Both are acquired identically, so they share one table. An embedding row carries dimensions and max_input_tokens, budgets memory from its own architecture rather than a chat model's, reports only the embedding operation so a chat request never routes to it, and is excluded from the small/medium/large ladder; those tiers answer "how big a chat model should I run", which an embedding model has no answer to. for entry in catalog.models_for(kind="embedding"): print(entry.id, entry.embedding.dimensions) best_at is a closed vocabulary (general-chat, coding, reasoning, vision, long-context, and nine others); a free-text tag set would drift into synonyms nobody can filter on. Will It Run? local_catalog() classifies every entry against this machine: view = client.local_catalog("llama-cpp", best_at="coding") for entry in view.runnable: print(entry.model.id, entry.fit.level, entry.fit.reasons[0]) Level Meaning gpu Fits comfortably in the accelerator's memory budget. cpu Too large for the GPU, but fits in system RAM. Runs, more slowly. tight Fits, with little room left for a longer context. no Exceeds this machine's memory entirely. unknown The numbers needed to judge are not available. Entries come back best-fit-first, deterministically ordered, and every verdict carries its reasons, so "why did it say tight?" is answerable from the returned object alone: entry.fit.reasons # ('needs 8.4 GiB of VRAM against a 9.0 GiB budget — it fits, but with little room # for a longer context', # 'planning for the vulkan runtime; installing the CUDA runtime would give better # throughput on this NVIDIA device') When the Model Runs Somewhere Else Ollama can point at another machine, and probing this machine would then describe the wrong computer. Since no Ollama API reports its host's specifications, the view says so instead of guessing, and allows the caller to supply the numbers: view = await client.local_catalog("ollama") view.hardware_source # 'unavailable' — ask the user for the remote host's specs specs = local.HardwareProfile.from_user_input(ram_gb=64, vram_gb=24, accelerator="cuda") view = await client.local_catalog("ollama", hardware=specs) view.hardware_source # 'provided' Anything omitted stays unknown rather than becoming zero. A remote engine may also sit behind a metered proxy, so its per-token cost reports as unknown rather than the genuine zero a loopback engine gets; see cost. Using a Pick as Your Default Tier The bridge between the two shapes is one call: catalog = load_default_catalog().with_alias_target("medium", "llama-cpp", "qwen2.5-14b-instruct") client = ai.Client(providers=[...], catalog=catalog) client.generate(prompt, target="medium") # now resolves to the user's pick Overlays produce new catalogs rather than mutating shared ones. Applications can overlay whole model entries the same way: the supported route for models the bundled catalog excludes, such as anything under non-commercial or research-only terms. Acquiring a Pick report = client.acquire_model("qwen2.5-7b-instruct", progress=on_progress) report.plan.quantization # 'Q5_K_M' — chosen for this machine, not assumed report.entry.handle # the file llama-server will be launched against The quantization is chosen, not assumed: the highest-quality rung whose weights and KV cache fit the memory budget, preferring a rung resident on the GPU over a better one that would page through the CPU. Below Q4, the policy prefers a smaller model at a good quantization over a bigger model at a bad one, so when nothing at Q4_K_M or better fits, acquisition refuses with the arithmetic rather than handing back a two-bit quantization. Pass local.VariantPrefs(allow_low_quality=True) to override. For vLLM the ladder has hard hardware gates (FP8 needs NVIDIA compute capability 8.9, Marlin GPTQ 8.0, AWQ 7.5); an unreported capability excludes a gated variant, since guessing produces a download that fails at model load. Know the Cost Before You Pay It report = await client.acquire_model("gpt-oss-120b", dry_run=True) report.plan.total_bytes # 63_387_346_208 report.plan.already_have_bytes # what a previous interrupted run already fetched report.plan.remaining_bytes # what this run would actually transfer Nothing is written; this is what an application needs to put a real confirmation dialog in front of a sixty-gigabyte download. Before any transfer starts, free disk space is checked against what remains plus ten percent. Progress reports aggregate figures for the whole acquisition: the total is known before the first byte (sizes are pinned in the catalog), bytes already on disk count toward the fraction, and rate and ETA stay None until there is a real sample. Callbacks are throttled and may arrive from a worker thread; a sink that raises is recorded once as a warning and then dropped, because a broken progress bar must not fail a download. Interrupting is not deleting: cancel and the partial transfers stay on disk, run again and it resumes. Nothing is registered until every file verifies, so a half-complete model is invisible rather than half-usable. What Gets Verified Where the file came from What is checked A pinned catalog artifact The SHA-256 in the catalog. A mismatch is a hard failure. A Hugging Face repository The digest the API reports for that file, checked against the bytes. A URL supplied with a digest The supplied digest. A mismatch is a hard failure. A URL supplied with no digest Refused, unless allow_unverified=True is passed. API digests are trust-on-first-use: the API is trusted for what the bytes should be, then the bytes are verified against it, which turns a later upstream change into a detectable event. An unverified file is recorded as unverified all the way out to locate_model(): "AnyInfer checked these bytes" and "AnyInfer found these bytes" are different claims. Two rules apply to every acquisition and are not optional. File names from a remote API are treated as attacker-influenced input: absolute paths, .. segments, drive letters, NUL bytes, and reserved Windows device names are rejected, and every destination is checked for containment after resolution so a symlink cannot escape. And pickle-format weights (*.bin, *.pt, *.pth, *.ckpt) are never fetched by default, because they execute arbitrary code on load; acquisition fails with a hint naming the risk. Finding It Again located = client.locate_model("qwen2.5-7b-instruct") located.path # a file for GGUF, a directory for a snapshot located.verified # whether every file was checked against a digest located.launch_hints # {'engine': 'llama.cpp', 'model': '…', 'ctx_size': 32768, …} Lookups do no network I/O and no re-hashing (size and modification time are compared against the index; pass verify=True to force the full check). launch_hints is advisory data, not process control: the llama.cpp supervisor consumes the hints automatically, while for vLLM they are what a developer would paste into a command line; AnyInfer acquires and locates vLLM weights but does not start a vLLM process. Models live under revision-scoped directories in the platform's data directory (ANYINFER_MODEL_DIR or Client(model_dir=...) overrides it), indexed by a store.json that is a cache, not the truth: a missing or corrupt index is tolerated and rebuild_index() recovers by rescanning. Nothing is ever evicted automatically: disk_usage() and remove_model() give an application what it needs to build its own policy. Engines That Keep Their Own Store Some engines already have a store, a registry, and a downloader. For those the useful operation is not "download these bytes" but "make yourself ready": report = client.pull_model("ollama", "qwen3:8b") report.already_present # True when nothing had to move report.bytes_transferred Those bytes land in the engine's store under the engine's own name; locate_model() will not find them, because they are not AnyInfer's to find. Which providers work this way is declared on the descriptor. anyinfer models pull ollama qwen3:8b is the same call from a shell. Where the Numbers Come From Every hash, size, revision, and verification date in the catalog is read from the upstream API by a pin script and written verbatim; an entry that cannot be verified is not shipped. A weekly job re-checks every pinned file and every Ollama tag digest against upstream and opens a human-reviewed pull request when something moved. A catalog entry therefore either points at bytes that hash to what was recorded, or it fails loudly. Key Takeaways One catalog serves both "give me a default tier" and "show me what fits", and a user's pick bridges into the alias ladder with with_alias_target(). Fit verdicts are five-state (gpu, cpu, tight, no, unknown) and always carry their reasons. Acquisition chooses the quantization for the machine, checks disk space first, resumes after interruption, and registers nothing until every file verifies. Verification is per-file against pinned or API-reported digests; unverified files are refused by default and marked unverified forever if allowed. See Also The local subsystem: hardware detection, tuning, and supervision. Run a model locally: the task walkthrough. Capabilities and provenance: how catalog data feeds capability assembly. --- # Concepts / Credentials and Redaction Source: https://anyinfer.dev/concepts/credentials/ Credentials and Redaction Two guarantees: A credential can be referenced rather than embedded, so config files stay safe to commit and share. A resolved secret can never appear in a log line, error message, telemetry event, or traceback. flowchart LR A["'env://KEY'"] --> C[Resolver chain] B["'credential://system/id'"] --> C C --> D[Resolved secret] D --> E[Registered for redaction] References Three forms ship in v1: ai.ProviderSettings.of("openai", api_key="sk-literal-value") # literal ai.ProviderSettings.of("openai", api_key="env://OPENAI_API_KEY") # environment ai.ProviderSettings.of("openai", api_key="credential://system/openai") # OS keyring The keyring form needs the [keyring] extra. A missing extra is a ConfigError with an install hint, not an ImportError: ConfigError: the 'credential://' scheme requires the keyring extra (hint: pip install 'anyinfer[keyring]') Every failure is actionable in the same way: CredentialError: environment variable OPENAI_API_KEY is not set (hint: export OPENAI_API_KEY= and retry) A typo'd scheme is not treated as a literal secret: LiteralResolver declines anything that looks like a known scheme, so env:/OPENAI_KEY fails loudly. As a rule of thumb: env:// is the usual choice for containers and CI, and credential:// for a workstation. A literal key is fine in test code and poor in config. Using the OS Keyring Install the extra, store the secret, and reference it: pip install "anyinfer[keyring]" keyring set AnyInfer openai-api-key client = ai.Client( [ ai.ProviderSettings.of("openai", api_key="credential://system/openai-api-key"), ] ) The service name is AnyInfer; the identifier is the developer's to choose, and the reference string is safe to commit: it names where the secret lives, not the secret. Keyring failures are actionable like the rest: CredentialError: no credential stored under 'openai-api-key' (hint: store it with keyring under service 'AnyInfer') CredentialError: no usable OS credential store is available on this system (hint: configure a system keyring, or use 'env://VAR_NAME' instead) Headless Linux often has no usable vault; env:// is the right answer there, and the error says so rather than leaving the developer to guess. Redaction Is Automatic and Global Every secret resolved through the credential chain is registered for redaction the moment it is resolved. From then on, it is stripped from: every error's detail, hint, and raw_text; every telemetry event field; recorded test cassettes, before they touch disk. raise AuthError(f"invalid key {secret}") # detail: "invalid key [redacted]" The registry is process-global by design. Secrets are process-global facts, and redaction has to apply even to errors raised by code that never saw the client that resolved them. Values shorter than six characters are not registered: redacting them would corrupt unrelated text far more often than it would protect anything. An application can register additional secrets it resolves itself: ai.register_secret(my_token) Custom Resolvers For anything beyond the keyring (AWS Secrets Manager, HashiCorp Vault), applications plug in their own vault: class VaultResolver: def handles(self, reference: str) -> bool: return reference.startswith("vault://") def resolve(self, reference: str) -> str: return my_vault.read(reference[len("vault://") :]) chain = ai.default_resolver() chain.add(VaultResolver()) # takes precedence over the built-ins client = ai.Client(providers, resolver=chain) The chain registers resolved secrets for redaction, not the individual resolvers, so a third-party resolver cannot forget to. Backend Credentials Never Transit the Sidecar When the sidecar is running, it authenticates clients to itself with its own bearer token. The provider credentials it uses stay on the server. A client pointed at the frontend never sees, sends, or needs them. Key Takeaways Credentials are referenced (env://, credential://), never embedded, so config stays safe to commit. Redaction is automatic, global, and applies from the moment a secret resolves, including to code that never saw the client that resolved it. Backend credentials never transit the sidecar; it authenticates clients to itself with its own bearer token. See Also Telemetry: payload privacy, the other half of the security posture. Shared configuration: where credential references live in the config file. --- # Concepts / Telemetry Source: https://anyinfer.dev/concepts/telemetry/ Telemetry and Observers The telemetry contract is typed in-process events delivered to registered observers. OpenTelemetry is an optional bridge over that contract, not the contract itself. Nothing is written anywhere by default. A deployment that wants zero telemetry pays nothing, including no dependency. flowchart LR A[Request lifecycle] --> B[TelemetryEvent] B --> C[Registered observers] C --> D[Your metrics / logs] B -.optional.-> E[OpenTelemetry bridge] Subscribing class Recorder: def on_event(self, event: ai.TelemetryEvent) -> None: match event: case ai.FirstToken(at_ms=ms): metrics.ttft.observe(ms) case ai.RequestCompleted(usage=usage, timing=timing): metrics.tokens.inc(usage.output_tokens or 0) case ai.RequestFailed(error=error): log.warning("request failed: %s", error.detail) client = ai.Client(providers, observers=[Recorder()]) # or later: client.subscribe(Recorder()) Observers are synchronous and dispatched inline, so keep on_event fast; queue anything slow. An observer that raises is isolated: the exception is swallowed and warned about once, because a broken telemetry sink must never fail a generation. The Events Request lifecycle: RequestStarted · TargetResolved · AttemptStarted · FirstToken · AttemptCompleted · RetryScheduled · FallbackTriggered · RepairAttempted · RequestCompleted · RequestFailed Degradation: the ones that make silent failures visible: Event Emitted when ParameterDropped A provider accepts a parameter and discards it, or honors it only in part. UsageEstimated A usage figure was derived rather than reported. ProviderDiagnostic A provider reported something about its own runtime. Anything AnyInfer drops or estimates is observable; a degraded request always leaves evidence. ProviderDiagnostic covers the case the other two cannot: the request worked, nothing was dropped or estimated, and it still took thirty seconds because the model had spilled out of VRAM since yesterday. See runtime diagnostics. Context reduction: ContextReduced (counts and ceilings only, never content). Local subsystem: ServerLifecycle · DownloadProgress Payloads Are Off by Default Prompt and response text are None unless an observer explicitly opts in: client.subscribe(audit_log, payloads=True) # sees prompt and response text client.subscribe(metrics) # never does Stripping happens per observer, so one payload-consuming sink does not leak text to the others. Everything still passes redaction first. Correlating Events Every request-lifecycle event carries a request_id. A fallback chain that tried three targets emits one RequestStarted and one terminal event with the same id, so a trace reads as a single request rather than three disconnected ones. The OpenTelemetry Bridge from anyinfer import otel otel.install(client) # spans and metrics, payload-free Requires the [otel] extra; nothing OTel-related is imported otherwise. What the bridge emits (one span per request, attempts as span events, GenAI semantic-convention metrics) is covered in the observability guide. The bridge is one consumer of the event contract. Applications that want structured data in-process (a JSONL trail, a SQLite evidence table) consume the events directly rather than round-tripping through spans. Key Takeaways Telemetry is typed in-process events to registered observers; OpenTelemetry is an optional bridge, not the contract itself. Prompt and response payloads are off by default, per observer, so one consumer's opt-in never leaks text to another. Degradation is observable: ParameterDropped and UsageEstimated mean a dropped parameter or a derived usage figure always leaves evidence. See Also The event stream: the other event channel, for response content. Capabilities: why ParameterDropped exists. How-to: observability --- # Concepts / Run Manifests Source: https://anyinfer.dev/concepts/run-manifests/ Run Manifests A run manifest is the portable explanation of one generation. It records the route that won, attempts and fallback, structured-output and cache mechanisms, context reductions, capability provenance, usage, cost, and timing. It is derived from the same typed events and final result the client already produces; it is not another measurement or log stream. Every buffered Generation carries generation.manifest by default. A stream exposes its current manifest through stream.manifest, including after cancellation. Set manifests=False on the client to avoid allocating manifests, or pass manifest=False on one call. result = client.generate("Summarize this", target="medium") print(result.manifest.to_json()) The default record contains shapes and decisions, never prompts, completions, schemas, tool arguments, or document bodies. manifest_payloads=True is an explicit client-wide opt-in; captured strings still pass through credential redaction. AnyInfer never writes manifests; callers decide whether and where to serialize them. Why Manifests Make Useful Golden Files Model prose changes. Routing and policy decisions should not change accidentally. The testing helper removes request IDs and wall-clock fields, then compares the stable remainder with a checked-in JSON file. See the offline testing guide and the complete example. For the related question "did my request's resolution change across targets", see the portability diff. The Format The canonical, machine-readable JSON Schema ships in the package: import jsonschema import anyinfer as ai jsonschema.validate(generation.manifest.to_dict(), ai.manifest_json_schema()) The schema requires the format, request identity, request summary, route, usage, and timing facets; other facets describe capabilities, attempts, structured output, prompt caching, context reduction, dropped parameters, notes, and explicitly opted-in payloads. Cost values are decimal strings or null, and capability facts retain their provenance. This is a pre-1.0 contract. The top-level format is currently "1", and readers must ignore unknown keys: adding a field does not change the format, while changing an existing field's meaning does. Key Takeaways A manifest explains one generation's decisions (route, mechanisms, reductions, provenance) and is payload-free unless payloads are opted in. The library never writes manifests to disk; serialization is the caller's call. Manifests make good golden files because the testing helper strips the volatile fields, leaving only decisions that should not change accidentally. See Also Test your application offline: manifests as golden files. Regression-test fallback and repair: the runnable example. Telemetry: the live event channel manifests are derived from. --- # Concepts / Arena Runs Source: https://anyinfer.dev/concepts/arena/ Arena Runs An arena sends the same request to a fixed set of targets and selects one answer, keeping every candidate as evidence. A three-target arena costs up to three ordinary generations before selection, and the judge or synthesize strategies add one more. It is an on-demand comparison tool, not a router that learns from winners; results are not stored, ranked across runs, or fed back into future target selection. In order to compare targets without spending anything, use the portability diff instead. The strongest mode is structured consensus: candidates must satisfy the same schema, their canonical JSON values are grouped without regard to object key order, and the largest exact group wins. import anyinfer as ai policy = ai.ArenaPolicy( targets=("openai:gpt-5-mini", "anthropic:claude-haiku-4-5", "ollama:qwen3:8b"), strategy="consensus", min_candidates=2, ) result = client.generate( "Classify this ticket.", schema={ "type": "object", "properties": {"label": {"type": "string"}}, "required": ["label"], "additionalProperties": False, }, arena=policy, ) print(result.structured) print(result.arena.agreement) for candidate in result.arena.candidates: print(candidate.generation, candidate.error) The other strategies: without a schema, consensus announces a degradation to first_valid, since free-form text has no exact equality rule. cheapest never treats an unknown cost as zero, and fastest uses measured completion timing. judge asks one named target to choose a candidate through a forced schema; synthesize asks it to produce an additional answer while retaining all original candidates, marked separately. Candidate envelopes are anonymized by default; set reveal_targets=True only when the selector genuinely needs provider identity. The Same Policy on Every Surface The CLI (anyinfer run --arena ... --arena-strategy consensus) and the OpenAI-compatible sidecar (an anyinfer_arena request extension) reach the same client-layer policy as the Python call above. The sidecar's response remains a valid single-choice OpenAI completion, with content-free candidate evidence added under anyinfer_arena; streaming buffers candidates and emits only the selected answer, so branches never interleave on the wire. Tool Loops and Spend Ceilings run_tools(..., arena=policy, max_rounds=R) runs one isolated conversation per candidate, with a provider-call ceiling of one round-trip per candidate per round plus the optional judge or synthesis call. No candidate sees another candidate's tool results. Arena spend is estimated and reserved for the whole run before any branch dispatches, so a spend-ceiling refusal produces zero provider calls. If a failed candidate's paid usage cannot be recovered faithfully, the aggregate is marked incomplete rather than presenting an understated total. Use anyinfer run --dry-run --arena ... to inspect the call ceiling and summed cost range without sending anything. Key Takeaways An arena multiplies cost by its target count; use it to answer a question, not as standing routing. The free alternative for capability comparisons is compare(). Structured consensus is the mode with a real equality rule; text-only requests degrade to first_valid and say so. Every candidate is returned, so the selected answer never erases the evidence behind it. Spend is reserved up front: a ceiling refusal costs zero provider calls. See Also Compare targets without spending: the zero-cost alternative for capability questions. Structured output: the schema contract consensus depends on. Cost and spending: ceilings and unknown-cost handling. --- # Examples / Examples Source: https://anyinfer.dev/examples/ Examples Small, complete programs, not fragments. Each one is a pattern the library was designed around. The shape of every example is exercised in CI against the in-process fake providers (tests/test_docs_examples.py); whether a program runs offline as written, and against what, is stated on each page. Example What it shows Structured summaries with a fallback chain Schema-validated output, bounded repair, multi-provider fallback, and the attempt trail A local tool-calling assistant The @ai.tool decorator, the tool loop, and running fully local Distill a corpus Map/reduce over material that will never fit, with cost preflight and a deterministic reducer Regression-test fallback and repair A golden run manifest that asserts inference behavior instead of model prose Semantic search over a small corpus Embedding a corpus, index/query space safety, and reranking, with your own in-memory similarity math Comparing targets without spending anything is a guide rather than an example; see Will my request survive a target change? If you are new to the library, read the Quickstart first; these examples assume you know what a target is. --- # Examples / Structured Summaries with Fallback Source: https://anyinfer.dev/examples/summarize-with-fallback/ Structured Summaries with a Fallback Chain A command-line tool that turns arbitrary text into a schema-validated summary, staying up when a provider is not: it tries Anthropic first, falls back to OpenAI, and finally to a local Ollama model. As written it needs ANTHROPIC_API_KEY, OPENAI_API_KEY, and a running Ollama; the shape itself is exercised in CI against in-process fakes. """summarize.py — `python summarize.py < release-notes.txt`""" import json import sys import anyinfer as ai SUMMARY_SCHEMA = { "type": "object", "properties": { "headline": {"type": "string"}, "topics": {"type": "array", "items": {"type": "string"}}, }, "required": ["headline", "topics"], } client = ai.Client( [ ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ai.ProviderSettings.of("openai", api_key="env://OPENAI_API_KEY"), ai.ProviderSettings.of("ollama"), ] ) result = client.generate( "Summarize this text:\n" + sys.stdin.read(), route=ai.Route( targets=( "anthropic:claude-sonnet-4-5", "openai:gpt-5.2", "ollama:qwen3:8b", ), retry=ai.Retry(max_attempts=2), ), schema=SUMMARY_SCHEMA, repair=ai.Repair(max_attempts=1), ) print(json.dumps(result.structured, indent=2)) print(f"via {result.target} using {result.structured_mechanism}", file=sys.stderr) for attempt in result.attempts: print(f" {attempt.target}: {attempt.outcome}", file=sys.stderr) What to Notice result.structured is always valid against SUMMARY_SCHEMA: validation happens client-side regardless of which provider answered, and result.structured_mechanism tells you how it was enforced (grammar, json_schema, json_mode, or prompt). See structured output. The fallback chain is data, not code. Route.targets is an ordered tuple; retry policy applies per target. No try/except pyramid, and the attempt trail (result.attempts) records every hop for your logs. Credentials never appear in source. env://ANTHROPIC_API_KEY is a credential reference; the resolved secret is registered for redaction, so it cannot leak through errors or telemetry. The local fallback needs no key at all. If both hosted providers are down, the same call lands on Ollama, and if everything fails, you get one AllTargetsFailedError carrying the per-target causes, not the last exception to happen to escape. See Also Add a fallback chain: retry policy and route design. Enforce a JSON schema: mechanisms and repair. --- # Examples / A Local Tool-Calling Assistant Source: https://anyinfer.dev/examples/local-tool-agent/ A Local Tool-Calling Assistant An assistant that answers questions about your project by calling Python functions you hand it. It does not run offline as written: it needs a running Ollama with qwen3:8b pulled. Since that is the only requirement, nothing leaves your machine and no API key is involved; the same program points at any hosted provider by changing the target string. """assistant.py — `python assistant.py "what does pyproject.toml declare?"`""" import sys from pathlib import Path import anyinfer as ai @ai.tool def read_file(path: str) -> str: """Read a file from the current project directory.""" return Path(path).read_text(encoding="utf-8") @ai.tool def list_files(pattern: str = "*") -> str: """List project files matching a glob pattern.""" return "\n".join(str(p) for p in Path.cwd().glob(pattern)) client = ai.Client([ai.ProviderSettings.of("ollama")]) result = client.run_tools( sys.argv[1], tools=[read_file, list_files], target="ollama:qwen3:8b", ) print(result.text) The model decides when to call read_file or list_files; AnyInfer runs the function, feeds the result back, and loops until the model produces a final answer (bounded: a runaway loop raises ToolLoopError rather than spinning). To stream the answer token by token instead of waiting for it, see streaming; nothing else about the program changes. What to Notice @ai.tool derives the wire schema from the signature: name, docstring, and type hints become the provider-facing tool spec, and read_file.spec shows exactly what the model is told. The loop lives in the core, not your code. run_tools handles the call → execute → feed-back cycle identically on every provider that supports tools; the conformance matrix says which do. If qwen3:8b is not pulled yet, or you would rather have AnyInfer supervise a llama-server for you, the local inference guide covers the end-to-end path, including picking a model tier that fits your hardware. See Also Run the tool loop: rounds, bounds, and error handling. Stream to a terminal: the same conversation, token by token. Run a local model end to end: pulling models and supervised llama-server. --- # Examples / Distill a Corpus Source: https://anyinfer.dev/examples/distill-a-corpus/ Distill a Corpus distill reads material that will never fit at any fidelity and writes something shorter: each chunk is summarized against your question, then the notes are synthesized into one answer. Since that spends inference (a corpus of N chunks costs N+1 requests), it is a separate function rather than a select() strategy, and this example needs a live provider: as written, Anthropic with ANTHROPIC_API_KEY set. The Basic Shape import anyinfer as ai from anyinfer import context client = ai.AsyncClient( [ ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) result = await context.distill( changelog_text, "What changed for end users in this release?", client=client, target="anthropic:claude-sonnet-4-5", ) print(result.text) print(f"{result.calls} calls, {result.usage.output_tokens} output tokens") result.calls and result.usage report the fan-out and aggregate spend, so the multiplier is never a surprise. source may be raw text or ContextDocument values; documents split per document, because a document boundary is a natural chunk boundary. Know the Cost Before You Commit A distillation over 200 chunks is 201 requests. Size it first: documents = [context.ContextDocument.of(p, t) for p, t in collected] chunks = sum(len(context.split_document(d)) for d in documents) estimate = client.budget( [ai.user("...one representative chunk...")], target="anthropic:claude-sonnet-4-5", ).estimated_cost print(f"about {chunks + 1} calls") if estimate is not None: print(f"roughly ${estimate.low * chunks:.2f}–${estimate.high * chunks:.2f}") Afterwards, result.usage.cost_usd is what it cost, wherever the provider reports cost. Hierarchical Reduce, When There Are Many Notes With enough chunks, the notes themselves exceed the window. distill handles that by reducing in batches (sized by what fits, not by note count) and then reducing those summaries: result = await context.distill(huge_corpus, question, client=client, target=target) print(result.reduce_depth) # 1 for a single pass, higher when it recursed A single-pass merge would overflow here. Variations If your notes merge structurally (a union of entries, a concatenation, a JSON merge), supply a reducer and the reduce call disappears; the map phase is still N calls, the reduce is free and reproducible: import json def merge_findings(notes): findings = [] for note in notes: try: findings.extend(json.loads(note)["findings"]) except (ValueError, KeyError): continue return json.dumps({"findings": findings}, indent=2) result = await context.distill( documents, "List every configuration key this code reads.", client=client, target="anthropic:claude-sonnet-4-5", map_instructions=( 'Return JSON: {"findings": [{"key": "...", "file": "..."}]}. ' "Use only what this part contains." ), reducer=merge_findings, ) assert result.calls == result.chunk_count # map phase only The built-in prompts are mechanical scaffolding ("here is part 3 of 9, take notes"), not application prose. Replace them when the framing matters: result = await context.distill( transcripts, "What did customers complain about?", client=client, target="anthropic:claude-sonnet-4-5", map_instructions=( "Read this support transcript excerpt. List each distinct complaint with the " "product area it concerns. Quote the customer's own words where possible." ), reduce_instructions=( "Group these complaints by product area, most frequent first. Preserve the " "customer quotes." ), ) Concurrency defaults to 4, and a fan-out is somebody's rate limit. Failures propagate as normal provider errors, so retry and fallback stay on your Route, not duplicated inside the reducer: result = await context.distill( documents, question, client=client, target=target, concurrency=2, ) From a synchronous application, use distill_sync; chunks process one at a time, since concurrency is the async path's feature: with ai.Client([ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY")]) as client: result = context.distill_sync( corpus, question, client=client, target="anthropic:claude-sonnet-4-5", ) See Also Context reduction: when to distill instead of select. Fitting a corpus to a budget: the non-inference strategies. --- # Examples / Regression-Test Fallback and Repair Source: https://anyinfer.dev/examples/golden-manifest/ Regression-Test Fallback and Repair This test asserts the part of inference that should be deterministic (which route ran and which mechanism enforced the schema), and it runs offline: a scripted provider and stored fixtures, no network, no spend. The pytest plugin stores the normalized run manifest in manifests/fallback-and-repair.json beside the test. import anyinfer as ai from anyinfer.testing import ScriptedFailure, ScriptedModel def test_fallback_contract(anyinfer_client, anyinfer_scripted, anyinfer_golden_manifest): provider = anyinfer_scripted( [ ScriptedModel( "primary", failures=(ScriptedFailure(status=503, retry_after_s=0.0),), ), ScriptedModel("fallback", structured={"answer": "stable"}), ] ) client = anyinfer_client(provider) result = client.generate( "answer", route=ai.Route( targets=(provider.target("primary"), provider.target("fallback")), retry=ai.Retry(max_attempts=1, backoff_base_s=0.0), ), schema={ "type": "object", "properties": {"answer": {"type": "string"}}, "required": ["answer"], }, ) anyinfer_golden_manifest(result.manifest, "fallback-and-repair") Create or refresh goldens with pytest --update-manifests, and review the JSON diff like a behavior change: a different target, extra attempt, weaker schema mechanism, or new reduction should be intentional. What to Notice The golden file is a normalized run manifest, so the assertion covers route, attempts, and schema mechanism rather than model prose. The scripted 503 on primary puts a real fallback hop in the manifest; the anyinfer_scripted and anyinfer_golden_manifest fixtures come from the pytest plugin described in testing your application offline. A golden manifest asserts "did this run's behavior change"; compare_diff asserts "did this request's resolution change"; see the portability diff. See Also Run manifests: what the golden file records. Test your application offline: the pytest plugin and fixtures. Will my request survive a target change?: the request-level diff beside this run-level golden. --- # Examples / Semantic Search over a Small Corpus Source: https://anyinfer.dev/examples/semantic-search/ Semantic Search over a Small Corpus For a corpus small enough to hold in memory, semantic search needs no vector store: this example embeds a handful of documents, embeds a query with the matching intent, and ranks by cosine similarity itself, with persistence left to the optional anyinfer-store add-on. As written it runs offline against anyinfer.testing.FakeEmbeddingRerankProvider (a deterministic pseudo-embedding that proves the wiring, not retrieval quality), so point EMBED_TARGET/RERANK_TARGET at a real provider (ollama:nomic-embed-text, openai:text-embedding-3-small, a local TEI server) when you want ranking by meaning. import math import anyinfer as ai from anyinfer.testing import FakeEmbeddingRerankProvider provider = FakeEmbeddingRerankProvider( "offline", embedding_dimensions={"embed-small": 8}, rerank_models=["rerank-small"] ) registry = ai.ProviderRegistry(load_builtins=False, load_entry_points=False) provider.register(registry) EMBED_TARGET = "offline:embed-small" RERANK_TARGET = "offline:rerank-small" CORPUS = [ "The moon landing happened in 1969", "Sourdough bread needs a live starter", "Apollo 11 was the spacecraft that carried astronauts to the moon", ] QUERY = "Apollo spacecraft that reached the moon" def cosine_similarity(a: tuple[float, ...], b: tuple[float, ...]) -> float: dot = sum(x * y for x, y in zip(a, b, strict=True)) norm_a = math.sqrt(sum(x * x for x in a)) norm_b = math.sqrt(sum(y * y for y in b)) return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0 with ai.Client( [ai.ProviderSettings.of(provider.provider_id)], registry=registry, use_default_catalog=False, ) as client: # Query and document embeddings must be built with matching intent — see below — # and, for cross-provider safety, from the same target. corpus_embedded = client.embed(CORPUS, target=EMBED_TARGET, input_type="document") query_embedded = client.embed(QUERY, target=EMBED_TARGET, input_type="query") query_vector = query_embedded.vectors[0].values ranked = sorted( zip(CORPUS, corpus_embedded.vectors, strict=True), key=lambda pair: cosine_similarity(query_vector, pair[1].values), reverse=True, ) best_match, _ = ranked[0] # the fake's hash-based vectors make this arbitrary; see above # A reranker scores the same corpus against a query in one call, no manual # similarity math needed — worth reaching for once a corpus outgrows "just loop over # the vectors yourself." reranked = client.rerank(QUERY, CORPUS, target=RERANK_TARGET) assert CORPUS[reranked.items[0].index] == ( "Apollo 11 was the spacecraft that carried astronauts to the moon" ) Intents (input_type), the embedding-space safety rule that result.space implements, and batching are core concepts, covered in Embeddings and reranking; what follows is specific to building a small in-memory index. Index/Query Compatibility, Applied result.space is exactly what you store alongside the vectors so a later query can check it matches before comparing anything: if query_embedded.space.compatible_with(stored_space): ... # safe to compare Fallback and Local Embeddings operation_routes={"embedding": ai.Route(targets=[...])} on the client (or --config's operation_routes key) sets the default route embed() uses when no target=/route= is passed; it is the same mechanism default_route gives generation, kept separate so an embedding fallback chain is never accidentally reused for chat traffic. Local engines are first-class fallback members: TEI, Ollama, and LM Studio all embed, so a chain like [local-tei:bge-large, openai:text-embedding-3-small] tries the free local model first and only spends money if it is unreachable. See Also Embeddings and reranking: intents, spaces, and batching. Embed, store, and query — without a database: the same loop persisted in a single SQLite file. Error catalog: the refusal a mismatched embedding fallback raises. --- # Providers / Providers Source: https://anyinfer.dev/providers/ Providers This section is the compatibility inventory: dedicated adapters for protocols that require real translation, plus declarative presets for OpenAI-compatible services and engines. Breadth is useful, but it is not AnyInfer's product boundary; start with why and when to use AnyInfer when choosing an integration layer. The generated complete inventory is the full accessible rendering: all 106 providers (20 dedicated adapters, 86 presets), each with its target prefixes, key variable or default endpoint, and notes. OpenAI openai: Responses API, reasoning-token accounting. Hosted Anthropic anthropic: / claude: Messages API, extended thinking deltas. Hosted Google Gemini gemini: / google: Native generateContent, thinking levels, discovered windows. Hosted DeepSeek deepseek: Separate reasoning channel, split cache accounting. Hosted xAI (Grok) xai: / grok: Provider-reported cost, discovered pricing. Hosted Google Vertex AI vertex: Gemini with GCP auth; project-scoped addressing. Enterprise AWS Bedrock bedrock: Converse API, SigV4 or API key, binary streaming. Enterprise Cohere cohere: Native v2 chat, grounded generation, thinking channel. Hosted Voyage AI voyage: Specialist embeddings and reranking; query/document intents. Hosted Jina AI jina: Specialist embeddings and reranking; full task vocabulary. Hosted Text Embeddings Inference tei: Local embeddings and reranking; retrieval-only, one model per server. Local LM Studio lm-studio: Native discovery: context, quantization, residency. Local Ollama ollama: Native API, grammar schemas, phase timings. Local llama.cpp llama-cpp: Supervised llama-server, loopback only. Local OpenAI-compatible openai-compat: Any /chat/completions endpoint by URL. Local Hosted Hosted & local presets groq: together: mistral: vllm: … Eighty-six OpenAI-compatible services and engines, preconfigured. Local Hosted OpenRouter openrouter: Rich discovered pricing and context data. Hosted Nebius Token Factory nebius: Live pricing, context, quantization, and reasoning channels. Hosted Azure AI Foundry azure-foundry: / azure: max_completion_tokens, API key or Entra auth. Enterprise GitHub Copilot copilot: auto sentinel, CLI-delegated auth. Hosted Microsoft 365 Copilot m365-copilot: / m365: Interactive auth only, the most constrained provider. Enterprise See the conformance matrix for exactly which behaviors each one supports, generated from test results rather than asserted. What Is the Same Everywhere Since the core owns orchestration, routing and retries, structured-output validation, cost accounting, timing, telemetry, and the event stream behave identically no matter which provider served a request. The concepts section documents each. What Differs, and How You Find Out Real differences are surfaced, never hidden: Capability flags say what a model supports, with provenance. structured_mechanism on each result says how a schema was actually enforced. ParameterDropped events fire when a provider accepts a parameter and discards it. Provider pages document the rest. Reaching Provider-Specific Parameters Anything a provider supports that AnyInfer does not model is reachable verbatim: client.generate( prompt, target="ollama:qwen3:8b", provider_options={"ollama": {"keep_alive": "10m", "num_gpu": 99}}, ) Options are namespaced by provider id and passed straight through to the matching adapter; the core never inspects them. A developer should never have to fork the library to reach a provider-specific feature. Adding Your Own Third-party adapters register through the anyinfer.providers entry-point group and prove themselves with the same conformance suite the built-ins run. See writing an adapter. --- # Providers / Every Provider Source: https://anyinfer.dev/providers/all/ Every Provider This is AnyInfer's compatibility inventory, not its primary value proposition: 106 providers comprising 20 dedicated adapters with provider-specific behavior and 86 presets over the shared OpenAI-compatible adapter. Each is a first-class target prefix: groq:, vllm:, bedrock:. This page is generated It is rendered from the provider registry by scripts/generate_provider_index.py and verified by a test, so it cannot drift from what the library actually ships. Counts and columns come from the code. import anyinfer as ai # Any row below works the same way — pick the id from the "Target" column. client = ai.Client([ai.ProviderSettings.of("groq", api_key="env://GROQ_API_KEY")]) result = client.generate(prompt, target="groq:llama-3.3-70b-versatile") New to these? Start with choosing a provider; the per-provider quirks for the preset table are in hosted & local presets. Dedicated Adapters These need more than declarative endpoint and auth settings: a native request shape, special auth flow, richer discovery, or provider-specific stream handling. Each has its own adapter and guide. Provider Target Kind What it adds Anthropic anthropic: / claude: Hosted Messages API, extended thinking deltas; any Anthropic-shaped endpoint Azure AI Foundry azure-foundry: / azure: / foundry: Hosted Deployment-addressed, api-version pinning AWS Bedrock bedrock: / aws-bedrock: / amazon-bedrock: Hosted Converse API, SigV4 or API key, binary event-stream framing Cohere cohere: Hosted Native v2 chat, grounded generation, thinking channel GitHub Copilot copilot: / github-copilot: Hosted GitHub Copilot subscription; auth delegated to the Copilot CLI DeepSeek deepseek: Hosted Separate reasoning channel, split cache accounting Google Gemini gemini: / google: / google-gemini: / ai-studio: Hosted Native generateContent, thinking levels, discovered windows Jina AI jina: / jinaai: Hosted Specialist embeddings and reranking; full task vocabulary llama.cpp (supervised llama-server) llama-cpp: / llamacpp: / llama: Local Supervised llama-server, loopback only LM Studio lm-studio: / lmstudio: Local Native discovery: context, quantization, residency Microsoft 365 Copilot m365-copilot: / m365: Hosted Microsoft 365 Copilot Chat, Entra auth Nebius Token Factory nebius: / nebius-token-factory: / token-factory: Hosted Verbose listing: discovered pricing, context and quantization Ollama ollama: Local Native API, grammar schemas, phase timings OpenAI openai: Hosted Responses API, reasoning-token accounting OpenAI-compatible endpoint openai-compat: / openai-compatible: / oai-compat: Hosted Any /chat/completions endpoint by URL OpenRouter openrouter: Hosted Router across upstreams, discovered per-model pricing Text Embeddings Inference tei: / text-embeddings-inference: Local Local embeddings and reranking; retrieval-only, one model per server Google Vertex AI vertex: / vertex-ai: / google-vertex: Hosted Gemini with GCP auth; project-scoped addressing Voyage AI voyage: / voyageai: Hosted Specialist embeddings and reranking; query/document intents xAI (Grok) xai: / grok: Hosted Provider-reported cost, discovered pricing Presets These speak the OpenAI chat-completions dialect closely enough that one shared adapter covers them; what differs is declarative — endpoint, auth spelling, token-field name, model listing, reasoning translation. See presets for the quirk notes. Hosted Services Provider Target Key (conventional env var) Notes AI21 Labs ai21: AI21_API_KEY Jamba model family; max_tokens caps at 4096 Arcee AI arcee: ARCEE_API_KEY Trinity models; the Conductor router (models.arcee.ai/v1) accepts model='auto' as a base-URL override Avian avian: AVIAN_API_KEY Keys carry a literal avian- prefix, which is part of the key rather than something to strip Baseten Model APIs baseten: BASETEN_API_KEY Fixed shared catalog; model listing reports pricing and context metadata Zhipu BigModel (GLM, mainland) bigmodel: / zhipu-cn: ZHIPU_API_KEY The mainland platform behind GLM, and a separate account from z-ai: keys do not cross between them Cerebras Inference cerebras: CEREBRAS_API_KEY Wafer-scale speed; combining tools with response_format is model-dependent Chutes chutes: CHUTES_API_KEY Decentralized open-model serving Clarifai clarifai: CLARIFAI_PAT The credential is a personal access token, not a per-app key Cloudflare AI Gateway (unified) cloudflare-ai-gateway: / cf-ai-gateway: CF_AIG_TOKEN The multi-provider gateway, not Workers AI: it fronts OpenAI, Anthropic, Groq and others behind provider/model ids Cloudflare Workers AI cloudflare-workers-ai: / workers-ai: / cloudflare: CLOUDFLARE_API_TOKEN @cf/author/model ids; the base URL embeds your account id Alibaba Model Studio (Qwen) dashscope: / qwen: / alibaba-qwen: / model-studio: DASHSCOPE_API_KEY International endpoint; keys are region-specific Databricks Model Serving databricks: / mosaic: DATABRICKS_TOKEN Base URL is your workspace host; a personal access token authenticates DeepInfra deepinfra: DEEPINFRA_API_KEY Pay-per-token open models; service_tier extension for priority/flex DigitalOcean Inference digitalocean: / do-inference: / digitalocean-inference: MODEL_ACCESS_KEY Serverless catalog on a fixed host; model-access keys are scopable per-model Featherless AI featherless: FEATHERLESS_API_KEY Very large HF-repo-id catalog, case-sensitive; subscription plans are concurrency-limited rather than token-metered Fireworks AI fireworks: / fireworks-ai: FIREWORKS_API_KEY Model ids look like accounts/fireworks/models/…; over-long max_tokens is silently truncated unless context_length_exceeded_behavior='error' Groq groq: GROQ_API_KEY LPU-served open models; rejects logprobs/logit_bias-style parameters Helicone AI Gateway helicone: / helicone-gateway: HELICONE_API_KEY Routing with observability Hugging Face Inference Providers huggingface: / hf: / huggingface-router: HF_TOKEN Routes HF-hub model ids across serving partners; append :provider to pin one (e.g. moonshotai/Kimi-K2-Instruct:groq) Tencent Hunyuan hunyuan: / tencent: HUNYUAN_API_KEY Reasoning is mostly a model choice (the hunyuan-t1-* line), though hunyuan-a13b instead toggles it in-prompt with a /no_think prefix Hyperbolic hyperbolic: HYPERBOLIC_API_KEY Open-model serving; reasoning models emit inline content Hyperstack AI Studio hyperstack: HYPERSTACK_API_KEY Base URL and model id are both read off the AI Studio playground's API panel, since they follow your deployment rather than a fixed catalog Inception (Mercury) inception: / mercury: INCEPTION_API_KEY Diffusion LLMs rather than autoregressive ones, which shows up in the stream: with provider_options {'diffusing': True} the model emits blocks of noisy tokens that are refined in place, so deltas revise earlier text instead of only appending Inference.net inference-net: / inference: INFERENCE_API_KEY Serverless ids plus team/model deployments; BYOK passthrough via provider headers LightOn Paradigm lighton: / paradigm: LIGHTON_API_KEY EU document-intelligence platform LiteLLM Proxy litellm: / litellm-proxy: — Self-hosted gateway over 100+ providers; authenticate with a proxy-issued virtual key Martian Gateway martian: MARTIAN_API_KEY creator/model ids; routes across upstream providers MiniMax minimax: MINIMAX_API_KEY M-series models; thinking controls via provider_options ({'thinking': {'type': …}}) Mistral AI (La Plateforme) mistral: / mistral-ai: MISTRAL_API_KEY Uses random_seed instead of seed; safe_prompt via provider_options ModelScope (API-Inference) modelscope: / ms-inference: MODELSCOPE_SDK_TOKEN Alibaba's model community Moonshot AI (Kimi) moonshot: / kimi: MOONSHOT_API_KEY Kimi model family; thinking controls via provider_options ({'thinking': …}) Nous Research (Portal) nous: / nousresearch: / hermes: NOUS_API_KEY Hermes models, capitalized ids (Hermes-4-405B) Novita AI novita: / novita-ai: NOVITA_API_KEY max_tokens is required by the API; reasoning models stream reasoning_content Nscale nscale: NSCALE_API_KEY No enforced rate limits; model listing reports pricing and context length Nutanix Enterprise AI nutanix: / nai: NUTANIX_API_KEY On-prem GPT-in-a-Box deployments; the host is your own cluster endpoint NVIDIA NIM (build.nvidia.com) nvidia: / nim: / nvidia-nim: NVIDIA_API_KEY Hosted NIM catalog; self-hosted NIM containers expose the same surface on your own base URL Oracle OCI Generative AI oci-genai: / oci: / oracle: OCI_GENAI_API_KEY Base URL is region-templated Ollama Cloud ollama-cloud: / ollama-turbo: OLLAMA_API_KEY Ollama's hosted catalog, distinct from the local ollama: adapter and needing a real key OVHcloud AI Endpoints ovhcloud: / ovh: OVH_AI_ENDPOINTS_ACCESS_TOKEN EU-hosted unified gateway Parasail parasail: PARASAIL_API_KEY parasail- prefixed model ids; per-model thinking controls (chat_template_kwargs, thinking_budget) via provider_options Perplexity Sonar perplexity: PERPLEXITY_API_KEY Grounded web search built in; search_results ride on the raw payload (retain_raw=True), search filters via provider_options Poe (Quora) poe: POE_API_KEY Hundreds of models and community bots behind one subscription Portkey AI Gateway portkey: PORTKEY_API_KEY Routing, caching and fallbacks over many providers PPIO ppio: / ppinfra: PPIO_API_KEY Formerly PPInfra: the host moved to api.ppio.com/openai, and the widely copied api.ppinfra.com/v3/openai is the legacy spelling Baidu Qianfan (ERNIE) qianfan: / baidu: / ernie: QIANFAN_API_KEY v2 takes a single permanent bearer key shaped bce-v3/ALTAK-/ — pass it whole, since the embedded slashes are part of the key Reka AI reka: REKA_API_KEY Multimodal (image/video/audio) Requesty Router requesty: REQUESTY_API_KEY vendor/model ids across many upstreams Runpod Serverless runpod: RUNPOD_API_KEY The base URL embeds your serverless endpoint id SambaNova Cloud sambanova: SAMBANOVA_API_KEY Model listing reports live per-model pricing and context metadata Sarvam AI sarvam: SARVAM_API_KEY Indic-language models Scaleway Generative APIs scaleway: SCW_SECRET_KEY EU-sovereign; ids carry quantization suffixes (:fp8, :int4) SiliconFlow siliconflow: / silicon-flow: SILICONFLOW_API_KEY The .com host serves the international account; api.siliconflow.cn is the mainland one and keys are not interchangeable Snowflake Cortex snowflake-cortex: / cortex: / snowflake: SNOWFLAKE_PAT Base URL embeds your account identifier; authenticate with a programmatic access token iFlytek Spark spark: / iflytek: SPARK_API_PASSWORD The HTTP surface takes a single bearer APIPassword from the console — not the legacy AppID/APIKey/APISecret triple, which belongs to the WebSocket path and does not work here StepFun stepfun: / step: STEP_API_KEY Step models Together AI together: / together-ai: TOGETHER_API_KEY Large open-model catalog; org/model ids (e.g. deepseek-ai/…) Upstage (Solar) upstage: / solar: UPSTAGE_API_KEY Solar family; reasoning_effort semantics differ per model — solar-mini rejects the parameter entirely, solar-open2 reasons unless disabled Vast.ai Serverless vast-ai: / vast: VAST_API_KEY The base URL ends in your endpoint name and carries no /v1 Venice AI venice: VENICE_API_KEY Privacy-focused; max_tokens is deprecated in favour of max_completion_tokens Vercel AI Gateway vercel-ai-gateway: / vercel: / ai-gateway: AI_GATEWAY_API_KEY creator/model ids across upstream providers; gateway-normalized reasoning object BytePlus ModelArk (Volcengine Ark) volcengine: / ark: / doubao: / bytedance: ARK_API_KEY International (BytePlus) endpoint; the mainland edition is a separate account and host (ark.cn-beijing.volces.com/api/v3) IBM watsonx.ai (model gateway) watsonx: / ibm-watsonx: WATSONX_API_KEY The OpenAI-compatible model gateway (beta, IBM Cloud only), which sidesteps the native API's request-body project scoping and version pinning — providers are registered per project ahead of time instead Z.ai (Zhipu GLM) z-ai: / zai: / glm: ZAI_API_KEY GLM model family; temperature range is 0-1, thinking controls via provider_options ({'thinking': {'type': …}}) Local Engines & Self-Hosted Servers Local engines need no API key and default to loopback. Where the address is yours — a cluster host, a dynamically assigned port — the preset requires a base URL instead. Provider Target Default endpoint Notes Aphrodite Engine aphrodite: http://127.0.0.1:2242/v1 vLLM fork with extra samplers, on port 2242 rather than vLLM's 8000; sampler extensions via provider_options Docker Model Runner docker-model-runner: / dmr: http://127.0.0.1:12434/engines/v1 Built into Docker Desktop Microsoft Foundry Local foundry-local: / foundry-local-service: you supply it On-device ONNX serving GenieX (formerly Nexa SDK) geniex: / nexa: http://127.0.0.1:18181/v1 On-device Snapdragon inference, now published by Qualcomm as GenieX — the CLI is geniex serve, not the older nexa serve GPT4All gpt4all: http://localhost:4891/v1 Minimal local server (enable in settings); no streaming or tool calling documented Jan jan: http://127.0.0.1:1337/v1 Desktop app's local API server (enable it in Jan's settings) KoboldCpp koboldcpp: / kobold: http://127.0.0.1:5001/v1 OpenAI-compatible surface beside the native Kobold API on one port; sampler extensions via provider_options KServe kserve: you supply it Kubernetes model serving Lemonade Server lemonade: http://127.0.0.1:13305/v1 AMD-sponsored server with Ryzen AI NPU backends Llama Stack llama-stack: / llamastack: http://127.0.0.1:8321/v1 Meta's server, fronting vLLM/Ollama/hosted backends llama-swap llama-swap: http://127.0.0.1:8080/v1 A proxy that swaps the upstream llama-server/vLLM process to match each request's model field, so the model id is a config profile name rather than a file llamafile llamafile: http://127.0.0.1:8080/v1 Single self-contained executable serving one model; examples report the model id as the literal string LLaMA_CPP LocalAI localai: / local-ai: http://127.0.0.1:8080/v1 Serves many models at once; ids are gallery names or GGUF filenames MLC-LLM mlc-llm: / mlc: http://127.0.0.1:8000/v1 Compiled-model serving (mlc_llm serve); documents /v1/models and /v1/chat/completions only OpenLLM openllm: http://127.0.0.1:3000/v1 BentoML's server on port 3000; one model per process, launched as 'openllm serve model:version' RamaLama ramalama: http://127.0.0.1:8080/v1 Container-based runner SGLang sglang: http://127.0.0.1:30000/v1 Engine extras (separate_reasoning, top_k, min_p) via provider_options TabbyAPI tabbyapi: / tabby: http://127.0.0.1:5000/v1 ExLlama-family serving; inference calls use the x-api-key header text-generation-webui text-generation-webui: / oobabooga: / textgen-webui: http://127.0.0.1:5000/v1 Start with --api; the model listing reports only the loaded model Text Generation Inference tgi: / text-generation-inference: http://127.0.0.1:3000/v1 Hugging Face's server, default port 3000 NVIDIA Triton (OpenAI frontend) triton: / triton-openai: http://127.0.0.1:9000/v1 The OpenAI frontend listens on 9000 — port 8000 is Triton's own KServe HTTP endpoint, not this one vLLM vllm: http://127.0.0.1:8000/v1 Serves one model per process; engine extras (guided decoding, top_k) via provider_options Xinference xinference: / xorbits: http://127.0.0.1:9997/v1 Serves many models at once; ids are the model UIDs you launched Not Yet Covered Replicate — its predictions API is asynchronous and per-model, with no chat-completions route to normalize; api.replicate.com/openapi.json declares 26 paths and none of them is one. A dedicated async adapter remains the only option. Writer (Palmyra) — serves POST /v1/chat, not /chat/completions, so no OpenAI client can reach it. The best candidate for the next dedicated adapter. Anything else with an OpenAI-compatible endpoint already works today without waiting for a preset — point the generic adapter at it: ai.ProviderSettings.of("openai-compat", base_url="https://your-host/v1", api_key="…") --- # Providers / Hosted & Local Presets Source: https://anyinfer.dev/providers/presets/ Hosted & Local OpenAI-Compatible Presets One implementation, many brandings. Every provider on this page speaks the chat-completions dialect closely enough that AnyInfer's shared OpenAI-compatible adapter covers it, so each ships as a preset: a first-class registered provider with the endpoint, auth spelling, quirks, and capabilities filled in ahead of time. import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("groq", api_key="env://GROQ_API_KEY"), ai.ProviderSettings.of("together", api_key="env://TOGETHER_API_KEY"), ai.ProviderSettings.of("vllm"), # local engines need no key ] ) result = client.generate(prompt, target="groq:llama-3.3-70b-versatile") result = client.generate(prompt, target="together:deepseek-ai/DeepSeek-V3") result = client.generate(prompt, target="vllm:qwen3-8b") Everything the core owns (routing, retries, structured output, telemetry, cost accounting) works identically through a preset. Provider-specific extras go through provider_options, passed to the provider verbatim. The full preset roster (86 services and engines, with each one's target prefix, conventional key variable or default endpoint, and one-line notes) is generated from the registry in the complete inventory, so it cannot drift from what the library ships. What a Preset Does and Does Not Change A preset only adjusts declarative knobs on the shared adapter: the endpoint and how the credential is spelled (Authorization: Bearer vs x-api-key); the output-token parameter name (max_tokens vs max_completion_tokens); whether GET /models exists (absent → discovery reports nothing, and the health probe answers optimistically since there is nothing cheap to probe); whether a base URL is yours rather than the vendor's, as with the account-scoped and region-scoped enterprise endpoints; how normalized reasoning effort is translated, where the provider documents a control; which parameters the provider accepts and silently discards, so they surface as ParameterDropped telemetry instead (Perplexity ignores tools, for example). Anything beyond that (thinking budgets, search filters, sampler extensions) is the provider's own vocabulary and goes through the escape hatch: client.generate( prompt, target="dashscope:qwen-plus", provider_options={"dashscope": {"enable_thinking": True, "thinking_budget": 2048}}, ) Base URLs That Are Yours Some presets need a base URL because it is yours, not theirs: ai.ProviderSettings.of( "cloudflare-workers-ai", base_url="https://api.cloudflare.com/client/v4/accounts//ai/v1", api_key="env://CLOUDFLARE_API_TOKEN", ) ai.ProviderSettings.of("litellm", base_url="http://litellm.internal:4000", api_key="sk-…") ai.ProviderSettings.of( "snowflake-cortex", base_url="https://.snowflakecomputing.com/api/v2/cortex/v1", api_key="env://SNOWFLAKE_PAT", ) ai.ProviderSettings.of( "databricks", base_url="https:///serving-endpoints", api_key="env://DATABRICKS_TOKEN", ) Local Engines Local-engine presets need no key, default to loopback, and a bare hostname expands with the engine's conventional port (ProviderSettings.of("vllm", base_url="gpu-box") → http://gpu-box:8000). The complete inventory lists every engine with its default endpoint. Ports are the easiest thing to get wrong here, and a wrong one fails only at request time, so each is taken from the engine's own documentation and pinned by a test. Two cannot be pinned at all: RamaLama starts at 8080 and walks upward when that is taken, and Foundry Local picks its port when the service starts; read what ramalama serve printed, or foundry service status, rather than trusting a default. Three of these are addressed at a path that is not /v1, which is the likeliest reason a correct key still 404s: Docker Model Runner serves /engines/v1, KServe prefixes its routes with /openai, and older Llama Stack builds nest them under /v1/openai/v1. For deeper local integration (model loading, residency, VRAM awareness), see the dedicated Ollama and llama.cpp adapters. Known Quirks Most preset differences are mechanical. These change behavior, so they are worth knowing before debugging them: Tencent Hunyuan stops after the stop sequence, where OpenAI stops before it. The stop strings will appear in the output. Code that uses a stop token as a delimiter and then splits on it will silently see an extra fragment. Helicone inverts the routing syntax. Most routers take vendor/model; Helicone takes model/vendor (gpt-4o-mini/openai), and a bare model id allows the gateway to choose the upstream. Inception's diffusion models revise text they already streamed. With provider_options={"inception": {"diffusing": True}}, deltas carry noisy tokens that later deltas refine in place rather than appending to. Code that concatenates deltas will produce nonsense; the flag is off by default for exactly that reason. Vast.ai routes by base URL, not the model field. The base URL ends in the endpoint name and carries no /v1, and the proxy ignores the model field; the served model is whatever the endpoint was configured with, so any non-empty model string works and a typo there fails silently instead of erroring. Sarvam reasons unless told not to. reasoning_effort defaults to low rather than off, so requests expected to be cheap will think first. Embeddings Chat compatibility does not imply embeddings compatibility: an OpenAI-shaped /v1/chat/completions says nothing about whether /v1/embeddings exists at all, so every preset stays generation-only by default. Four have been verified live against their own documentation and opt in: Together AI, Fireworks AI, DeepInfra, and Mistral. result = client.embed( ["first text", "second text"], target="together:togethercomputer/m2-bert-80M-8k-retrieval", ) Together's response carries no usage block, so result.usage stays unset for that preset specifically; every other verified preset reports it. Mistral's dimensionality control is spelled output_dimension, not the shared dialect's dimensions, so a dimensions= request on mistral: is silently ignored on the wire; use provider_options={"mistral": {"output_dimension": N}} instead. Every other preset remains generation-only, including presets whose underlying engine is known to serve OpenAI-compatible embeddings in general (self-hosted engines like vLLM, for instance); verifying that a specific deployment exposes it is outside what a static preset table can promise, so it stays off unless independently confirmed. See contracts/openai-compat-presets.md for verification dates and sources. Cost Accounting Presets participate in cost computation like every other provider: models with entries in the bundled pricing table report usage.cost_usd automatically, and unknown prices remain unknown rather than reading as zero. Anthropic-Compatible Endpoints Several of these providers (Moonshot, Z.ai, MiniMax, SambaNova, Vercel's gateway, and others) also expose an Anthropic-Messages-compatible endpoint, reachable by pointing the Anthropic adapter at it. Use the OpenAI-compatible preset unless Messages-dialect behavior is specifically needed. Verification Endpoint, auth, and quirk data for every preset was verified against the provider's live documentation; the per-provider details, dates, and sources live in the contract snapshot, which the provider drift check re-audits. --- # Providers / OpenAI Source: https://anyinfer.dev/providers/openai/ OpenAI Uses the Responses API, OpenAI's current surface, which exposes reasoning effort and reasoning-token accounting the older chat-completions shape does not. For the chat-completions dialect, point openai-compat at https://api.openai.com/v1 instead. streaming structured output tool calls health discovery Setup client = ai.Client( [ ai.ProviderSettings.of("openai", api_key="env://OPENAI_API_KEY"), ] ) result = client.generate(prompt, target="openai:gpt-5") Supported Behavior Support Streaming Native, typed events Structured output json_schema via text.format Tools Native Reasoning reasoning.effort, plus reasoning-token counts Usage Input, output, cached, reasoning tokens Cost Cataloged pricing Reasoning result = client.generate(prompt, target="openai:gpt-5", reasoning="high") result.usage.reasoning_tokens Effort levels pass straight through: minimal, low, medium, high. Embeddings The dedicated adapter serves POST /v1/embeddings through the shared OpenAI-compatible dialect: result = client.embed( ["first text", "second text"], target="openai:text-embedding-3-small", ) Requests larger than the API's 2,048-input ceiling are split by the core and re-assembled in input order. Requested dimensions are forwarded (text-embedding-3 and later). OpenAI's request schema has no input-intent concept, so passing input_type adds a warning to the result rather than silently doing nothing. There is no reranking endpoint on this API. Multimodal Inputs Images and files are projected to Responses API input_image and input_file content items. Inline bytes become data URLs; remote URLs stay remote. Audio input is model-specific, so capability data must not be read as a promise that every OpenAI model accepts it. Notes System messages become the top-level instructions field. The output-token parameter is max_output_tokens. A response truncated by the token cap reports finish_reason == "length". Request-level extras such as store and service_tier pass through the escape hatch: provider_options = {"openai": {"store": False, "service_tier": "flex"}}. Wire Contract For the exact request/response fields this adapter depends on, see contracts/openai.md. --- # Providers / Anthropic Source: https://anyinfer.dev/providers/anthropic/ Anthropic The Messages API over raw httpx2. Registered as anthropic, with the alias claude. streaming structured output (emulated) tool calls health discovery Setup client = ai.Client( [ ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"), ] ) result = client.generate(prompt, target="anthropic:claude-sonnet-4-5") No extra required. Supported Behavior Support Streaming Native, typed SSE events Structured output Emulated as a forced tool call Tools Native Reasoning Extended thinking, budgeted in tokens Usage Input, output, plus cache read/write Cost Cataloged pricing Reasoning Since Anthropic budgets thinking in tokens rather than naming levels, reasoning effort maps to a budget: Effort Wire form minimal {"type": "disabled"} low 1024 tokens medium 4096 tokens high 16384 tokens Thinking arrives as ReasoningDelta events on the event stream. Thinking starts the first-token clock (the model is working and the user sees activity), but its text is excluded from the answer. Structured Output Anthropic has no response_format field, so a schema becomes a single forced tool call, which the API does constrain. The caller still gets a normal validated result.structured; the emulation is invisible except in structured_mechanism. See structured output for how mechanisms are chosen. Pointing This Adapter Elsewhere Several other services expose an Anthropic-Messages-shaped endpoint. This adapter serves any of them through a base_url override; nothing else changes: ai.ProviderSettings.of( "anthropic", base_url="https://api.deepseek.com/anthropic", api_key="env://DEEPSEEK_API_KEY", ) DeepSeek, xAI, and several of the presets document such endpoints. Prefer the provider's own adapter or preset unless Messages-dialect behavior is specifically needed, since provider-specific features are wired only there. Multimodal Inputs Images and PDF documents accept inline bytes or provider-fetchable URLs. The adapter emits native image/document content blocks. Audio input is not part of this Messages projection and fails explicitly. See multimodal inputs for the normalized input model. Notes System messages become the top-level system field. max_tokens is required by the API; AnyInfer sends 4096 when none is set rather than letting the request fail with a 400. Tool results ride on a user turn in this dialect, not a tool role. Model listing is cursor-paginated, and pagination is followed automatically. Wire Contract For the exact request/response fields this adapter depends on, see contracts/anthropic.md. --- # Providers / Google Gemini Source: https://anyinfer.dev/providers/gemini/ Google Gemini The native generateContent protocol. Google's OpenAI-compatibility layer is documented as beta and ignores parameters it does not implement, while thinking levels, response schemas, safety settings, and context caching are native-only or better supported. streaming structured output tool calls reasoning health discovery (context windows) Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("gemini", api_key="env://GEMINI_API_KEY"), ] ) result = client.generate(prompt, target="gemini:gemini-2.5-flash") The key is sent as x-goog-api-key. google:, google-gemini:, and ai-studio: are accepted as aliases of gemini:. Reasoning Gemini names thinking levels rather than budgeting tokens, so the four normalized effort levels map straight across: result = client.generate(prompt, target="gemini:gemini-2.5-pro", reasoning="high") print(result.usage.reasoning_tokens) # thoughts, reported separately print(result.usage.output_tokens) # answer + thoughts, because both bill as output Gemini's own candidatesTokenCount excludes thoughts even though they bill at the output rate, so AnyInfer reports output_tokens as the sum; cost stays right, and reasoning_tokens keeps the breakdown visible. Thinking text arrives as ReasoningDelta events and is excluded from result.text: with client.stream(prompt, target="gemini:gemini-2.5-flash", reasoning="medium") as stream: for event in stream: if isinstance(event, ai.ReasoningDelta): print("[thinking]", event.text, end="") elif isinstance(event, ai.TextDelta): print(event.text, end="") Models that cannot disable thinking (2.5 Pro, and the Gemini 3 family) clamp a low request upward server-side rather than failing. Structured Output SUMMARY = { "type": "object", "properties": { "headline": {"type": "string"}, "points": {"type": "array", "items": {"type": "string"}}, }, "required": ["headline", "points"], } result = client.generate(article, target="gemini:gemini-2.5-flash", schema=SUMMARY) print(result.structured["headline"]) Gemini's responseSchema accepts an OpenAPI subset, and rejects the entire request on an unknown keyword. AnyInfer projects the schema down to the accepted subset before sending (dropping things like $schema, pattern, and unevaluatedProperties), and then validates the response against the original schema. Some wire-level strictness is lost, never result correctness. Tool Calling result = client.generate(prompt, target="gemini:gemini-2.5-flash", tools=[lookup_spec]) for call in result.tool_calls: print(call.name, call.arguments) Gemini emits complete function calls rather than streamed argument fragments, and supports several calls in one turn. Tool results are sent back on a user turn as functionResponse parts; the adapter handles that translation, so run_tools() works the same as everywhere else. Embeddings Gemini embeds through batchEmbedContents, so batches are native: result = client.embed( ["What is deep learning?"], target="gemini:gemini-embedding-2", dimensions=768, # 128-3072; both models default to 3072 ) The legacy gemini-embedding-001 accepts task types, mapped from input_type (query → RETRIEVAL_QUERY and so on). The current gemini-embedding-2 documents no task types (prompt instructions replace them), so an input_type there is never sent and the result says so in a warning. No batch ceiling is documented, so requests above the library's sanity ceiling are refused rather than split at a guessed size; set BatchPolicy.max_items_override after independently verifying a limit. There is no reranking endpoint on this API. Discovery The model listing reports real limits, so context windows carry discovered provenance: for model in client.models("gemini"): caps = model.capabilities if caps and caps.context_window: print(model.id, caps.context_window.value, caps.context_window.provenance) Reaching Native Features Anything AnyInfer does not model (context caching, safety settings, grounded search, numeric thinking budgets) passes straight through the escape hatch: client.generate( prompt, target="gemini:gemini-2.5-flash", provider_options={ "gemini": { "cachedContent": "cachedContents/abc123", "safetySettings": [ {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_ONLY_HIGH"} ], } }, ) Content Filtering A prompt Gemini blocks returns no candidates at all, with the reason on promptFeedback. That surfaces as finish_reason == "content_filter" rather than an empty successful answer, and a Route with content_policy_targets can redirect it to a differently-governed provider. Multimodal Inputs Images, documents, and audio use native inlineData blocks for bytes and fileData for remote references. Support and limits remain model-specific. Wire Contract For the exact request/response fields this adapter depends on, see contracts/gemini.md. See Also Capabilities and provenance: how discovered limits are ranked. Routing and rate limits: including the content-policy fallback chain. Google Vertex AI: the same models with GCP auth. --- # Providers / DeepSeek Source: https://anyinfer.dev/providers/deepseek/ DeepSeek An OpenAI-compatible dialect with three deltas that would otherwise cost a developer silently: reasoning arrives on its own channel, thinking is on by default, and cache accounting is automatic and split. streaming structured output (JSON mode) tool calls reasoning cache accounting Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("deepseek", api_key="env://DEEPSEEK_API_KEY"), ] ) result = client.generate(prompt, target="deepseek:deepseek-v4-pro") Two models are served: deepseek-v4-flash and deepseek-v4-pro. The old deepseek-chat / deepseek-reasoner aliases were discontinued in July 2026. Reasoning Thinking is on by default. Chain-of-thought streams as ReasoningDelta events, separate from the answer: with client.stream(prompt, target="deepseek:deepseek-v4-pro") as stream: for event in stream: if isinstance(event, ai.ReasoningDelta): print("[thinking]", event.text, end="") elif isinstance(event, ai.TextDelta): print(event.text, end="") Requesting an effort level enables thinking explicitly and sets the level. Because DeepSeek accepts low/high/max, AnyInfer maps minimal and low to low, and medium and high to high, rather than sending a value the API would rewrite: result = client.generate(prompt, target="deepseek:deepseek-v4-pro", reasoning="low") To turn thinking off (a behavior change, not an effort setting): client.generate( prompt, target="deepseek:deepseek-v4-pro", provider_options={"deepseek": {"thinking": {"type": "disabled"}}}, ) Sampling is ignored while thinking DeepSeek silently discards temperature and top_p in thinking mode, which is the default. AnyInfer declares both as ignored, so setting one raises a ParameterDropped telemetry event instead of silently doing nothing. Cache Accounting Context caching is automatic: no opt-in, no cache-control parameters. DeepSeek reports the split, and cache hits bill at a much lower rate: result = client.generate(long_prompt, target="deepseek:deepseek-v4-flash") print(result.usage.input_tokens) # hits + misses print(result.usage.cache_read_tokens) # the part that was cheap Cost is a ceiling here The bundled pricing table records the standard (cache-miss) rate, so usage.cost_usd overstates spend on cache-heavy workloads; supply capability_overrides for the blended rate. The Anthropic-Compatible Endpoint DeepSeek also exposes a Messages endpoint at https://api.deepseek.com/anthropic, which the Anthropic adapter serves through a base-URL override; see pointing that adapter elsewhere. Use the native deepseek: provider unless Messages-dialect behavior is specifically needed; the reasoning channel and cache accounting above are only wired there. Wire Contract For the exact request/response fields this adapter depends on, see contracts/deepseek.md. See Also Presets: other OpenAI-compatible providers serving DeepSeek models. Prompt caching: how cache reads show up in usage and cost. --- # Providers / xAI (Grok) Source: https://anyinfer.dev/providers/xai/ xAI (Grok) An OpenAI-compatible dialect whose distinctive value is reported cost: xAI reports the exact amount billed on every response, and its model listing carries real prices and context windows. streaming structured output tool calls reasoning discovery (pricing + context) Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("xai", api_key="env://XAI_API_KEY"), ] ) result = client.generate(prompt, target="xai:grok-4.5") grok: is accepted as an alias. Exact Cost Most providers report token counts and leave the arithmetic to a price table. xAI returns cost_in_usd_ticks: the amount actually billed, including server-side tool fees and tiered pricing, and AnyInfer adopts it directly for cost accounting: result = client.generate(prompt, target="xai:grok-4.5") print(result.usage.cost_usd) # what you were charged, not an estimate That matters here because several of xAI's billing rules can't be reproduced from per-token rates: prompts of 200k tokens or more bill every token at the higher tier, web search costs $5 per 1,000 calls, and code execution costs $5 per 1,000 invocations. Discovery The language-models listing reports context windows and per-token prices, so capabilities arrive with discovered provenance rather than cataloged estimates: for model in client.models("xai"): caps = model.capabilities if caps and caps.pricing: print(model.id, caps.pricing.value.input_per_1m, caps.pricing.provenance) If that endpoint is unavailable, discovery degrades to the plain model listing (ids only) rather than failing. Reasoning result = client.generate(prompt, target="xai:grok-4.5", reasoning="medium") minimal clamps to low: only some Grok models accept none, and silently disabling reasoning on a reasoning model would change the answer more than was asked for. Pass provider_options={"xai": {"reasoning_effort": "none"}} to disable it deliberately on a model that supports it. The grok-4.20 generation ships as separate -reasoning and -non-reasoning model ids instead of taking the parameter. The Anthropic-Compatible Endpoint xAI also exposes a Messages endpoint at https://api.x.ai, which the Anthropic adapter serves through a base-URL override; see pointing that adapter elsewhere. Use the native xai: provider unless Messages-dialect behavior is needed; the reported cost and discovered pricing above are only wired there. Wire Contract For the exact request/response fields this adapter depends on, see contracts/xai.md. See Also Capabilities and provenance: why discovered pricing outranks the bundled table. --- # Providers / Google Vertex AI Source: https://anyinfer.dev/providers/vertex/ Google Vertex AI The same Gemini models as the AI Studio API, over the same protocol, with enterprise addressing and Google Cloud authentication. AnyInfer reuses the Gemini adapter's translation wholesale: Vertex changes where requests go and how they are signed, not what they look like. streaming structured output tool calls reasoning discovery (no listing endpoint) Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of( "vertex", options={"project": "my-gcp-project", "location": "global"}, ), ] ) result = client.generate(prompt, target="vertex:gemini-2.5-flash") project is required; it is part of the request path, not a header. location defaults to global; newer models are served only from the global endpoint, while a regional value (us-central1) selects that region's host. vertex-ai: and google-vertex: are accepted aliases. Authentication Vertex takes an OAuth access token, not an API key, so the credential is acquired and refreshed instead of configured once. Three ways, in precedence order: Application default credentialsA service-account keyA pre-acquired token Install google-auth, then let its standard credential chain select metadata-server, workload-identity, or local gcloud credentials: pip install google-auth ai.ProviderSettings.of("vertex", options={"project": "my-project"}) AnyInfer uses google-auth when it is installed. The library is optional because it is not needed for the explicit-token or service-account paths. pip install "anyinfer[vertex]" ai.ProviderSettings.of( "vertex", options={ "project": "my-project", "credentials_file": "/secrets/sa.json", }, ) Falls back to GOOGLE_APPLICATION_CREDENTIALS. The JWT is signed and exchanged in-house, so this works without google-auth (though signing needs an RSA implementation, and the error says so if none is available). ai.ProviderSettings.of( "vertex", api_key="env://GCP_ACCESS_TOKEN", options={"project": "my-project"} ) From gcloud auth print-access-token. Used verbatim and never refreshed; its lifetime is the caller's to manage. Note this is an access token, not a Gemini API key; the two are not interchangeable. Acquired tokens are cached until two minutes before expiry, so a long-running client pays for one exchange per hour, not one per request. Everything Else Is Gemini Thinking levels, response schemas, function calling, and usage accounting all behave exactly as on the Gemini page, including output_tokens counting answer plus thinking: result = client.generate(prompt, target="vertex:gemini-2.5-pro", reasoning="high") print(result.usage.reasoning_tokens) Discovery Vertex exposes no listing endpoint comparable to AI Studio's, so client.models("vertex") returns an empty list; under the provenance rules, an invented inventory must not be presented as discovery. Name models explicitly in the target, and supply capability_overrides to make their windows known. Health checks that a token can be acquired, without spending a generation. Embeddings Embeddings do not reuse the Gemini shape: Vertex's own text-embeddings API uses a predict verb with an instances/parameters body, so VertexAdapter overrides embedding translation instead of Gemini's batchEmbedContents: result = client.embed( ["first text", "second text"], target="vertex:text-embedding-005", ) gemini-embedding-001 accepts only one input per request (a documented Vertex limit), so the core's batching policy fans a multi-text call into one request per input. text-embedding-005 and text-multilingual-embedding-002 accept up to five. dimensions= requests native truncation via outputDimensionality; input_type= maps to Vertex's task_type (query→RETRIEVAL_QUERY, document→RETRIEVAL_DOCUMENT, classification→CLASSIFICATION, clustering→CLUSTERING). Claude on Vertex Vertex also serves Anthropic models, but through a different surface (rawPredict/streamRawPredict with the Messages body). This adapter does not cover it; point the Anthropic adapter at that endpoint instead. See Also Contract snapshot Google Gemini: the same models with API-key auth. --- # Providers / AWS Bedrock Source: https://anyinfer.dev/providers/bedrock/ AWS Bedrock The Converse API: Bedrock's unified interface, where one request shape serves Claude, Nova, Llama, Mistral, and DeepSeek alike. Generation never uses InvokeModel and its per-model request bodies; only embeddings do, because Converse has no embeddings surface. streaming structured output tool calls reasoning cache accounting Setup Two ways to authenticate. A Bedrock API key is the simplest: import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of( "bedrock", api_key="env://AWS_BEARER_TOKEN_BEDROCK", options={"region": "us-east-1"}, ), ] ) result = client.generate(prompt, target="bedrock:us.anthropic.claude-sonnet-4-5") Or AWS credentials, which are SigV4-signed per request: ai.ProviderSettings.of("bedrock", options={"region": "us-west-2"}) With no explicit key, credentials are resolved in order: explicit aws_access_key_id / aws_secret_access_key options, then boto3's chain if it is installed (it knows about SSO caches, instance metadata, and profiles), then AWS_ACCESS_KEY_ID and friends from the environment. Neither boto3 nor any other SDK is a dependency; signing is implemented against the standard library. aws-bedrock: and amazon-bedrock: are accepted aliases. Model Ids Bedrock accepts base model ids, inference-profile ids, and full ARNs. Cross-region inference profiles carry a region prefix: client.generate(prompt, target="bedrock:us.anthropic.claude-sonnet-4-5") client.generate(prompt, target="bedrock:amazon.nova-pro-v1:0") Ids containing colons work because targets split on the first colon only. Binary Streaming ConverseStream answers with AWS's vnd.amazon.eventstream framing and offers no SSE or JSON alternative. AnyInfer decodes it, including verifying both frame checksums; from the application's side it is an ordinary stream: with client.stream(prompt, target="bedrock:us.anthropic.claude-sonnet-4-5") as stream: for event in stream: if isinstance(event, ai.TextDelta): print(event.text, end="", flush=True) print(stream.result.usage.input_tokens) Why usage only appears at the end Bedrock sends token counts only in the terminal metadata event, after messageStop. A client that stopped at the stop reason would report zero tokens for every request; AnyInfer reads through to the metadata frame. Structured Output Converse has no response-format field, so a schema is emulated as a single forced tool call (the same approach the Anthropic adapter takes, because the API genuinely constrains tool input): result = client.generate(article, target="bedrock:amazon.nova-pro-v1:0", schema=SUMMARY) print(result.structured["headline"]) print(result.structured_mechanism) # "json_schema" See structured output for how mechanisms are chosen and validated. Reasoning Converse has no reasoning parameter of its own; extended thinking is a model-specific field, so normalized effort travels in additionalModelRequestFields: result = client.generate(prompt, target="bedrock:us.anthropic.claude-sonnet-4-5", reasoning="high") Thinking text arrives as ReasoningDelta events and stays out of result.text. Models without extended thinking ignore the field. Provider-Specific Parameters Anything Converse does not model (Claude's top_k, guardrail configuration, a service tier) passes through provider_options, under keys such as additionalModelRequestFields and guardrailConfig. See the escape hatch. Discovery and Health Discovery reads the Bedrock control plane, a different host than the runtime. An account without bedrock:ListFoundationModels gets an empty list rather than an error; a permission gap should not make the provider look broken. Health makes no network call at all: every runtime endpoint costs a generation, and the control plane may be denied by policy even when inference works perfectly. It reports whether credentials are present. Pricing Bedrock prices per model and per region; the bundled table carries the common us-east-1 on-demand rates for the Nova family and Claude, and capability_overrides covers rates that differ; see cost and spending. Embeddings Since Converse has no embeddings surface, embeddings go through the older InvokeModel action (a separate code path from generation, sharing only auth and addressing): result = client.embed( ["first text", "second text"], target="bedrock:amazon.titan-embed-text-v2:0", ) Titan Text Embeddings V2 accepts one inputText per call, so the adapter declares max_batch_inputs=1 and the core's batching policy fans a multi-text call into one request per input. dimensions= requests native truncation to 1024 (default), 512, or 256; the model has no input-intent concept. Cohere Embed v3 runs on the same action, under a different body shape selected by the cohere. model-id prefix. It is batch-capable (up to 96 texts per call) and requires input_type: result = client.embed( ["first text", "second text"], target="bedrock:cohere.embed-english-v3", input_type="document", ) Bedrock's Cohere embed response reports no token usage, so result.usage is None, never a guessed value. Reranking Rerank is a third action entirely: bedrock-agent-runtime's POST /rerank, a different host than InvokeModel/Converse (though SigV4-signed under the same bedrock service name). It is model-agnostic at the wire level: the same request/response shape serves both amazon.rerank-v1:0 and cohere.rerank-v3-5:0, selected only by the modelArn the adapter builds from the model id. Up to 1,000 documents per call; top_n maps to the action's native numberOfResults. No usage/search-unit field is reported. result = client.rerank( query="What is Amazon Bedrock?", documents=["Amazon Bedrock is a fully managed service.", "Amazon S3 is storage."], target="bedrock:cohere.rerank-v3-5:0", ) Multimodal Inputs Converse image, document, and audio blocks are used directly. Inline inputs are base64; remote image/document references must be S3 URIs. The selected model still decides which block types it accepts. Wire Contract For the exact request/response fields this adapter depends on, see contracts/bedrock.md. See Also Anthropic: Claude direct, without the Bedrock layer. Routing and rate limits: retries and fallback across regions or providers. --- # Providers / Cohere Source: https://anyinfer.dev/providers/cohere/ Cohere The native v2 Chat API, chosen over the OpenAI compatibility layer because v2 is where the things worth choosing Cohere for live: grounded generation with document citations, a separate thinking channel, and usage that distinguishes what was processed from what was billed. streaming structured output tool calls reasoning discovery (context lengths) Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of("cohere", api_key="env://CO_API_KEY"), ] ) result = client.generate(prompt, target="cohere:command-a-03-2025") Dialect Differences Cohere's API diverges from the OpenAI shape in ways the adapter normalizes, but which show up when reaching past it with provider_options: AnyInfer Cohere finish_reason == "stop" COMPLETE (uppercase enum) tool_choice="required" "REQUIRED"; there is no way to name one tool Sampling(top_p=...) p, not top_p optional streaming stream is required on every request Unknown finish reasons normalize to "other" rather than propagating. Reasoning Cohere budgets thinking in tokens, so normalized effort maps to a budget: minimal disables thinking, and low/medium/high map to increasing token budgets. Thinking blocks arrive as ReasoningDelta events on the event stream and stay out of result.text. result = client.generate(prompt, target="cohere:command-a-03-2025", reasoning="high") Usage Accounting Cohere reports both billed_units and tokens. AnyInfer's counts follow tokens (what the model actually processed, which is what a context window measures): result = client.generate(prompt, target="cohere:command-a-03-2025") print(result.usage.input_tokens) # processed If billed units are needed for cost reconciliation, build the client with retain_raw=True and read them off result.raw. Grounded Generation client.generate( question, target="cohere:command-a-03-2025", provider_options={ "cohere": { "documents": [{"id": "doc1", "data": {"text": "..."}}], "citation_options": {"mode": "ACCURATE"}, } }, ) Document grounding and citations are reachable through the escape hatch. Citations are not yet surfaced as typed results; read them from result.raw until they are modeled. Embeddings and Reranking Cohere serves both operations natively (POST /v2/embed, POST /v2/rerank), and is the first provider here with native input intents and native rerank scores: docs = client.embed( ["the cat sat on the mat", "stock markets rallied"], target="cohere:embed-v4.0", input_type="document", ) ranked = client.rerank( "where did the cat sit", ["stock markets rallied", "the cat sat on the mat"], target="cohere:rerank-v3.5", top_n=1, ) Three things worth knowing: input_type is required. Cohere's embed API demands an intent and documents no default, so an intent-less embed() is refused with a hint rather than guessed; query and document embeddings are not comparable unless produced with matching intents. Batching engages at 96 inputs. The endpoint accepts at most 96 texts per call; larger requests are split by the core and re-assembled in input order, invisibly. Requested dimensions are forwarded as output_dimension (embed-v4 models only). Rerank usage is search units, not tokens. Live rerank responses report only billed_units.search_units, which AnyInfer never encodes as fake token counts, so result.usage is typically empty for rerank. The billed units are on result.raw["meta"]["billed_units"] for cost reconciliation. Discovery The model listing reports real context lengths, so windows carry discovered provenance: for model in client.models("cohere"): caps = model.capabilities if caps and caps.context_window: print(model.id, caps.context_window.value, caps.context_window.provenance) Every model is listed (embedding and rerank models included), with its operations derived from the listing's endpoints field, so client.models("cohere", operation="embedding") answers from discovery rather than a guess. Wire Contract For the exact request/response fields this adapter depends on, see contracts/cohere.md. See Also Structured output: how the schema mechanism is chosen. --- # Providers / Azure AI Foundry Source: https://anyinfer.dev/providers/azure-foundry/ Azure AI Foundry An openai-compat subclass carrying Azure's parameter renames and its two authentication modes. streaming structured output tool calls health discovery Setup API keyEntra client = ai.Client( [ ai.ProviderSettings.of( "azure-foundry", base_url="https://.services.ai.azure.com/openai/v1", api_key="env://AZURE_AI_KEY", ), ] ) result = client.generate(prompt, target="azure-foundry:gpt-5") pip install "anyinfer[azure]" az login ai.ProviderSettings.of( "azure-foundry", base_url="https://.services.ai.azure.com/openai/v1", # no api_key: DefaultAzureCredential is used ) Aliases: azure, foundry. Differences from openai-compat Azure Output-token parameter max_completion_tokens Reasoning effort flat reasoning_effort field Auth header api-key, or Authorization: Bearer for Entra API version optional api-version query parameter Sending max_tokens to Azure is rejected outright, which is why the subclass exists. API Versions ai.ProviderSettings.of("azure-foundry", base_url=..., api_version="2024-10-21") Only needed for deployments that still require it; the newer /openai/v1 surface does not. Chat, embeddings, and model listing all carry it consistently. Embeddings result = client.embed( ["first text", "second text"], target="azure-foundry:text-embedding-3-small", ) target's model half is the deployment name, not necessarily the underlying model's catalog id. The same POST {base_url}/embeddings surface as chat (deployment-less on /openai/v1, or api-version-pinned on the older surface) speaks the identical OpenAI-compatible body. Azure documents the same request ceilings OpenAI itself does: 2,048 inputs per request, 8,192 tokens per input, and 300,000 tokens aggregate. Since the deployment name is tenant-chosen, AnyInfer does not declare these as static per-model capabilities; a request larger than what the deployment accepts surfaces as a provider error rather than a pre-flight refusal. Troubleshooting could not acquire an Entra token: run az login, or configure a service principal in the environment. The error names the scope it tried. azure-foundry requires the base URL of your Foundry resource: the resource endpoint is deployment-specific and cannot be defaulted. Wire Contract For the exact request/response fields this adapter depends on, see contracts/azure-foundry.md. --- # Providers / GitHub Copilot Source: https://anyinfer.dev/providers/copilot/ GitHub Copilot The only adapter that is not raw HTTP. Copilot is reached by driving the Copilot CLI as a subprocess runtime through github-copilot-sdk, so there is no wire protocol for AnyInfer to speak (the slim core grants this one SDK exception, behind an extra). streaming structured output (prompt-only) tool calls health discovery Setup pip install "anyinfer[copilot]" copilot login # authentication is delegated to the CLI client = ai.Client([ai.ProviderSettings.of("copilot")]) result = client.generate(prompt, target="copilot:gpt-4.1") No API key is handled by AnyInfer; the CLI owns the credential. Override CLI discovery with options={"cli_path": "/path/to/copilot"} or the COPILOT_CLI_PATH environment variable. Alias: github-copilot. Supported Behavior Support Streaming SDK event callbacks Structured output Prompt-injected only (no native mode) Tools Not supported; declared as ignored, so requesting them emits ParameterDropped Usage Input, output, cache read/write, reasoning tokens Cost Not reported The auto Sentinel result = client.generate(prompt, target="copilot:auto") Copilot picks the model at request time. Capabilities for auto are therefore the conjunction across every model it might choose: the minimum of each numeric bound, the intersection of feature flags. Claiming more would be a promise that could not be verified until a request failed. See capabilities. Token Calibration The CLI runtime builds its own request around the prompt (an agent system preamble, its built-in tool declarations, workspace framing), none of which appears in the messages this library serializes. Copilot's reported prompt tokens therefore run well above the bytes sent, and consistently enough to correct for, so the descriptor declares a token calibration: a 2.4× multiplier plus 1,200 flat tokens. budget = client.budget(messages, target="copilot:auto") budget.estimate.envelope.tokens # the harness, counted separately from your prompt It moves the planning figure only, so budget() packs conservatively here while the pre-dispatch gate stays as permissive as it is everywhere else. Structured Output Copilot has no structured-output mode, so a schema is described in the prompt and validated client-side (the core's fallback path for providers without a native mode). The caller still gets a validated result.structured; structured_mechanism will read "prompt". Consider repair=ai.Repair(max_attempts=1) here: prompt-only enforcement has a higher first-attempt failure rate than grammar or json_schema modes. Troubleshooting the copilot provider requires the github-copilot-sdk extra: pip install 'anyinfer[copilot]'. AuthError mentioning login: run copilot login. CLI not found: install the Copilot CLI and put it on PATH, or set COPILOT_CLI_PATH. Notes Sessions take a system prompt plus one user turn, so prior turns are folded into the user prompt with role markers rather than being silently dropped. Wire Contract For the exact request/response fields this adapter depends on, see contracts/copilot.md. --- # Providers / Microsoft 365 Copilot Source: https://anyinfer.dev/providers/m365-copilot/ Microsoft 365 Copilot The most constrained provider in the set. streaming structured output (prompt-only) tool calls health (no-op) discovery Setup pip install "anyinfer[azure]" client = ai.Client( [ ai.ProviderSettings.of( "m365-copilot", options={"tenant_id": "...", "client_id": "..."}, ), ] ) result = client.generate(prompt, target="m365-copilot:m365-copilot") Alias: m365. Interactive Authentication There is no client-credential or daemon flow for this API. Sign-in opens a browser. Consequently: It cannot run headless, in CI, or in a container without a human present. It is exempt from live conformance testing. health() does not trigger a sign-in: a health probe that opens a browser window would be a hostile surprise, and the router calls it speculatively. If a token is already held, supply it and skip the interactive flow: ai.ProviderSettings.of("m365-copilot", api_key="env://M365_TOKEN") This is the only workable path for automated use. Supported Behavior Support Streaming No (a whole response, emitted as one delta) Structured output Prompt-injected only Tools No Sampling controls Ignored by the service Usage Generally absent Schema repair Capped at one round trip Ignored Parameters Temperature, top-p, max tokens, stop sequences, tools, and reasoning effort are declared on the descriptor as ignored, so requesting them emits a ParameterDropped telemetry event rather than silently doing nothing: class Watch: def on_event(self, event): if isinstance(event, ai.ParameterDropped): log.warning("%s ignored %s", event.target, event.parameter) That event matters: a temperature=0 that had no effect is otherwise indistinguishable from one that worked. Schema Repair The descriptor caps schema repair at a single round trip, since every request here is an interactively-authenticated Graph call against service-kept conversation state (the most expensive request shape in the registry). Asking for more is clamped rather than refused, and the clamp is reported the same way an ignored parameter is: result = client.generate( prompt, target="m365-copilot:default", schema=PERSON, repair=ai.Repair(max_attempts=3), ) # ParameterDropped(parameter="repair.max_attempts", reason="... at most 1 ... 3 requested") Prompt-only schema enforcement plus one repair is a real failure rate. Validate result.structured defensively, and prefer a provider with a native structured-output mode when the shape matters more than the M365 grounding does. Notes Citations and attributions are retained on result.raw rather than normalized: build the client with retain_raw=True to keep them, since raw payloads are discarded by default. v1 has no typed model for them, and inventing one would freeze a shape before it is understood. A 401/403 hints at both re-authentication and the tenant licensing and admin-consent requirements, because those are the usual real causes. For automation, streaming, tools, or sampling control, use another provider; GitHub Copilot covers the subscription-billed case with fewer constraints. This adapter is for applications with a genuine M365 Copilot requirement. Wire Contract For the exact request/response fields this adapter depends on, see contracts/m365-copilot.md. --- # Providers / OpenRouter Source: https://anyinfer.dev/providers/openrouter/ OpenRouter An openai-compat subclass. Its distinctive value is the model listing: OpenRouter reports per-model context length and per-token pricing, so its costs carry discovered provenance rather than cataloged estimates. streaming structured output (model-dependent) tool calls health discovery (pricing + context) Setup client = ai.Client( [ ai.ProviderSettings.of( "openrouter", api_key="env://OPENROUTER_API_KEY", options={"http_referer": "https://myapp.example", "x_title": "My App"}, ), ] ) result = client.generate(prompt, target="openrouter:anthropic/claude-sonnet-4.5") Model ids are namespaced vendor/model. The attribution headers are optional. Discovery The listing's prices are parsed with Decimal and arrive with discovered provenance, beating the bundled table. Feature flags come from each model's supported_parameters, where absence means unsupported: OpenRouter enumerates what a model accepts, so claiming more would send requests the upstream provider silently drops. Notes Keep-alive comment lines (: OPENROUTER PROCESSING) are ignored by the SSE parser. A 402 (insufficient credits) is reported distinctly, hinting to add credits or pick a free-tier model. Upstream routing means the served model may differ from the one requested; the response echoes what actually served it. Wire Contract For the exact request/response fields this adapter depends on, see contracts/openrouter.md. --- # Providers / Nebius Token Factory Source: https://anyinfer.dev/providers/nebius/ Nebius Token Factory Nebius uses the OpenAI chat-completions dialect but exposes a richer model listing. AnyInfer uses that listing to discover current context windows, quantization, feature support, and prices instead of shipping a catalog that will age. streaming structured output (model-dependent) tool calls (model-dependent) reasoning channel discovery (pricing + context) Setup import anyinfer as ai client = ai.Client( [ ai.ProviderSettings.of( "nebius", api_key="env://NEBIUS_API_KEY", ), ] ) result = client.generate( "Explain why the sky appears blue.", target="nebius:deepseek-ai/DeepSeek-V3", reasoning="medium", ) Model ids use the provider's namespaced catalog. A suffix such as -fast is part of the model id and selects a separately priced flavor; AnyInfer does not rewrite it. Discovery The adapter asks for the verbose model list, so client.models("nebius") reports each model's context window, pricing, quantization, and feature flags with discovered provenance. If an endpoint does not support the verbose query, the adapter falls back to the ordinary listing and returns model ids without inventing metadata. Reasoning Normalized reasoning levels are sent as reasoning_effort. Reasoning fragments are surfaced as ReasoningDelta events on the event stream and remain separate from Generation.text. The upstream API also accepts reasoning levels outside AnyInfer's normalized four-level scale; send those explicitly through provider_options when needed. Wire Contract For the exact fields this adapter sends and reads, see contracts/nebius.md. --- # Providers / Ollama Source: https://anyinfer.dev/providers/ollama/ Ollama Uses Ollama's native /api/chat API, not its /v1 OpenAI-compatibility layer. The native API carries grammar-enforced structured output, per-phase nanosecond timings, keep_alive session retention, and reasoning via think; the /v1 layer does not implement all of these and drops the parameters it lacks without reporting it. streaming structured output (grammar) tool calls health discovery embeddings Setup client = ai.Client([ai.ProviderSettings.of("ollama")]) result = client.generate(prompt, target="ollama:qwen3:8b") Defaults to http://127.0.0.1:11434. A bare hostname expands automatically, so base_url="myserver" becomes http://myserver:11434. Model names may contain colons: "ollama:qwen3:8b" is the provider ollama and the model qwen3:8b, because targets split on the first colon only. Supported Behavior Support Streaming Native NDJSON Structured output Grammar-enforced via format Tools Native Reasoning think, with effort levels Usage Input and output tokens Phase timings Model load, prefill, decode Sessions keep_alive Structured Output Ollama compiles the schema to a decoding grammar. Two consequences AnyInfer handles: Grammar-hostile keywords (minLength, maxLength, huge minItems/maxItems) are stripped for the wire only; the original schema still validates the response. The schema is also injected into the prompt. A grammar guarantees well-formed JSON, not meaningful JSON; a model never shown the schema emits schema-shaped nonsense. Phase Timings result.timing.phases # {"model_load_ms": 300.0, "prefill_ms": 200.0, # "decode_ms": 1000.0, "provider_total_ms": 1500.0} A large model_load_ms on a first request is the model being read from disk. GPU Spill Diagnostics The slowest failure Ollama has is not a failure. A model that no longer fits in VRAM alongside whatever else the GPU is holding is loaded anyway, with the overflow served from system memory; the request succeeds, the answer is correct, and it takes an order of magnitude longer than the same model took yesterday. The wire says nothing about it. /api/ps reports how much of each resident model is actually in VRAM, so the adapter reads it and says so: for note in client.diagnostics("ollama"): print(note.code, note.message) # ollama.gpu-spill qwen3:8b is only 45% resident in VRAM; the rest runs on the CPU, # which is far slower. Free GPU memory, or choose a smaller model # or quantization. The same text lands on result.warnings for any request that hit it, and as a ProviderDiagnostic event; see runtime diagnostics. Costs nothing: /api/ps is a local read, never a generation. A model within 5% of full residency is not reported; Ollama's own sizes wobble by a few megabytes, and a warning on every healthy load is one nobody reads. Embeddings Ollama's native POST /api/embed is a batch-capable embedding endpoint. (The older POST /api/embeddings route is deprecated and singular-input; this adapter does not speak it.) result = client.embed(["Why is the sky blue?", "Why is the grass green?"], target="ollama:nomic-embed-text") print(result.space.dimensions, len(result.vectors)) Batch input is native: every text in one call is sent as one array, not simulated with repeated requests. Requested dimensions are forwarded when the model supports native dimensionality reduction. Ollama documents no native rerank endpoint, so reranking is unsupported for this provider. Multimodal Inputs Vision-capable models receive inline images through the native message images field. Remote image URLs, documents, and audio are refused rather than silently dropped. Notes A missing model produces ModelNotFoundError hinting ollama pull . Usage arrives only on the terminal object; the core synthesizes a UsageUpdate event so streaming consumers see it exactly as they would from any other provider. Native extras pass through provider options: provider_options = {"ollama": {"keep_alive": "10m", "num_ctx": 8192, "num_gpu": 99}}. Wire Contract For the exact request/response fields this adapter depends on, see contracts/ollama.md. --- # Providers / LM Studio Source: https://anyinfer.dev/providers/lm-studio/ LM Studio LM Studio's local server. Generation uses its OpenAI-compatible endpoint, a shared and well-understood dialect; discovery uses the native API, which reports what a local engine's model list actually holds. streaming structured output tool calls discovery (context, quantization, residency) health Setup import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("lm-studio")]) result = client.generate(prompt, target="lm-studio:qwen3-8b") Defaults to http://127.0.0.1:1234/v1, LM Studio's conventional address. A bare hostname expands to that port, so a server on another machine needs only its name: ai.ProviderSettings.of("lm-studio", base_url="gpu-box") # http://gpu-box:1234 An API token is only needed when LM Studio's authentication has been enabled. lmstudio: is an accepted alias. Discovery The compatibility endpoint lists model ids. The native one lists what matters for a local engine: for model in client.models("lm-studio"): caps = model.capabilities print(model.id, caps.context_window.value, caps.local.quantization) # qwen3-8b 32768 Q4_K_M Context length, quantization, artifact size, tool-use and reasoning support, all with discovered provenance, because the server reported them rather than a table guessing. Embedding models are filtered out; they are not chat models. Older LM Studio builds have no native API. A 404 there degrades to the OpenAI listing (ids alone) rather than failing. Residency On a local engine the difference between a fast request and a thirty-second wait is whether the model is already loaded. Health says so: health = client.health("lm-studio") print(health.detail) # "loaded: qwen3-8b" or "no model loaded; the first request will load one" The detail names every resident model, so an application that prefers an already-loaded model over one that would first be read from disk can route on it. Reasoning LM Studio names reasoning levels: result = client.generate(prompt, target="lm-studio:qwen3-8b", reasoning="medium") minimal maps to the server's low rather than off; disabling reasoning changes the answer more than reducing it does. Pass provider_options={"lm-studio": {"reasoning": "off"}} to turn it off deliberately. Embeddings Embedding models loaded in LM Studio serve through the OpenAI-compatible dialect: result = client.embed(["hello"], target="lm-studio:text-embedding-nomic-embed-text-v1.5") client.models("lm-studio", operation="embedding") lists which loaded models embed; the native listing's type field distinguishes them, so nothing is guessed. LM Studio documents no request limits for the endpoint; for large corpora set BatchPolicy.max_items_override to a size the machine handles. Model Management This adapter reads inventory but does not manage it: loading, unloading, and downloading stay in LM Studio's own UI and CLI (lms load, lms unload). Requests load a model on demand as usual. For an engine AnyInfer supervises end to end (downloading artifacts, tuning for the local hardware, and managing the server process), see llama.cpp. For other engines, vLLM, SGLang, KoboldCpp, Jan, GPT4All, text-generation-webui, and TabbyAPI are all preconfigured presets, and any OpenAI-compatible server works through openai-compat. Wire Contract For the exact request/response fields this adapter depends on, see contracts/lm-studio.md. See Also Ollama: the other local engine with native discovery. The local subsystem: hardware detection and supervision. --- # Providers / llama.cpp Source: https://anyinfer.dev/providers/llama-cpp/ llama.cpp A supervised llama-server subprocess speaking the OpenAI-compatible dialect over loopback. In-process llama-cpp-python is not supported: one wire protocol for every engine, crash isolation, and no GPU-wheel build matrix in the dependency tree. streaming structured output (grammar) tool calls vision with projector health catalog discovery Setup client = ai.Client( [ ai.ProviderSettings.of( "llama-cpp", options={ "posture": "balanced", "idle_ttl_s": 900, }, ), ] ) result = client.generate(prompt, target="llama-cpp:qwen2.5-7b-instruct-q4-k-m") The model reference is a catalog artifact id, not a file path. That one call resolves the artifact, downloads and verifies it, tunes a server for the local hardware, starts it, and answers. Install a pinned runtime with anyinfer runtime install, or supply an existing llama-server through the binary option. With multiple installed runtimes, set runtime to cuda, vulkan, metal, rocm, or cpu; the default auto selects the highest-ranked installed backend the detected hardware can drive. Aliases: llamacpp, llama. Options Option Default Meaning catalog client's active catalog Optional direct-adapter override for artifact resolution. runtime auto Installed backend family to use; auto selects the best usable one. binary — Optional executable override; takes precedence over runtime. model_dir platform cache Where GGUF files are stored. posture balanced conservative, balanced, or aggressive. hardware detected A pre-detected profile, to skip re-probing. idle_ttl_s 900 Unload after this long with no active streams. max_resident 1 Concurrent servers before eviction. auto_download True Fetch a missing artifact rather than failing. allow_remote_exposure False Bind a non-loopback address. progress — Download progress callback. The typed form of this table, for programmatic construction: anyinfer.providers.llama_cpp.LlamaCppOptions dataclass LlamaCppOptions( catalog: Catalog | None = None, binary: str = "llama-server", runtime: _RuntimeChoice = "auto", model_dir: Path | None = None, posture: Posture = "balanced", hardware: HardwareProfile | None = None, idle_ttl_s: float | None = 900.0, max_resident: int = 1, auto_download: bool = True, allow_remote_exposure: bool = False, progress: ProgressCallback | None = None, ) Adapter configuration, supplied through ProviderSettings.options. Attributes: Name Type Description catalog Catalog | None Catalog resolving artifact ids to pinned downloads. binary str Optional path to llama-server. It overrides runtime. runtime _RuntimeChoice Installed backend family, or "auto" for the best usable one. model_dir Path | None Where artifacts are stored. posture Posture Tuning posture. hardware HardwareProfile | None Pre-detected hardware, to avoid re-probing. idle_ttl_s float | None Unload a server after this long with no active streams. max_resident int How many servers may run at once. auto_download bool Fetch a missing artifact rather than failing. allow_remote_exposure bool Bind a non-loopback address; loopback-only by default. progress ProgressCallback | None Download progress callback. from_mapping classmethod from_mapping(options: Mapping[str, Any]) -> LlamaCppOptions Build options from a provider settings mapping, ignoring unknown keys. Scalars are coerced, because the same options reach here from two directions: a Python caller passing real Path/float/bool values, and a config file or settings UI whose setup-spec values are strings by construction. Rejecting the latter would make every declared field unusable from the very config format the setup spec exists to drive; coercing an unreadable value would be worse, so a malformed one raises rather than silently reverting to the default. Supported Behavior Support Streaming Native SSE Structured output Grammar (GBNF), compiled from the schema Tools Native, via --jinja Usage Input and output tokens Cost Free (a genuine zero, not an unknown) Images OpenAI-compatible image content when the artifact pins a projector Embeddings --embeddings-started server, genuinely OpenAI-shaped /v1/embeddings Vision Models and Projector Companions A vision artifact is two verified files: the model GGUF and its multimodal projector. AnyInfer counts both for fit and download admission, fetches both through the normal model store, starts llama-server with --mmproj, and advertises VISION with catalog provenance. The bundled Qwen2.5-VL entry includes its pinned projector. An image request against an artifact without a projector fails before generation instead of starting a text-only server that would ignore the image. Documents and audio are not projected through the llama.cpp adapter. Structured Output llama.cpp compiles response_format.json_schema into a real GBNF grammar. As with Ollama, the schema is also injected into the prompt, because a grammar constrains form without conveying meaning. Tool Calling The tuner always emits --jinja. Without it, llama-server cannot apply a model's chat template and tool calling silently does not work at all. Embeddings --embeddings can only be set when llama-server starts. This is live-verified: an already-running chat server answers every /v1/embeddings call with a 501 asking for a restart with the flag, and there is no way to toggle it afterward. So embed() never reuses a chat server's resident process, even for the same GGUF: it starts (or reuses) a second one, keyed separately, specifically for embedding calls. result = client.embed( ["first text", "second text"], target="llama-cpp:nomic-embed-text-v1.5", ) Once started with --embeddings, the endpoint is genuinely OpenAI-shaped, and the same code path every hosted OpenAI-compatible provider uses handles it with no llama.cpp-specific parsing. Supervision Servers bind 127.0.0.1. A non-loopback bind requires allow_remote_exposure=True. VRAM admission is checked before spawning, so an oversized model is refused with a clear message instead of crashing the child process. Serialized model swaps, readiness blocking, and the idle timer are covered in the local subsystem. CPU Fallback VRAM admission control refuses a model that cannot fit, but it also does something quieter: on a machine where the weights plus KV cache leave no room, the plan offloads no layers and the model is served entirely on the CPU. That is the right call: a slow answer beats no answer, and it is invisible from the result. So the adapter reports it as a runtime diagnostic: for note in client.diagnostics("llama-cpp"): print(note.code, note.message) # llama-cpp.cpu-only qwen3-8b is being served with no layers offloaded, so it runs on # the CPU despite this machine having a cuda accelerator. ... Reported only when an accelerator was actually detected; on a CPU-only machine this is the plan working, not the plan degrading, and read from the supervisor's own state, so it costs nothing and never triggers hardware detection on its own. Wire Contract For the exact request/response fields this adapter depends on, see contracts/llama-cpp.md. --- # Providers / Text Embeddings Inference Source: https://anyinfer.dev/providers/tei/ Text Embeddings Inference Hugging Face's TEI server, spoken in its native dialect; the one local provider with a real reranking endpoint. TEI serves exactly one model per container (an embedding model or a reranker, chosen at startup), so this provider is also the library's first retrieval-only adapter: it declares no generation at all. embeddings reranking discovery (model + operation) generation Setup Run a server: docker run -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \ --model-id BAAI/bge-large-en-v1.5 Then point a client at it: import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("tei")]) # defaults to 127.0.0.1:8080 result = client.embed(["What is deep learning?"], target="tei:bge-large") Since the server holds one model, the model half of the target is advisory; client.models("tei") reports the real id and operation, discovered from GET /info. An embedder and a reranker are therefore two servers, configured as two instances: client = ai.Client([ ai.ProviderSettings(provider_id="tei", alias="tei-embed", base_url="http://127.0.0.1:8080"), ai.ProviderSettings(provider_id="tei", alias="tei-rerank", base_url="http://127.0.0.1:8081"), ]) ranked = client.rerank("the query", docs, target="tei-rerank:bge-reranker") Notes Vectors are unit-normalized by default: the server's documented normalize: true default is left in force and reported on result.space.normalized; a provider_options={"tei": {"normalize": False}} override is reported as sent. No usage: TEI's response body carries no token counts, so result.usage stays empty; the x-compute-tokens header real servers send is absent from the published API document, so it sits on the contract's watchlist rather than in accounting. top_n is applied client-side: the endpoint has no native parameter; the adapter sorts by score and truncates, and the contract snapshot records that. Batch ceiling is per-deployment: GET /info reports max_client_batch_size for the running server, so no static limit is declared. For corpora above it, pass batch=BatchPolicy(max_items_override=). An --api-key-protected server takes api_key="env://TEI_API_KEY". Wire Contract For the exact request/response fields this adapter depends on, see contracts/tei.md. It was verified against real servers on 2026-08-24 (text-embeddings-inference 1.8.3 serving BAAI/bge-small-en-v1.5 and BAAI/bge-reranker-base), and that traffic is committed as cassettes, so the lane replays in CI without a server. See Also Voyage AI and Jina AI: the hosted counterparts to this retrieval-only shape. Embeddings and reranking: the normalized operations and batching rules. --- # Providers / Voyage AI and Jina AI Source: https://anyinfer.dev/providers/retrieval/ Voyage AI and Jina AI The two hosted specialist retrieval providers: embeddings and reranking, no generation. Both serve native query/document input intents and a real reranker; they differ in task vocabulary, batch ceilings, and truncation behavior. Setup VoyageJina import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("voyage", api_key="env://VOYAGE_API_KEY")]) docs = client.embed(corpus, target="voyage:voyage-3.5", input_type="document") query = client.embed([question], target="voyage:voyage-3.5", input_type="query") ranked = client.rerank(question, candidates, target="voyage:rerank-2.5", top_n=5) import anyinfer as ai client = ai.Client([ai.ProviderSettings.of("jina", api_key="env://JINA_API_KEY")]) docs = client.embed(corpus, target="jina:jina-embeddings-v3", input_type="document") query = client.embed([question], target="jina:jina-embeddings-v3", input_type="query") ranked = client.rerank(question, candidates, target="jina:jina-reranker-v3", top_n=5) Provider Differences Voyage AI Jina AI Input intents query and document only; classification/clustering have no wire value, are never sent, and the result carries a warning All four: query → retrieval.query, document → retrieval.passage, classification verbatim, clustering → separation (Jina's clustering-flavored task, recorded in the contract) Embedding batches Batching engages at 1,000 inputs No documented ceiling: Jina batches internally, so no limit is invented; requests above the library's sanity ceilings refuse locally, and BatchPolicy.max_items_override applies an independently verified limit Rerank documents 1,000 is a hard cap; the core refuses larger requests unless rerank_cross_batch opts into chunk-local rankings top_n is taken natively by the reranker Truncation Defaults on server-side: over-length inputs are cut, not rejected. Disable per call with provider_options={"voyage": {"truncation": False}} — Dimensionality — Matryoshka truncation via dimensions=; late_chunking via provider_options={"jina": {"late_chunking": True}} Notes Two behaviors are shared: No model listing. Neither API has a listing endpoint, so client.models("voyage") and client.models("jina") return empty lists; the verified model set ships in each descriptor's capabilities. Usage is total_tokens only: input_tokens stays unknown rather than assumed. Wire Contract For the exact request/response fields each adapter depends on, see contracts/voyage.md and contracts/jina.md. See Also Text Embeddings Inference: the local counterpart to this retrieval-only shape. Embeddings and reranking: intents, spaces, and batching. --- # Providers / OpenAI-Compatible Source: https://anyinfer.dev/providers/openai-compat/ OpenAI-Compatible The base dialect for any endpoint speaking POST /chat/completions: vLLM, LM Studio, an externally-run llama-server, a corporate gateway, or OpenAI itself. streaming structured output (server-dependent) tool calls health discovery Setup client = ai.Client( [ ai.ProviderSettings.of( "openai-compat", base_url="http://localhost:8000/v1", api_key="env://MY_API_KEY", # optional for keyless local servers ), ] ) result = client.generate(prompt, target="openai-compat:my-model") base_url is required; there is no sensible default for "any server". Aliases: openai-compatible, oai-compat. Supported Behavior Support Streaming SSE Structured output json_schema or json_object, where the server implements it Tools Native Usage When the server reports it Capabilities Are Unknown by Default AnyInfer cannot know what an arbitrary server supports, so features default to a conservative set and structured output falls back to prompt injection unless told otherwise. Client-side validation means the caller still gets a validated result either way. In order to declare what a server actually does, register a descriptor with richer default_capabilities; see writing an adapter. Servers That Ignore stream Some endpoints accept stream: true and answer with a buffered body anyway. The adapter detects that and consumes the body rather than paying for a second request, so consumer code is unaffected. Known Divergences max_tokens vs max_completion_tokens: subclasses override this; the base sends max_tokens. stream_options.include_usage is not universally implemented. Usage is simply absent when a server omits it, never fabricated. response_format support varies widely between implementations, which is one of the reasons validation is always client-side. Wire Contract For the exact request/response fields this adapter depends on, see contracts/openai-compat.md. --- # Reference / Reference Source: https://anyinfer.dev/reference/ Reference Look things up. SDK reference: the generated API reference: every public class, function, and event, from the docstrings. Error catalog: every exception, when it is raised, whether it retries, and the hint the user will see. Conformance matrix: what each provider actually supports, from test results rather than assertion. Shared configuration: provider settings, environment variables, and the common JSON file. Run manifests: the manifest format, serialization compatibility, and its executable JSON Schema. Glossary: the vocabulary this project uses precisely. API Stability The public API is fully typed (the package ships a py.typed marker) and documented in its docstrings. The stability commitments cover the top-level anyinfer namespace plus the subpackage surfaces the guides teach: anyinfer.config, anyinfer.local, anyinfer.serve, anyinfer.testing, and anyinfer.otel. Anything under anyinfer._client is private, and within every module only the names in __all__ are public; everything else is an implementation detail that may change without notice. help(ai.AsyncClient.generate) in a REPL always matches the published SDK reference, because both come from the same docstrings. Module Map Module Responsibility anyinfer.types Frozen domain types. Zero I/O. anyinfer.errors The exception hierarchy. anyinfer._client AsyncClient, the sync Client, and the tool loop. anyinfer.registry Provider descriptors and collision-safe registration. anyinfer.config Shared, versioned JSON configuration. anyinfer.routing Routes, retries, health gating, attempt accounting. anyinfer.schema Mechanism selection, projection, validation, repair. anyinfer.events Telemetry events and observer dispatch. anyinfer.redaction The secret-redaction registry. anyinfer.credentials Credential resolvers. anyinfer.catalog Alias catalog and target resolution. anyinfer.capabilities Capability assembly, cost computation, token estimation, context budgets, and the pre-dispatch gate. anyinfer.local Hardware, backends, tuning, downloads, supervision. anyinfer.providers One module per adapter. anyinfer.testing Fakes, cassettes, and the conformance suite. anyinfer.serve The OpenAI-compatible frontend. anyinfer.otel The OpenTelemetry bridge. --- # Reference / SDK Reference Source: https://anyinfer.dev/reference/api/ SDK Reference Generated from the docstrings of every public symbol in anyinfer; coverage is a CI gate, so nothing here is an empty page. The core surface is importable from the top-level package; the local, serve, and testing subsystems from their subpackages: import anyinfer as ai Page Covers Clients and streams Client, AsyncClient, streams, ProviderSettings, the @tool decorator Requests and messages GenerationRequest, messages, sampling, schemas, tool specs Results and stream events Generation, usage, timing, the typed event stream Routing Route, Retry, target resolution Embeddings and reranking The embed()/rerank() request and result types, embedding spaces, batch policy Capabilities Provenance-tagged model capabilities and pricing Context reduction Fitting a document corpus to a token budget Telemetry and redaction Observers, telemetry events, redaction, the OpenTelemetry bridge Registry, catalog, credentials Provider descriptors, the model catalog, credential resolvers Configuration The shared JSON loader and validated config object Local inference anyinfer.local: hardware, backends, tuning, downloads, supervision Serve anyinfer.serve: the embeddable frontend and its OpenAI codec Testing utilities anyinfer.testing: fakes, cassettes, the conformance suite Portability diff anyinfer.compare_diff: snapshot compare() output and diff two snapshots Vector store add-on anyinfer_store: the small-scale embedded vector store, a separate distribution Errors The exception hierarchy and its structured fields For how to use these rather than their signatures, start with the concepts and guides. --- # Reference / Clients and Streams Source: https://anyinfer.dev/reference/api/client/ Clients and Streams The two entry points, Client (sync) and AsyncClient (async), expose the same surface; the sync client is a facade over the async core (see the architecture overview). anyinfer.Client Client( providers: Sequence[ProviderSettings] | None = None, *, registry: ProviderRegistry | None = None, catalog: Catalog | None = None, route: Route | None = None, operation_routes: Mapping[str, Route] | None = None, observers: Sequence[Observer] | None = None, resolver: ResolverChain | None = None, retain_raw: bool = False, repair: Repair | None = None, use_default_catalog: bool = True, estimator: TokenEstimator | None = None, context_gate: bool = True, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, arenas: Mapping[str, ArenaPolicy] | None = None, spend: SpendPolicy | None = None, ledger: SpendLedger | None = None, pricing_table: PricingTable | None = None, manifests: bool = True, manifest_payloads: bool = False, capability_overrides: Mapping[str, ModelCapabilities] | None = None, model_dir: Path | None = None, ) The synchronous inference client. A thin facade: every method schedules work on one background event loop that owns the real AsyncClient. Safe to call from multiple threads, and concurrent requests still overlap on the loop. Args are identical to AsyncClient. model_store property model_store: ModelStore The store acquired model weights live in. __enter__ __enter__() -> Client Enter a context that closes the client on exit. __exit__ __exit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the client on context exit. close close() -> None Close adapters, stop the background loop, and join its thread. subscribe subscribe( observer: Observer, *, payloads: bool = False ) -> None Register a telemetry observer. unsubscribe unsubscribe(observer: Observer) -> None Remove a telemetry observer. models models( provider_id: str, *, operation: InferenceOperation | None = None, ) -> Sequence[DiscoveredModel] List a provider's models. See AsyncClient.models. health health(provider_id: str) -> Health Probe a provider's readiness. diagnostics diagnostics(provider_id: str) -> Sequence[Diagnostic] Ask a provider what it has noticed about its own runtime. See AsyncClient.diagnostics. resolve resolve(target: Target) -> ResolvedTarget Resolve a target string without issuing a request. pull_model pull_model( provider_id: str, model: str, *, progress: Any | None = None, timeout_s: float = PULL_TIMEOUT_S, ) -> PullReport Tell an engine that keeps its own store to make a model available. See AsyncClient.pull_model. session session(target: Target) -> Session Open a handle that lets a provider keep what it already knows. See AsyncClient.session. benchmark benchmark( target: Target, *, prompt_tokens: int = BENCHMARK_PROMPT_TOKENS, output_tokens: int = BENCHMARK_OUTPUT_TOKENS, timeout_s: float = 120.0, store: MeasurementStore | None = None, progress: Callable[[BenchmarkSample], None] | None = None, ) -> Measurement Measure what a target actually does, with one deterministic request. See AsyncClient.benchmark. probe probe( target: Target, *, features: Sequence[Feature] | None = None, timeout_s: float = 30.0, record: bool = True, ) -> ProbeReport Measure what a target actually supports, one request per feature. See AsyncClient.probe. verify verify( target: Target, *, timeout_s: float = 60.0, operation: InferenceOperation = "generation", ) -> Verification Prove a target works by asking it something, end to end. See AsyncClient.verify. probe_embedding probe_embedding( target: Target, *, timeout_s: float = 30.0, record: bool = True, ) -> EmbeddingProbeReport Measure an embedding target with one real call. See AsyncClient.probe_embedding. local_catalog local_catalog( provider_id: str | None = None, *, hardware: HardwareProfile | None = None, best_at: str | None = None, posture: Posture = "balanced", ) -> CatalogView Browse the local model catalog, annotated with how each entry fits. See AsyncClient.local_catalog. acquire_model acquire_model( model_id: str, *, engine: str | None = None, variant_id: str | None = None, hardware: HardwareProfile | None = None, progress: ProgressSink | None = None, prefs: VariantPrefs | None = None, dry_run: bool = False, token: str | None = None, ) -> AcquisitionReport Download a catalog model's weights. See AsyncClient.acquire_model. The progress sink is invoked from the background loop thread, so it must not block and must not call back into this client. installed_models installed_models() -> Sequence[StoreEntry] Every model acquired into this client's store. locate_model locate_model( model_id: str, *, variant_id: str | None = None, engine: str | None = None, verify: bool = False, ) -> ResolvedModel | None Find an acquired model on disk. See AsyncClient.locate_model. remove_model remove_model(entry_id: str) -> RemovalReport Delete an acquired model. See AsyncClient.remove_model. spend spend() -> SpendTotals What this client has spent so far. See AsyncClient.spend. budget budget( messages: MessagesInput, *, target: Target, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), sampling: Sampling | None = None, output_reserve_tokens: int | None = None, ) -> ContextBudget Compute the context budget for a request without sending it. Pure computation — runs directly on the calling thread. See AsyncClient.budget(). compare compare( messages: MessagesInput | GenerationRequest, *, targets: Sequence[Target], schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, refresh: bool = False, ) -> tuple[TargetComparison, ...] Compare request portability without generating. See AsyncClient.compare. compare_embedding compare_embedding( inputs: str | Sequence[str], *, targets: Sequence[Target], input_type: EmbeddingInputIntent | None = None, refresh: bool = False, ) -> tuple[EmbeddingTargetComparison, ...] Compare embedding request portability without dispatching. See AsyncClient.compare_embedding. generate generate( messages: MessagesInput, *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, max_input_part_bytes: int | None = None, max_input_bytes: int | None = None, session: Session | None = None, manifest: bool | None = None, ) -> Generation Generate a single result. See AsyncClient.generate(). embed embed( inputs: str | Sequence[str], *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, input_type: Literal[ "query", "document", "classification", "clustering" ] | None = None, dimensions: int | None = None, expected_space: EmbeddingSpace | None = None, allow_incompatible_fallback: bool = False, batch: BatchPolicy | None = None, timeout_s: float | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, retain_raw: bool | None = None, manifest: bool | None = None, ) -> EmbeddingResult Embed one or more texts into vectors. See AsyncClient.embed(). rerank rerank( query: str, documents: Sequence[str | RerankDocument], *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, top_n: int | None = None, batch: BatchPolicy | None = None, timeout_s: float | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, return_documents: bool = False, retain_raw: bool | None = None, manifest: bool | None = None, ) -> RerankResult Rank documents by relevance to a query. See AsyncClient.rerank(). run_tools run_tools( messages: MessagesInput, *, tools: Sequence[Any], target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, max_rounds: int = DEFAULT_MAX_ROUNDS, **kwargs: Any, ) -> Generation Generate, dispatching tools until the model answers. See AsyncClient.run_tools(). stream stream( messages: MessagesInput, *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, max_input_part_bytes: int | None = None, max_input_bytes: int | None = None, session: Session | None = None, manifest: bool | None = None, ) -> SyncStream Start a streaming generation, returning a blocking iterator. See AsyncClient.stream(). Use the result as a context manager so that leaving the block early cancels the in-flight request. anyinfer.AsyncClient AsyncClient( providers: Sequence[ProviderSettings] | None = None, *, registry: ProviderRegistry | None = None, catalog: Catalog | None = None, route: Route | None = None, operation_routes: Mapping[str, Route] | None = None, observers: Sequence[Observer] | None = None, resolver: ResolverChain | None = None, retain_raw: bool = False, repair: Repair | None = None, use_default_catalog: bool = True, estimator: TokenEstimator | None = None, context_gate: bool = True, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, arenas: Mapping[str, ArenaPolicy] | None = None, spend: SpendPolicy | None = None, ledger: SpendLedger | None = None, pricing_table: PricingTable | None = None, manifests: bool = True, manifest_payloads: bool = False, capability_overrides: Mapping[str, ModelCapabilities] | None = None, model_dir: Path | None = None, ) The asynchronous inference client. Parameters: Name Type Description Default providers Sequence[ProviderSettings] | None Per-provider settings. The order given is the preference order used when resolving catalog aliases. None registry ProviderRegistry | None Provider registry; defaults to the process-wide one. None catalog Catalog | None Alias catalog; defaults to the bundled catalog. Pass None explicitly via use_default_catalog=False to disable alias resolution. None route Route | None Default route applied when a call names no target. None observers Sequence[Observer] | None Telemetry observers, registered payload-free. None resolver ResolverChain | None Credential resolver chain. None retain_raw bool Keep the provider's raw payload on results. Off by default because raw payloads carry response text that payload-free telemetry deliberately omits. False repair Repair | None Default repair budget for schema violations. None use_default_catalog bool Load the bundled catalog when catalog is not supplied. True estimator TokenEstimator | None Token counting strategy for budgets and the pre-dispatch gate. Defaults to the dependency-free byte heuristic. None context_gate bool Fail a target before dispatch when the request provably cannot fit its known context window. Only trusted-provenance windows gate, and only on the estimate's floor, so a heuristic never refuses a request that might have fit. True history HistoryPolicy | None Conversation-compaction policy applied when a request outgrows its target's window. None — the default; never compacts. This is the client's half of the overflow answer; Route.context_window_targets is the other half, and the policy's mode decides which is tried first. None spend SpendPolicy | None Ceiling on what this client may spend, checked before dispatch. None — the default; never refuses anything. Not an organization quota: it governs this client object in this process and nothing else. None ledger SpendLedger | None Spend rollup to record into. One is created automatically when spend is set; supply your own to share a total between clients or to read it without a policy in force. None cache CachePolicy | None Prompt-cache placement applied to every request that does not carry its own. None — the default; never engages a provider's cache, because caching changes what a provider bills and how long it keeps a copy of the prompt. A request's own cache overrides this. Every frontend built on this client inherits it. None pricing_table PricingTable | None Model pricing supplying the catalog layer of capability assembly. Defaults to the table bundled with this release; pass the result of fetch_pricing() for newer numbers. None manifests bool Assemble a RunManifest for every call, reachable as Generation.manifest. On by default: it allocates one small object per in-flight request, writes nothing, sends nothing, and is content-free — the invited/uninvited line in this library has always been about spend and side effects, and a manifest has neither. Switch it off to skip the allocation entirely. True manifest_payloads bool Capture prompt, response, schema, and tool-call text into the manifest's payloads facet, redacted. Off by default and independent of observer payload opt-in, so a manifest cannot start carrying prompt text because some unrelated telemetry sink asked for it. False capability_overrides Mapping[str, ModelCapabilities] | None Deliberate corrections keyed by "provider:model". Every supplied field is applied at override provenance — the strongest layer, outranking discovery and probes, so a wrong upstream number can always be fixed locally. None model_dir Path | None Where acquired model weights are stored. Defaults to the per-OS data directory, overridable with ANYINFER_MODEL_DIR. None catalog property catalog: Catalog | None The alias catalog in force, if any. model_store property model_store: ModelStore The store acquired model weights live in. __aenter__ async __aenter__() -> AsyncClient Enter an async context managing this client's adapters. __aexit__ async __aexit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the client on context exit. aclose async aclose() -> None Close every adapter this client built. subscribe subscribe( observer: Observer, *, payloads: bool = False ) -> None Register a telemetry observer. Parameters: Name Type Description Default observer Observer The sink. required payloads bool Opt in to prompt and response text. Off by default, so no observer sees payload text it did not explicitly ask for. False unsubscribe unsubscribe(observer: Observer) -> None Remove a telemetry observer. models async models( provider_id: str, *, operation: InferenceOperation | None = None, ) -> Sequence[DiscoveredModel] List a provider's models, recording what they report about capabilities. Parameters: Name Type Description Default provider_id str The configured provider to list. required operation InferenceOperation | None Keep only models known to serve this operation — via a discovered operation tag, or the descriptor's static embedding/rerank capability tables. A model whose operations are unknown is included only for "generation" on a generation-capable provider (the pre-filter behaviour); for embedding and rerank, unknown support is never guessed into the listing. None lists everything. None operations_for operations_for( target: Target, ) -> frozenset[InferenceOperation] Which inference operations the resolved target is known to serve. Model-level facts win: a discovered operation tag, else membership in the descriptor's static embedding/rerank capability tables. A model with no model-level facts on a generation-capable provider reports {"generation"} — the assumption every listing made before operations existed — and never has embedding or rerank support guessed in. Parameters: Name Type Description Default target Target A target string or catalog alias; resolved without dispatching. required Raises: Type Description ConfigError If the target cannot be resolved at all. health async health(provider_id: str) -> Health Probe a provider's readiness. diagnostics async diagnostics(provider_id: str) -> Sequence[Diagnostic] Ask a provider what it has noticed about its own runtime. Answers the question a health probe cannot: not "can I reach it" but "is it in good shape" — a model that spilled out of VRAM, a runtime that fell back to the CPU, a supervised server nearing its memory ceiling. Requests to such a provider succeed; they are simply much slower than the caller expects, with nothing in the result to explain why. Providers that declare reports_diagnostics answer; the rest return nothing, as does one that fails to answer — this is advisory data and never raises. Parameters: Name Type Description Default provider_id str The configured provider to ask. required Returns: Type Description Sequence[Diagnostic] What the provider reported, most likely empty. resolve resolve(target: Target) -> ResolvedTarget Resolve a target string without issuing a request. session session(target: Target) -> Session Open a handle that lets a provider keep what it already knows. Every request is independent by default, which is right for one-shot work and wrong for a conversation. Providers that can carry state between turns each save something different — Copilot keeps the conversation server-side, llama.cpp keeps the model and its KV cache resident, Ollama keeps the model loaded, and a session is how a caller says "these requests belong together" without having to know which. A session never changes an answer; it is a performance and cost optimization. Opening one against a provider that cannot keep state is therefore allowed and merely inert: every request behaves exactly as it would have, and Session.supported and Session.reuse say so. with client.session("copilot:auto") as chat: await client.generate("Summarize this report.", session=chat) await client.generate("Now list the risks.", session=chat) Parameters: Name Type Description Default target Target The target this session's state belongs to. State is not portable, so a turn routed anywhere else runs without it. required Returns: Type Description Session The Session handle. Raises: Type Description ConfigError If the target cannot be resolved. verify async verify( target: Target, *, timeout_s: float = 60.0, operation: InferenceOperation = "generation", ) -> Verification Prove a target works by asking it something, end to end. health() answers "can I reach this endpoint", which is not the question behind a Test connection button: a credential can be valid for a model listing and not for inference, a model id can be a typo, a deployment can exist with no capacity, and a provider can answer fluently while never holding a schema. Only a real request distinguishes those, so this spends one — deliberately tiny, capped at VERIFY_MAX_OUTPUT_TOKENS output tokens, or VERIFY_REASONING_OUTPUT_TOKENS when the target is known to be a reasoning model and would otherwise spend the whole budget thinking before it said anything. Never raises for a provider problem: "this target is broken" is the answer to the question, not a failure to answer it. A malformed target, on the other hand, is the caller's mistake and still raises. Parameters: Name Type Description Default target Target The target to verify. A catalog alias resolves as usual. required timeout_s float Wall clock for the probe. 60.0 operation InferenceOperation Which operation to prove. "embedding" embeds one tiny probe text and judges the vector; "rerank" ranks two probe documents. Both spend one deliberately small real request, exactly like the generation probe. 'generation' Returns: Type Description Verification The Verification, whose reached Verification and ok distinguish "unreachable" from "reachable but could not hold the Verification shape". Raises: Type Description ConfigError If the target cannot be resolved at all. probe_embedding async probe_embedding( target: Target, *, timeout_s: float = 30.0, record: bool = True, ) -> EmbeddingProbeReport Measure an embedding target with one tiny real call. Embedding capability tables only carry what a provider documents; a self-hosted or preset endpoint often documents nothing. This spends one deliberately small request and measures what came back — the vector length, and whether it was unit-normalized — recording both at probed provenance so later calls (and capabilities_for consumers) see measured facts instead of blanks. This costs money and time, exactly like probe(): opt-in, one round trip. Parameters: Name Type Description Default target Target The embedding target to measure. required timeout_s float Wall clock for the probe call. 30.0 record bool Store the findings at probed provenance. True Returns: Type Description EmbeddingProbeReport The EmbeddingProbeReport with the measured facts. Raises: Type Description ConfigError If the target cannot be resolved, or its provider does not declare the embedding operation. AllTargetsFailedError If the probe call itself failed. probe async probe( target: Target, *, features: Sequence[Feature] | None = None, timeout_s: float = 30.0, record: bool = True, ) -> ProbeReport Measure what a target actually supports, one tiny request per feature. The capability layer's third tier, and the only one that is a measurement. The catalog says what a model should support and discovery says what a provider claims; for the compatibility surface — every preset endpoint, every self-hosted OpenAI-compatible server — both are educated guesses, and a server that accepts response_format while ignoring it is indistinguishable from one that honors it until a schema silently stops being enforced. This costs money and time: one round trip per feature, four by default. It is opt-in for that reason, and normally run once when an application first configures an endpoint rather than on every start. A probe that settles nothing records nothing. A provider that accepts the request and answers something unexpected is inconclusive, because a weak model and an ignored parameter look identical in one reply. Parameters: Name Type Description Default target Target The target to measure. required features Sequence[Feature] | None Which features to test; defaults to DEFAULT_PROBE_FEATURES. None timeout_s float Wall clock for each individual probe. 30.0 record bool Store the findings at probed provenance, so later requests choose mechanisms from measurement rather than assumption. Pass False to look without committing. True Returns: Type Description ProbeReport The ProbeReport: per-feature outcomes, ProbeReport what was recorded, and what it cost. Raises: Type Description ConfigError If the target cannot be resolved, or a feature was named that no probe can settle. benchmark async benchmark( target: Target, *, prompt_tokens: int = BENCHMARK_PROMPT_TOKENS, output_tokens: int = BENCHMARK_OUTPUT_TOKENS, timeout_s: float = 120.0, store: MeasurementStore | None = None, progress: Callable[[BenchmarkSample], None] | None = None, ) -> Measurement Measure what a target actually does, with one deterministic request. Capabilities describe a model; none of them says how fast it is here. For local inference that is the number that decides everything — the same weights on the same GPU differ by an order of magnitude depending on what else is resident and how many layers ended up offloaded, and it is the number an application needs to pick a default model or explain a slow session. Prefill and decode are reported separately, because a machine can be fast at one and slow at the other, and prefill throughput is reported only when the provider timed its own prefill phase. Deriving it from time-to-first-token would fold queueing and network latency into a figure labelled compute. This costs one real request of roughly prompt_tokens in and output_tokens out. Nothing is written anywhere unless store is passed. Parameters: Name Type Description Default target Target The target to measure. required prompt_tokens int Approximate prompt size. Large enough that prefill is a real phase rather than rounding error. BENCHMARK_PROMPT_TOKENS output_tokens int Output cap. Decode throughput needs enough tokens to average over. BENCHMARK_OUTPUT_TOKENS timeout_s float Wall clock for the request. 120.0 store MeasurementStore | None An application-owned store to record the result in. Omitted, the measurement is returned and forgotten. None progress Callable[[BenchmarkSample], None] | None Optional sink for live token-rate and local host-utilization samples. Token counts are estimated until the terminal provider usage arrives. None Returns: Type Description Measurement The Measurement, whose rates are Measurement None where nothing could be measured rather than zero. Raises: Type Description ConfigError If the target cannot be resolved. AllTargetsFailedError If the request itself failed — an unmeasurable target is a failure, unlike an unverifiable one. local_catalog async local_catalog( provider_id: str | None = None, *, hardware: HardwareProfile | None = None, best_at: str | None = None, posture: Posture = "balanced", ) -> CatalogView Browse the local model catalog, annotated with how each entry fits. Performs no network I/O: the catalog is bundled data and hardware detection is local and cached. Parameters: Name Type Description Default provider_id str | None Restrict to models one configured engine can serve. Its locality also decides whether this machine is the right one to probe. None hardware HardwareProfile | None Specs to judge against, overriding detection. This is how an application answers for a remote host after asking the user. None best_at str | None One category from the catalog's closed vocabulary. None posture Posture How much of the machine to budget. 'balanced' Returns: Type Description CatalogView The view. hardware_source == "unavailable" means the engine runs somewhere CatalogView AnyInfer cannot probe, and the application should collect the host's specs and CatalogView call again. Raises: Type Description ConfigError If best_at names a category no catalog entry uses. acquire_model async acquire_model( model_id: str, *, engine: str | None = None, variant_id: str | None = None, hardware: HardwareProfile | None = None, progress: ProgressSink | None = None, prefs: VariantPrefs | None = None, dry_run: bool = False, token: str | None = None, ) -> AcquisitionReport Download a catalog model's weights into this client's model store. The quantization is chosen, not assumed: the highest curated rung whose weights and KV cache fit this machine's budget. Pass variant_id to override that. dry_run=True resolves everything and reports the exact byte count without writing anything — what an application needs to confirm a large download with a user before starting it. Parameters: Name Type Description Default model_id str A catalog model id. required engine str | None "llama.cpp" or "vllm"; defaults to whatever the model offers. None variant_id str | None Acquire this exact variant, skipping selection. None hardware HardwareProfile | None Specs to select against, overriding detection. None progress ProgressSink | None Aggregate progress sink. See AcquisitionProgress for the threading contract it must honor. None prefs VariantPrefs | None Selection preferences, including the low-quality opt-in. None dry_run bool Plan and report without writing. False token str | None A credential for the source, when it needs one. Defaults to HF_TOKEN. None Returns: Type Description AcquisitionReport The report, naming the registered entry and what it cost. Raises: Type Description ConfigError If there is no catalog, the model is unknown, or nothing fits. LocalRuntimeError On a transfer failure, digest mismatch, or full disk. pull_model async pull_model( provider_id: str, model: str, *, progress: Callable[[TelemetryEvent], None] | None = None, timeout_s: float = PULL_TIMEOUT_S, ) -> PullReport Tell an engine that keeps its own store to make a model available. Distinct from acquire_model(), which fetches weights this library places and indexes. Some local engines — Ollama — already have a store, a registry, and a downloader; for those the useful operation is not "download these bytes" but "make yourself ready", and the bytes land in the engine's store under the engine's own name. Nothing is written to this client's model store and locate_model() will not find it, because it is not ours to find. Progress arrives as DownloadProgress events on the client's observers, and additionally on progress when one is given. Parameters: Name Type Description Default provider_id str The configured provider to pull on. required model str The model name in that engine's namespace, e.g. "qwen3:8b". required progress Callable[[TelemetryEvent], None] | None An extra sink for progress events, for a caller that wants them without registering an observer. None timeout_s float Wall clock for the whole transfer. Generous by default: a timeout that fires mid-download turns a slow link into a failure the user cannot act on. PULL_TIMEOUT_S Returns: Type Description PullReport The PullReport, which distinguishes a PullReport transfer from a model that was already present. Raises: Type Description ConfigError If the provider is unknown, or cannot pull. ModelNotFoundError If the engine's registry has no such model. LocalRuntimeError If the engine is unreachable or the pull fails. installed_models async installed_models() -> Sequence[StoreEntry] Every model acquired into this client's store. locate_model async locate_model( model_id: str, *, variant_id: str | None = None, engine: str | None = None, verify: bool = False, ) -> ResolvedModel | None Find an acquired model on disk, with advisory launch arguments. No network I/O. Verification is shallow by default — size and modification time against the index, because re-hashing forty gigabytes on every lookup would be absurd; verify=True forces the full check. remove_model async remove_model(entry_id: str) -> RemovalReport Delete an acquired model's files and unregister it. A model adopted from somebody else's cache is only unregistered; its files belong to whatever put them there. generate async generate( messages: MessagesInput, *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, max_input_part_bytes: int | None = None, max_input_bytes: int | None = None, session: Session | None = None, manifest: bool | None = None, ) -> Generation Generate a single result, draining the event stream internally. manifest overrides the client's manifest setting for this one call; None inherits it. Returns: Type Description Generation The assembled Generation. Raises: Type Description AllTargetsFailedError Every target failed. SchemaViolationError The response never satisfied the schema. embed async embed( inputs: str | Sequence[str], *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, input_type: Literal[ "query", "document", "classification", "clustering" ] | None = None, dimensions: int | None = None, expected_space: EmbeddingSpace | None = None, allow_incompatible_fallback: bool = False, batch: BatchPolicy | None = None, timeout_s: float | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, retain_raw: bool | None = None, manifest: bool | None = None, ) -> EmbeddingResult Embed one or more texts into vectors. Parameters: Name Type Description Default inputs str | Sequence[str] A single text, or an ordered sequence of texts to embed. Duplicates are preserved exactly. required target Target | None A single target, as for generate(). None route Route | Target | Sequence[Target] | None A fallback chain, as for generate(). Embedding fallback is safe-by-default: a fallback target is dispatched only when it is the identical provider:model as the route's primary target; anything else is refused before any request is sent, unless allow_incompatible_fallback is set. None input_type Literal['query', 'document', 'classification', 'clustering'] | None What the embedded text will be used for, when the target model distinguishes it. None dimensions int | None Requested output dimensionality, for models supporting native dimensionality reduction. None expected_space EmbeddingSpace | None An anyinfer.EmbeddingSpace the result must match; a successful-but-incompatible response is rejected rather than returned. None allow_incompatible_fallback bool Explicit opt-in permitting fallback to a target that cannot be proven to share the primary target's embedding space. Off by default because wrong-space vectors fail silently when compared; a result served this way always carries a warning naming both targets. False batch BatchPolicy | None Core-owned batching policy. A request larger than the target's verified batch limit is split into ordered chunks and re-assembled in input order; an unknown limit is never guessed — see anyinfer.BatchPolicy. None timeout_s float | None Per-attempt wall-clock budget. None provider_options Mapping[str, Mapping[str, Any]] | None Escape hatch, namespaced by provider id. None metadata Mapping[str, str] | None Caller-supplied labels carried through telemetry. None max_response_bytes int | None Hard cap on one provider response body. None retain_raw bool | None Keep the provider's raw response payload on the result. Defaults to the client's retain_raw setting. None manifest bool | None Overrides the client's manifest setting for this one call; None inherits it. None Returns: Type Description EmbeddingResult The assembled EmbeddingResult. Raises: Type Description AllTargetsFailedError Every target failed. ConfigError The resolved target does not support embedding, its response fails the embedding-space safety check, or an incompatible fallback was refused before dispatch. rerank async rerank( query: str, documents: Sequence[str | RerankDocument], *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, top_n: int | None = None, batch: BatchPolicy | None = None, timeout_s: float | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, return_documents: bool = False, retain_raw: bool | None = None, manifest: bool | None = None, ) -> RerankResult Rank documents by relevance to a query. Parameters: Name Type Description Default query str The query text every document is scored against. required documents Sequence[str | RerankDocument] Document texts, or anyinfer.RerankDocument values carrying caller-owned ids. Plain strings are assigned ids "0", "1", ... in order. required target Target | None A single target, as for generate(). None route Route | Target | Sequence[Target] | None A fallback chain, as for generate(). None top_n int | None Return only the top N ranked items. None batch BatchPolicy | None Core-owned batching policy. A rerank request larger than the target's verified document limit is refused rather than split, unless BatchPolicy.rerank_cross_batch explicitly accepts chunk-local rankings — scores from separate calls are not globally comparable. None timeout_s float | None Per-attempt wall-clock budget. None provider_options Mapping[str, Mapping[str, Any]] | None Escape hatch, namespaced by provider id. None metadata Mapping[str, str] | None Caller-supplied labels carried through telemetry. None max_response_bytes int | None Hard cap on one provider response body. None return_documents bool Echo document text back on each ranked item. False retain_raw bool | None Keep the provider's raw response payload on the result. Defaults to the client's retain_raw setting. None manifest bool | None Overrides the client's manifest setting for this one call; None inherits it. None Returns: Type Description RerankResult The assembled RerankResult. Raises: Type Description AllTargetsFailedError Every target failed. ConfigError The resolved target does not support reranking, or its response names a document index outside the request. stream stream( messages: MessagesInput, *, target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, max_input_part_bytes: int | None = None, max_input_bytes: int | None = None, session: Session | None = None, manifest: bool | None = None, ) -> AsyncStream Start a streaming generation. manifest overrides the client's manifest setting for this one call; None inherits it. Returns: Type Description AsyncStream An AsyncStream: an async iterator of AsyncStream StreamEvent, usable as an async context manager, AsyncStream exposing the final result as AsyncStream.result. run_tools async run_tools( messages: MessagesInput, *, tools: Sequence[Tool | Any], target: Target | None = None, route: Route | Target | Sequence[Target] | None = None, max_rounds: int = DEFAULT_MAX_ROUNDS, **kwargs: Any, ) -> Generation Generate, dispatching any tools the model calls, until it answers. Tools run sequentially, and a tool that raises becomes an error-flagged result the model can react to rather than an exception the caller must handle. Parameters: Name Type Description Default messages MessagesInput The starting conversation. required tools Sequence[Tool | Any] Callables or tool()-decorated tools. required target Target | None A single target, as for generate(). None route Route | Target | Sequence[Target] | None A route, as for generate(). None max_rounds int Maximum tool rounds before giving up. DEFAULT_MAX_ROUNDS **kwargs Any Forwarded to generate(). {} Returns: Type Description Generation The final Generation, once the model stops Generation calling tools. Raises: Type Description ToolLoopError If the model calls an unknown tool, or the round budget is exhausted. spend spend() -> SpendTotals What this client has spent so far. Returns zeros; never None, when no ledger is attached, so a caller reading this never has to branch on whether accounting was switched on. Check SpendTotals.unknown_requests before treating the figure as complete: requests against a target with no trusted pricing are counted there rather than being silently priced at zero. budget budget( messages: MessagesInput, *, target: Target, schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), sampling: Sampling | None = None, output_reserve_tokens: int | None = None, ) -> ContextBudget Compute the context budget for a request without sending it. This is the preflight calculator: apps assembling large prompts read remaining_tokens to decide how much more material fits, instead of hand-rolling window arithmetic per provider. Pure computation — no request is issued, no network is touched; capabilities come from the catalog plus whatever discovery or probes have already been recorded. Parameters: Name Type Description Default messages MessagesInput The conversation as assembled so far. required target Target The target to budget against. required schema SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None Structured-output schema the real request will carry, if any. None tools Sequence[ToolSpec] Tools the real request will offer, if any. () sampling Sampling | None Sampling controls; max_output_tokens shapes the output reserve. None output_reserve_tokens int | None Overrides the derived output reserve. None Returns: Type Description ContextBudget The computed ContextBudget. When the ContextBudget target's context window is unknown, the budget's verdict is None — never ContextBudget a guess. compare async compare( messages: MessagesInput | GenerationRequest, *, targets: Sequence[Target], schema: SchemaSpec | SupportsJSONSchema | Mapping[str, Any] | None = None, tools: Sequence[ToolSpec] = (), tool_choice: ToolChoice = "auto", sampling: Sampling | None = None, reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, repair: Repair | None = None, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, provider_options: Mapping[str, Mapping[str, Any]] | None = None, metadata: Mapping[str, str] | None = None, max_response_bytes: int | None = None, refresh: bool = False, ) -> tuple[TargetComparison, ...] Compare how one request would behave across targets without generating. Results preserve caller order and are never ranked or consumed by routing. With refresh=False (the default), no adapter is constructed and no network is touched. refresh=True may list models to refresh discovered capabilities. compare_embedding async compare_embedding( inputs: str | Sequence[str], *, targets: Sequence[Target], input_type: EmbeddingInputIntent | None = None, refresh: bool = False, ) -> tuple[EmbeddingTargetComparison, ...] Compare how one embedding request would behave across targets, without dispatching. Results preserve caller order and are never ranked. With refresh=False (the default), no adapter is constructed and no network is touched; refresh=True may list models to refresh discovered capabilities, exactly as compare(). anyinfer.SyncStream SyncStream(loop: _LoopThread, factory: Any) A blocking iterator over stream events, fed from the background loop. Use it as a context manager so an early exit cancels the underlying request instead of leaving it running: with client.stream(messages, target="ollama:qwen3:8b") as stream: for event in stream: ... final = stream.result result property result: Generation The final result. Raises: Type Description RuntimeError If the stream has not been consumed to completion. manifest property manifest: RunManifest | None What this call has done so far, as a RunManifest. The blocking mirror of AsyncStream.manifest, and readable at any point — including after close() cancelled the request, which is when it is most useful. None when the client was built with manifests switched off. __iter__ __iter__() -> Iterator[StreamEvent] Iterate events as they arrive. __next__ __next__() -> StreamEvent Block for the next event, re-raising loop-side exceptions here. __enter__ __enter__() -> SyncStream Enter a context that cancels the request on exit. __exit__ __exit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the stream, cancelling it if it was not fully consumed. close close() -> None Cancel the underlying request and drain any buffered events. collect collect() -> Generation Drain the stream and return the final result. anyinfer.AsyncStream AsyncStream( source: AsyncIterator[StreamEvent], *, builder: ManifestBuilder | None = None, ) An async iterator over stream events, with the final result attached. Supports the three consumption shapes the design targets: iterate deltas, watch for the first-token mark and then read the result, or ignore events and read the result. result property result: Generation The final result. Raises: Type Description RuntimeError If the stream has not been fully consumed yet. manifest property manifest: RunManifest | None What this call has done so far, as a RunManifest. Available at any point, which is the whole reason the handle lives on the stream rather than only on the result: a stream that was cancelled or that failed part-way has no Generation to carry a manifest, and that is precisely the call whose story a caller needs. Such a record has complete=False. None when the client was built with manifests switched off. __aiter__ __aiter__() -> AsyncStream Iterate stream events. __anext__ async __anext__() -> StreamEvent Yield the next event, capturing the result when the stream ends. __aenter__ async __aenter__() -> AsyncStream Enter a context that guarantees the stream is closed. __aexit__ async __aexit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the underlying generator, cancelling any in-flight request. aclose async aclose() -> None Close the stream early, releasing the provider connection. collect async collect() -> Generation Drain the stream and return the final result. anyinfer.MessagesInput module-attribute MessagesInput = str | Message | Sequence[Message] What callers may pass as messages: a bare prompt, one message, or a sequence. anyinfer.ProviderSettings dataclass ProviderSettings( provider_id: str, base_url: str | None = None, api_key: str | None = None, api_version: str | None = None, headers: Mapping[str, str] = dict(), options: Mapping[str, Any] = dict(), timeout_s: float = 120.0, transport: Any | None = None, alias: str | None = None, limits: RateLimits | None = None, ) How one provider instance should be configured on a client. A client may hold several instances of the same underlying engine — two Azure tenants, a local and a remote Ollama — by giving each one an alias. The alias is the instance's identity everywhere else: it is what a alias:model target names, what AdapterPool keys its adapters by, and what telemetry reports. Attributes: Name Type Description provider_id str Registered provider id or alias, e.g. "openai" or "claude". This selects the engine, which adapter is built and how it talks. alias str | None Instance id, when this is one of several instances of provider_id. Defaults to provider_id itself, which is the single-instance case. base_url str | None Endpoint override. Optional for providers with a default; required for ones that have none (openai-compat, azure-foundry). api_key str | None Credential for the provider. Accepts a reference ("env://OPENAI_API_KEY", "credential://system/openai") as well as a literal; it is resolved once, when the adapter is first built, and registered for redaction at that point. api_version str | None Version pin for providers that take one (Azure, Anthropic). headers Mapping[str, str] Extra headers merged into every request. options Mapping[str, Any] Provider-specific settings, per the provider's documented ProviderSetupSpec fields. timeout_s float Default per-request timeout for this provider. transport Any | None Test seam — an httpx2 transport that intercepts this provider's traffic (used by the fake-server and cassette modes). limits RateLimits | None Client-side pacing for this instance, or None for none. Rate limits belong to an account at a provider rather than to the application, which is why they are configured here and not as a client-wide policy. instance_id property instance_id: str This instance's identity: the alias when set, else the provider id. of classmethod of(provider_id: str, **kwargs: Any) -> ProviderSettings Build settings for a provider id, normalizing the id and any alias. anyinfer.Session Session(target: ResolvedTarget, *, supported: bool) A handle threading related requests through one provider's own state. Obtained from session(), passed to generate() or stream(), and threaded forward by the caller. Mutable by design — it is a handle, like a stream, not a domain value, and updated in place after each turn so a caller can keep passing the same object. with client.session("copilot:auto") as chat: first = client.generate("Summarize this report.", session=chat) follow = client.generate("Now list the risks.", session=chat) chat.reuse # 'resumed' — the provider kept the conversation Closing stops the handle being used; it does not reach out to the provider. Server-side state expires on the provider's own schedule, and a library that pretended otherwise would be making a promise it cannot keep. target property target: ResolvedTarget The provider and model this session's state belongs to. supported property supported: bool Whether this provider declares it can keep state between requests. False is not an error: the session is inert, every request behaves exactly as it would without one, and reuse says so on every turn. reuse property reuse: SessionReuse What happened on the most recent turn. turns property turns: int How many requests have been made with this session. active property active: bool Whether the provider is currently holding state for this session. state property state: Mapping[str, Any] The provider's opaque continuation data. Exposed for diagnostics and persistence, never interpreted by the core. Its contents are the provider's business and may change between releases of that provider, so treat it as a token rather than a structure. closed property closed: bool Whether this handle has been closed. applies_to applies_to(target: ResolvedTarget) -> bool Whether this session's state may be sent to target. Provider state is not portable: after a fallback to another provider, or a different model on the same one, the stored handle means nothing there. close close() -> None Stop using this handle. Idempotent. __enter__ __enter__() -> Session Enter a context that closes the handle on exit. __exit__ __exit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the handle. __aenter__ async __aenter__() -> Session Enter an async context that closes the handle on exit. __aexit__ async __aexit__( exc_type: type[BaseException] | None, exc: BaseException | None, tb: TracebackType | None, ) -> None Close the handle. __repr__ __repr__() -> str Show the target, turn count, and last reuse outcome. anyinfer.SessionReuse module-attribute SessionReuse = Literal['fresh', 'resumed', 'unsupported'] What happened on the most recent turn. fresh — the provider started new state (the first turn, or one it had expired). resumed — the provider continued state it already held. unsupported — nothing was reused, because this provider cannot or this turn went somewhere else. anyinfer.Verification dataclass Verification( target: ResolvedTarget | None, ok: bool, reached: bool = False, latency_ms: float = 0.0, detail: str = "", reply: str = "", mechanism: Mechanism | None = None, usage: Usage = Usage(), diagnostics: tuple[Diagnostic, ...] = (), ) What one end-to-end probe of a target found. Never raised, always returned: "this target is broken" is the answer to the question, not a failure to answer it. The two booleans are deliberately separate, because "unreachable" and "reachable but cannot hold a schema" call for completely different fixes. Attributes: Name Type Description target ResolvedTarget | None What the target string resolved to, and — for a provider that picks the model itself, which model actually served the request. ok bool The provider answered, in the shape asked for, with the expected content. reached bool The provider answered at all. True with ok false means the connection and credential are fine and the model's output was not. latency_ms float Wall-clock time for the whole probe, including any retry the route performed. Indicative only — one request is not a benchmark. detail str What went wrong, or empty when nothing did. reply str A bounded excerpt of what came back, for a human to look at. mechanism Mechanism | None The structured-output mechanism actually used, when one was. usage Usage Tokens the probe spent, as the provider reported them. diagnostics tuple[Diagnostic, ...] Anything the provider said about its own runtime while serving this. summary property summary: str One line suitable for a status area or a CLI. anyinfer.Measurement dataclass Measurement( identity: MeasurementIdentity, input_tokens: int | None = None, output_tokens: int | None = None, ttft_ms: float | None = None, total_ms: float = 0.0, prefill_tokens_per_s: float | None = None, decode_tokens_per_s: float | None = None, model_load_ms: float | None = None, measured_at: str | None = None, ) One target's measured throughput. Every rate is optional, and None means not measured rather than zero — the same tri-state rule cost and context windows follow. Attributes: Name Type Description identity MeasurementIdentity What was measured. input_tokens int | None Prompt tokens as the provider counted them. output_tokens int | None Generated tokens as the provider counted them. ttft_ms float | None Time to the first content delta, measured centrally. total_ms float Whole-request wall clock. prefill_tokens_per_s float | None Prompt tokens per second, only when the provider timed its own prefill phase. None otherwise, because deriving it from time-to-first-token would fold queueing and network latency into a figure labelled compute. decode_tokens_per_s float | None Generated tokens per second, from first token to completion. model_load_ms float | None How long the engine spent loading the model before it could answer, when it reported one. This is the warmth signal: a figure here means the run paid a cold start, and None means either the model was already resident or the engine does not report loads at all. Absent that distinction a caller has to measure every target twice and compare, which is what the demo did. Reported by Ollama on every request (load_duration, zero-ish when warm) and by the supervised llama.cpp runtime on the request that started its server. A hosted provider reports nothing: what a shared endpoint spent loading a model is not a property of this request. measured_at str | None ISO-8601 timestamp the caller stamped, when they stamped one. summary property summary: str One line for a status area or a CLI. to_json to_json() -> dict[str, Any] A plain-data form suitable for storage or a machine-readable CLI. from_json classmethod from_json(payload: Any) -> Measurement | None Rebuild a measurement from stored data, or None if it is unreadable. Never raises: a stored measurement is a cache entry, and an unreadable one means measure again, not fail. anyinfer.MeasurementIdentity dataclass MeasurementIdentity( provider_id: str, model: str, endpoint: str | None = None, host: str | None = None, runtime: str | None = None, ) What a measurement is a measurement of. Throughput is not a property of a model; it is a property of a model on an endpoint on a machine with a runtime. Change any of those and the old number is not stale, it is about something else, which is what fingerprint is for. Attributes: Name Type Description provider_id str The configured provider instance. model str The concrete model that served the request. endpoint str | None Normalized base URL, or None for a supervised in-process engine. host str | None A signature of the machine, for locally-executed targets only. None for hosted providers, where this machine's specs are irrelevant. runtime str | None The local runtime variant in use ("cuda", "metal", …), when one applies. fingerprint property fingerprint: str A stable hash of every field, for use as a store key. anyinfer.MeasurementStore MeasurementStore(path: Path | str) An optional, caller-owned file of past measurements. The library persists nothing on its own; an application that wants a "last measured" figure across restarts constructs one of these and points it somewhere. Entries are keyed by MeasurementIdentity.fingerprint, so a measurement taken against a different endpoint, machine, or runtime never masquerades as a fresher version of this one. Reads are total: a missing, truncated, or foreign file yields no entries rather than an exception, because a cache that can break a program is worse than no cache. path property path: Path Where this store reads and writes. get get(identity: MeasurementIdentity) -> Measurement | None The stored measurement for exactly this identity, if any. all all() -> tuple[Measurement, ...] Every stored measurement, oldest entry first. record record(measurement: Measurement) -> None Store a measurement, replacing any earlier one for the same identity. Writes atomically — a store half-written by an interrupted process would fail every subsequent read. anyinfer.BenchmarkSample dataclass BenchmarkSample( elapsed_ms: float, phase: Literal["warmup", "decode", "complete"], estimated_output_tokens: int = 0, output_tokens_per_s: float | None = None, resources: ResourceSample = ResourceSample(), ) One point in a live benchmark time series. The token count and instantaneous rate are estimates while streaming because provider usage is authoritative only at the terminal event. Resource fields are best-effort host readings and remain None when the platform cannot report them. anyinfer.BENCHMARK_PROMPT_TOKENS module-attribute BENCHMARK_PROMPT_TOKENS = 2048 Default prompt size for a measurement. Large enough that prefill is a real phase rather than rounding error, small enough that the whole measurement costs a fraction of a cent on a hosted provider. anyinfer.BENCHMARK_OUTPUT_TOKENS module-attribute BENCHMARK_OUTPUT_TOKENS = 128 Default output size. Decode throughput needs enough tokens to average over; a handful would mostly measure the first one. anyinfer.tool tool( func: Callable[..., Any] | None = None, *, name: str | None = None, description: str | None = None, ) -> Any Turn a function into a Tool, deriving its schema from the signature. Parameter types come from annotations and the description from the docstring, so a tool is declared once rather than being kept in sync with a hand-written schema: @ai.tool def read_file(path: str) -> str: """Read a project file.""" return Path(path).read_text() Parameters: Name Type Description Default func Callable[..., Any] | None The function to wrap, when used bare. None name str | None Overrides the function's name. None description str | None Overrides the docstring summary. None Returns: Type Description Any A Tool, or a decorator producing one. Raises: Type Description ToolLoopError If a parameter's annotation is not a supported JSON type. anyinfer.Tool dataclass Tool(spec: ToolSpec, func: Callable[..., Any]) A callable paired with the ToolSpec derived from its signature. spec instance-attribute spec: ToolSpec The declaration advertised to the model: name, description, and parameter schema. func instance-attribute func: Callable[..., Any] The wrapped callable. May be async def; the loop's dispatcher awaits its result. name property name: str The tool's name, as advertised to the model. call call(arguments: Mapping[str, Any]) -> Any Invoke the underlying function. An async def tool returns a coroutine here; the loop's dispatcher awaits it. --- # Reference / Requests and Messages Source: https://anyinfer.dev/reference/api/requests/ Requests and Messages The request side of the one primitive: everything a GenerationRequest can carry. See the event stream for how a request becomes output. anyinfer.GenerationRequest dataclass GenerationRequest( messages: tuple[Message, ...], schema: SchemaSpec | None = None, tools: tuple[ToolSpec, ...] = (), tool_choice: ToolChoice = "auto", sampling: Sampling = Sampling(), reasoning: ReasoningEffort | None = None, timeout_s: float | None = None, max_response_bytes: int = DEFAULT_MAX_RESPONSE_BYTES, max_input_part_bytes: int = DEFAULT_MAX_INPUT_PART_BYTES, max_input_bytes: int = DEFAULT_MAX_INPUT_BYTES, repair: Repair | None = None, provider_options: Mapping[ str, Mapping[str, Any] ] = dict(), metadata: Mapping[str, str] = dict(), history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, context: ContextRequest | None = None, ) A fully-specified generation request, independent of any provider. This type is a deliberate superset of the OpenAI chat-completions request surface so the serve frontend stays a lossless codec. provider_options is the escape hatch for provider-specific parameters: keys are provider ids, values are mappings passed to that provider's adapter verbatim. The special key "*" applies to whichever provider ends up serving the request, with a provider-specific namespace winning field-by-field over the wildcard. Attributes: Name Type Description messages tuple[Message, ...] The conversation so far, oldest turn first. schema SchemaSpec | None Structured-output contract to enforce, when one was requested. tools tuple[ToolSpec, ...] Tools the model is allowed to call. tool_choice ToolChoice Whether tool use is automatic, forbidden, required, or pinned to one named tool. sampling Sampling Sampling controls; unset fields fall through to provider defaults. reasoning ReasoningEffort | None Requested reasoning effort, for models that support it. timeout_s float | None Per-attempt wall-clock budget; None means DEFAULT_TIMEOUT_S. max_response_bytes int Cap on total streamed response bytes per attempt. repair Repair | None Budget for schema-repair round-trips; None disables repair. provider_options Mapping[str, Mapping[str, Any]] Provider-specific parameters keyed by provider id, passed to the serving adapter verbatim (see above for the "*" wildcard rule). metadata Mapping[str, str] Opaque caller-supplied labels, carried through unchanged and echoed in telemetry. history HistoryPolicy | None Per-request conversation-compaction policy, overriding the client's default. None means "use the client's". Carried on the request rather than passed alongside it so the serve frontend stays a lossless codec: an anyinfer_history object on the wire decodes into exactly this field. cache CachePolicy | None Per-request prompt-cache placement, overriding the client's default. None — the default — means no placement at all, since caching changes what a provider bills. Carried on the request for the same codec reason as history: an anyinfer_cache object on the wire decodes into this field. arena ArenaPolicy | None Per-request fixed fan-out policy, overriding the client's default. Adapters never see it; the client runs and selects every candidate before projection. context ContextRequest | None Explicit caller-approved documents to reduce for the resolved target. None preserves ordinary generation byte-for-byte. effective_timeout_s property effective_timeout_s: float The per-attempt timeout, resolving None to the default. __post_init__ __post_init__() -> None Enforce multimodal request byte ceilings before any adapter can run. with_messages with_messages( messages: Sequence[Message], ) -> GenerationRequest Return a copy of this request with different messages. Used by the repair loop and the tool loop, which extend the conversation without touching any other request field. anyinfer.Message dataclass Message(role: Role, content: tuple[ContentPart, ...]) One turn in a conversation. Attributes: Name Type Description role Role Who authored the turn. content tuple[ContentPart, ...] Ordered parts making up the turn — text runs, tool calls, and tool results. text property text: str Concatenation of this message's Text parts. anyinfer.Role module-attribute Role = Literal['system', 'user', 'assistant', 'tool'] Who authored a message. anyinfer.ContentPart module-attribute ContentPart = ( Text | ToolCall | ToolResult | ImagePart | DocumentPart | AudioPart ) A single piece of message content. anyinfer.Text dataclass Text(text: str) A run of plain text within a message. anyinfer.ImagePart dataclass ImagePart( data: bytes | None = None, url: str | None = None, media_type: str = "image/png", detail: Literal["auto", "low", "high"] | None = None, ) An image supplied inline or by remote URL for a multimodal model. __post_init__ __post_init__() -> None Require exactly one source and an image media type. anyinfer.DocumentPart dataclass DocumentPart( data: bytes | None = None, url: str | None = None, media_type: str = "application/pdf", filename: str | None = None, ) A document supplied inline or by remote URL for a capable model. __post_init__ __post_init__() -> None Require exactly one source and a non-empty media type. anyinfer.AudioPart dataclass AudioPart(data: bytes, media_type: str = 'audio/wav') Inline audio input for a multimodal model. __post_init__ __post_init__() -> None Require bytes and an audio media type. anyinfer.system system(text: str) -> Message Build a system message from plain text. anyinfer.user user(text: str) -> Message Build a user message from plain text. anyinfer.assistant assistant(text: str) -> Message Build an assistant message from plain text. anyinfer.Sampling dataclass Sampling( temperature: float | None = None, top_p: float | None = None, max_output_tokens: int | None = None, stop: tuple[str, ...] = (), ) Sampling controls. Every field defaults to None/empty meaning provider default. AnyInfer never invents a temperature: an unset value is omitted from the wire request entirely. Attributes: Name Type Description temperature float | None Randomness of token selection; higher values sample more freely. top_p float | None Nucleus-sampling cutoff — the probability mass considered for each token. max_output_tokens int | None Upper bound on how many tokens the model may generate. stop tuple[str, ...] Sequences that end generation as soon as one is produced. anyinfer.ReasoningEffort module-attribute ReasoningEffort = Literal[ "minimal", "low", "medium", "high" ] Normalized reasoning effort; each descriptor translates it to its provider's wire form. anyinfer.SchemaSpec dataclass SchemaSpec( json_schema: Mapping[str, Any], name: str = "response" ) A structured-output contract. The json_schema here is the canonical schema. Providers may receive a projected variant on the wire, but responses are always validated against this one. Attributes: Name Type Description json_schema Mapping[str, Any] The canonical JSON Schema every response is validated against. name str Label for the schema, passed to providers whose wire format names the expected response shape. coerce classmethod coerce( obj: SchemaSpec | SupportsJSONSchema | Mapping[str, Any], ) -> SchemaSpec Accept a SchemaSpec, a JSON-schema mapping, or a pydantic-style model. Parameters: Name Type Description Default obj SchemaSpec | SupportsJSONSchema | Mapping[str, Any] The schema in any accepted form. required Returns: Type Description SchemaSpec An equivalent SchemaSpec. Raises: Type Description TypeError If obj is none of the accepted forms. anyinfer.SupportsJSONSchema Bases: Protocol Duck type for pydantic-style models supplied as schemas. model_json_schema model_json_schema() -> Mapping[str, Any] Return the JSON Schema describing this model. anyinfer.Repair dataclass Repair(max_attempts: int = 1) Opt-in bounded repair budget for schema violations. Attributes: Name Type Description max_attempts int Maximum repair round-trips after a schema violation before the violation is surfaced as an error. anyinfer.ArenaPolicy dataclass ArenaPolicy( targets: tuple[str, ...], strategy: Literal[ "first_valid", "consensus", "cheapest", "fastest", "judge", "synthesize", ] = "first_valid", judge_target: str | None = None, instructions: str | None = None, concurrency: int = 4, min_candidates: int = 1, reveal_targets: bool = False, memoize_tools: Literal[ "read_only", "all", "opt_in", "off" ] = "read_only", ) Fan one request out to fixed targets and select after every branch finishes. __post_init__ __post_init__() -> None Reject an arena whose fixed bounds or strategy are not executable. anyinfer.ContextRequest dataclass ContextRequest( documents: tuple[ContextDocument, ...], query: str | None = None, strategy: str = "auto", max_tokens: int | None = None, placement: Literal["system", "prepend_user"] = "system", tuning: ContextTuning = DEFAULT_TUNING, max_request_documents: int = DEFAULT_REQUEST_DOCUMENTS, max_request_bytes: int = DEFAULT_REQUEST_BYTES, ) Documents explicitly approved by the caller for stateless reduction. __post_init__ __post_init__() -> None Reject inference-spending strategies and oversized request payloads. anyinfer.ToolSpec dataclass ToolSpec( name: str, description: str, parameters: Mapping[str, Any], annotations: ToolAnnotations = ToolAnnotations(), ) A tool the model may call. Attributes: Name Type Description name str Identifier the model uses to invoke the tool. description str What the tool does, shown to the model to guide when to call it. parameters Mapping[str, Any] JSON Schema describing the tool's arguments. annotations ToolAnnotations Untrusted behavioural hints from whatever declared the tool. Empty for tools declared in Python, where the code is its own description. anyinfer.ToolChoice module-attribute ToolChoice = Literal['auto', 'none', 'required'] | str "auto" | "none" | "required", or a specific tool name. anyinfer.ToolResult dataclass ToolResult( call_id: str, content: str, is_error: bool = False ) The outcome of executing a tool call, fed back to the model. Attributes: Name Type Description call_id str Id of the ToolCall this result answers. content str The tool's output, rendered as text for the model to read. is_error bool Whether the tool failed; the content then describes the failure. anyinfer.Target module-attribute Target = str Where a request should go. Grammar:: Target = ALIAS | PROVIDER ":" MODEL PROVIDER = [a-z0-9-]+ (after normalization: lowercased, stripped, "_" -> "-") MODEL = the remainder verbatim Split on the first colon only — model ids may themselves contain colons ("ollama:qwen3:8b"). A target without a colon must match a catalog alias. Prompt Caching Opt-in placement of a provider's prompt cache. Off unless asked for: caching changes what a provider bills and how long it keeps a copy of the prompt. What it caches is the prefix the caller sends, on the provider's side; it never skips a call or reuses an answer. anyinfer.CachePolicy dataclass CachePolicy( mode: CacheMode = "auto", min_segment_tokens: int = 1024, max_marks: int = 4, include_tools: bool = True, include_system: bool = True, ) Opt-in prompt-cache placement for one request. Off unless asked for. Caching changes what a provider bills and how long it retains a copy of the prompt, and neither is a decision this library makes on a caller's behalf — a request that carries no policy is cached exactly as much as it was before this existed, which is not at all. What it caches is the prefix you send, on the provider's side, for the provider's retention window. It never skips a call and never reuses an answer. Attributes: Name Type Description mode CacheMode off disables placement; auto uses the strongest available mechanism; explicit requires per-segment marks and reports a dropped parameter when the target has none. min_segment_tokens int Segments estimated smaller than this are not worth marking — below a provider's own floor a mark is billed as a cache write that no later read ever amortizes. max_marks int Most marks to place, clamped down to whatever the provider accepts. include_tools bool Whether tool declarations may be marked. They are stable across a conversation and often large, so they are usually the best single mark. include_system bool Whether the system block may be marked. active property active: bool Whether this policy will attempt any placement. __post_init__ __post_init__() -> None Reject a policy that cannot be applied. Raises: Type Description ValueError On an unknown mode, a negative floor, or a non-positive mark budget. anyinfer.CacheMode module-attribute CacheMode = Literal['off', 'auto', 'explicit'] What a request asks for. auto uses the strongest mechanism the target offers. explicit asks for marks and reports a dropped parameter if the target has none, which is the setting for a caller who would rather know than silently get nothing. anyinfer.CacheMechanism module-attribute CacheMechanism = Literal['explicit', 'implicit'] How a provider's prompt cache is engaged. explicit The provider accepts per-segment cache marks on the wire, so the core decides where the cacheable prefix ends and the adapter spells that mark. implicit The provider caches stable prefixes on its own. There is nothing to send; the core's only duty is to leave the prefix undisturbed and to notice when the caller's own request defeats it. --- # Reference / Results and Stream Events Source: https://anyinfer.dev/reference/api/results/ Results and Stream Events The output side: the final Generation, its usage and timing, and the typed events a stream yields on the way there. Ordering guarantees are documented in the event stream. The Final Result anyinfer.Generation dataclass Generation( text: str, structured: Any | None, tool_calls: tuple[ToolCall, ...], target: ResolvedTarget, finish_reason: FinishReason, usage: Usage, timing: Timing, structured_mechanism: Mechanism | None = None, cache_mechanism: CacheMechanism | None = None, repair_attempts: int = 0, attempts: tuple[AttemptRecord, ...] = (), warnings: tuple[str, ...] = (), raw: Any | None = None, manifest: RunManifest | None = None, arena: ArenaResult | None = None, context_reduction: ContextSummary | None = None, ) The final result of a generation request. Attributes: Name Type Description text str The assistant's full text output. Empty when the model answered only with tool calls or structured output. structured Any | None The parsed, schema-validated object when structured output was requested; None otherwise. tool_calls tuple[ToolCall, ...] Tool invocations the model requested, in order. Empty when none. target ResolvedTarget The provider and model that actually produced this result — after routing, so it may differ from the first target asked for. finish_reason FinishReason Normalized reason the generation stopped. usage Usage Token accounting, normalized across providers. timing Timing Centrally-measured latency for the winning attempt. structured_mechanism Mechanism | None How structured output was enforced for this result (grammar, json_schema, json_mode, or prompt); None when no schema was requested. cache_mechanism CacheMechanism | None How prompt caching was engaged (explicit marks, or implicit prefix stability); None when no policy was in force or the target offered nothing. Distinct from usage.cache_read_tokens, which is what the provider reported — this is what was asked of it. repair_attempts int How many schema-repair round-trips were needed before structured validated. 0 means the first response validated. attempts tuple[AttemptRecord, ...] The full routing trail, including failed and retried attempts. warnings tuple[str, ...] Non-fatal notices accumulated along the way (capability downgrades, estimated values, and the like). raw Any | None The provider-native response payload, as an escape hatch for fields the normalized types do not carry. None unless the request asked to keep it. manifest RunManifest | None The run manifest — one content-free record of which target won, which mechanisms were used, what was dropped or reduced, and what it cost. None when the client was built with manifests switched off. It is a projection of this call's telemetry events and this result, never an independent account of them. arena ArenaResult | None Every arena candidate and the terminal selection, or None for an ordinary generation. context_reduction ContextSummary | None Content-free account of per-request corpus reduction. anyinfer.Usage dataclass Usage( input_tokens: int | None = None, output_tokens: int | None = None, total_tokens: int | None = None, cache_read_tokens: int | None = None, cache_write_tokens: int | None = None, reasoning_tokens: int | None = None, cost_usd: Decimal | None = None, search_units: int | None = None, ) Token accounting for one generation. Every field is optional: a provider that does not report a number leaves it None rather than reporting a guess. Attributes: Name Type Description input_tokens int | None Tokens in the prompt, as counted by the provider. output_tokens int | None Tokens the model generated, including tool-call payloads. total_tokens int | None Prompt plus completion tokens; normalized fills it from the other two when the provider does not report it. cache_read_tokens int | None Prompt tokens served from the provider's prompt cache. cache_write_tokens int | None Prompt tokens written into the provider's prompt cache. reasoning_tokens int | None Tokens spent on hidden reasoning/thinking, where reported. cost_usd Decimal | None Cost of the call in US dollars, computed from per-token pricing when pricing is known. search_units int | None Provider-native billed search units (reranking). A distinct billing dimension with its own field on purpose — a search unit is never a token count, and encoding one as the other would fabricate usage. normalized normalized() -> Usage Fill total_tokens from input + output when both are known. merge merge(other: Usage) -> Usage Overlay other's known fields onto this one. Later usage reports win; None never overwrites a known value. Streaming providers report usage incrementally, so the router merges rather than replaces. This is an overlay, not addition — to total usage across the internal batches of one request, use Usage.sum(). sum classmethod sum(parts: Sequence[Usage]) -> Usage Total usage across the internal batches of one logical request. A field totals only when every part reports it; if any part is unknown, the total stays None — a partial sum would understate spend while reading as authoritative. An empty parts is all-unknown. anyinfer.Timing dataclass Timing( started_at: float, first_token_ms: float | None = None, total_ms: float = 0.0, output_tokens_per_s: float | None = None, phases: Mapping[str, float] = dict(), ) Centrally-measured timings for one attempt. All values are measured by the core against time.monotonic() so that definitions are identical across providers. phases carries provider-reported sub-timings (e.g. Ollama's model load) in milliseconds. Attributes: Name Type Description started_at float Monotonic-clock reading when the attempt began; meaningful only for computing intervals, not as wall-clock time. first_token_ms float | None Time to the first content event (text, reasoning, or tool-call delta) after attempt start; None when no content ever arrived. total_ms float Full duration of the attempt, start to completion. output_tokens_per_s float | None Decode throughput, measured from first token to completion; None when output tokens or first-token time are unknown. phases Mapping[str, float] Provider-reported sub-timings, keyed by phase name, in milliseconds. anyinfer.AttemptRecord dataclass AttemptRecord( target: ResolvedTarget, outcome: Outcome, error: ErrorInfo | None = None, timing: Timing | None = None, ) One entry in a request's routing trail. Attributes: Name Type Description target ResolvedTarget The provider and model this attempt was sent to. outcome Outcome How the attempt ended. error ErrorInfo | None Snapshot of the failure, for attempts that did not succeed. timing Timing | None Measured timings, when the attempt progressed far enough to have any. anyinfer.Outcome module-attribute Outcome = Literal[ "ok", "retried", "failed", "skipped_unhealthy", "redirected", ] How a single routing attempt ended. redirected marks a completed attempt whose content-filter refusal sent the route to Route.content_policy_targets instead of surfacing the refusal. anyinfer.ErrorInfo dataclass ErrorInfo( type_name: str, provider: str | None, phase: str, retryable: bool, http_status: int | None, detail: str, ) A serializable, already-redacted snapshot of any AnyInfer error. Captured from the exception at failure time, so attempt records and telemetry can carry the failure long after the exception itself has been handled. Attributes: Name Type Description type_name str Class name of the exception this snapshot was captured from. provider str | None Id of the provider involved, or None for provider-independent failures. phase str Request-lifecycle stage that failed (configure, discover, generate, stream, validate, or cleanup). retryable bool Whether retrying the identical request could plausibly succeed. http_status int | None Status code, for failures that came from an HTTP response. detail str Human-readable description, already redacted and capped at DETAIL_MAX_CHARS characters. anyinfer.Diagnostic dataclass Diagnostic( code: str, severity: DiagnosticSeverity, message: str ) Something a provider noticed about itself while serving requests. Not an error and not a capability: an error stops a request, and a capability is a fact about a model. This is the third thing — the request worked, and something about how it worked is worth saying out loud. A model that spilled out of VRAM and is now running at a tenth of its expected speed answers perfectly; the caller simply has no way to know why it took ninety seconds unless the provider says so. Diagnostics are advisory by construction: collecting them never fails a request, a provider that cannot answer reports nothing, and nothing here is load-bearing for routing. Content-free — a diagnostic describes the runtime, never the prompt. Attributes: Name Type Description code str Stable machine-readable identifier, e.g. "ollama.gpu-spill". Callers match on this; the message is for people. severity DiagnosticSeverity "warning" for a condition degrading this request, "info" for context that merely explains it. message str One human-readable sentence, already bounded and redacted. anyinfer.DiagnosticSeverity module-attribute DiagnosticSeverity = Literal['info', 'warning'] How much a runtime diagnostic should worry the caller. Never an error: a condition that should fail a request is an exception, not a note attached to a successful one. anyinfer.FinishReason module-attribute FinishReason = Literal[ "stop", "length", "tool_calls", "content_filter", "other", ] Normalized reason a generation stopped. anyinfer.ArenaResult dataclass ArenaResult( candidates: tuple[Candidate, ...], winner: Candidate | None, strategy: str, agreement: int | None = None, synthesized: Generation | None = None, calls: int = 0, memoized_tool_calls: int = 0, usage: Usage = Usage(), usage_complete: bool = True, ) Every candidate, terminal selection, and aggregate accounting for an arena. summary summary() -> str Render one content-free status line. anyinfer.Candidate dataclass Candidate( target: ResolvedTarget, generation: Generation | None = None, error: ErrorInfo | None = None, valid: bool | None = None, elapsed_ms: float = 0.0, rounds: int | None = None, tool_calls: int = 0, ) One arena branch's final answer or bounded, redacted failure. anyinfer.TargetComparison dataclass TargetComparison( requested: str, resolved: ResolvedTarget | None = None, resolvable: bool = True, reason: str = "", fits: bool | None = None, budget: ContextBudget | None = None, structured_mechanism: Mechanism | None = None, mechanism_rungs: tuple[MechanismRung, ...] = (), dropped: tuple[DroppedParameter, ...] = (), cache: CachePlan | None = None, cost: CostEstimate | None = None, capability_provenance: Mapping[ str, Provenance ] = dict(), notes: tuple[str, ...] = (), ) What one request would become on one target, without dispatching. Unresolvable targets are records rather than exceptions. Their target-dependent fields are None and reason says what configuration or identity was missing. to_dict to_dict() -> dict[str, Any] Return a stable JSON-safe representation. from_dict classmethod from_dict(data: Mapping[str, Any]) -> TargetComparison Rebuild a comparison produced by to_dict, ignoring unknown keys. anyinfer.EmbeddingTargetComparison dataclass EmbeddingTargetComparison( requested: str, resolved: ResolvedTarget | None = None, resolvable: bool = True, reason: str = "", fits: bool | None = None, dimensions: int | None = None, dimension_choices: tuple[int, ...] = (), max_batch_inputs: int | None = None, max_input_tokens: int | None = None, input_intents: tuple[EmbeddingInputIntent, ...] = (), normalized: bool | None = None, cost: CostEstimate | None = None, capability_provenance: Mapping[ str, Provenance ] = dict(), notes: tuple[str, ...] = (), ) What one embedding request would become on one target, without dispatching. A separate type from TargetComparison rather than an optional section grafted onto it: generation's dimensions — mechanism rungs, cache planning, structured-output fallback — have no embedding counterpart at all, so folding both into one type would mean every embedding comparison carries a dozen fields that are always None. The dimensions here (space capacity, batch limit, intents, pricing) are what an embedding call actually varies by. Unresolvable targets are records rather than exceptions, exactly as TargetComparison treats them: target-dependent fields are None/empty and reason says what was missing. to_dict to_dict() -> dict[str, Any] Return a stable JSON-safe representation. from_dict classmethod from_dict( data: Mapping[str, Any], ) -> EmbeddingTargetComparison Rebuild a comparison produced by to_dict, ignoring unknown keys. anyinfer.RunManifest dataclass RunManifest( format: str = MANIFEST_FORMAT, anyinfer_version: str = "", request_id: str = "", complete: bool = False, request: RequestFacet = RequestFacet(), route: RouteFacet = RouteFacet(), capability: CapabilityFacet = CapabilityFacet(), attempts: tuple[AttemptFacet, ...] = (), structured: SchemaFacet = SchemaFacet(), cache: CacheFacet = CacheFacet(), context: ContextFacet = ContextFacet(), dropped: tuple[DroppedParameter, ...] = (), usage: UsageFacet = UsageFacet(), timing: TimingFacet = TimingFacet(), notes: tuple[str, ...] = (), payloads: PayloadFacet | None = None, operation: InferenceOperation = "generation", embedding_space: EmbeddingSpace | None = None, ) One versioned, content-free record of what a single call did. Attributes: Name Type Description format str Manifest format version; see MANIFEST_FORMAT. anyinfer_version str The library version that produced it, so a stale record is recognisable as one. request_id str The correlation id every event for this call carried. complete bool Whether the call finished. False on a manifest read from a cancelled or still-running stream, where the record is a partial account rather than a wrong one. request RequestFacet Shape and fingerprints of what was asked for. route RouteFacet Targets requested, considered, and resolved. capability CapabilityFacet Provenance-tagged capabilities the call consumed. attempts tuple[AttemptFacet, ...] The routing trail, one entry per attempt. structured SchemaFacet The structured-output ladder and the repair loop. cache CacheFacet Prompt-cache plan and reported accounting. context ContextFacet Reductions applied before dispatch. dropped tuple[DroppedParameter, ...] Parameters the target would not honour. usage UsageFacet Token accounting and cost. timing TimingFacet Latency of the winning attempt. notes tuple[str, ...] Warnings and provider diagnostics, in the order they arrived. payloads PayloadFacet | None Prompt and response text, present only when explicitly asked for. operation InferenceOperation Which inference operation this record describes. "generation" manifests carry every facet; embedding and rerank manifests leave the generation-only facets (request, structured, cache, context, payloads) at their empty defaults. embedding_space EmbeddingSpace | None The vector-space identity an embedding call produced, so an index builder can persist exactly what a stored corpus was embedded with. None for every other operation. to_dict to_dict() -> dict[str, Any] Render the manifest as JSON-safe data. Returns: Type Description dict[str, Any] A plain dictionary of primitives, lists, and dictionaries — directly dict[str, Any] serializable with json.dumps. from_dict classmethod from_dict(data: Mapping[str, Any]) -> RunManifest Rebuild a manifest from to_dict output. Unknown keys are ignored, which is what makes the format's additive rule safe: a reader on an older release still loads a newer manifest. Parameters: Name Type Description Default data Mapping[str, Any] A mapping produced by to_dict, or parsed from one. required Returns: Type Description RunManifest The reconstructed manifest. to_json to_json(*, indent: int | None = 2) -> str Serialize the manifest to a JSON string. Parameters: Name Type Description Default indent int | None Indentation passed to json.dumps; None for the compact form. 2 Returns: Type Description str The serialized manifest. anyinfer.ContextSummary dataclass ContextSummary( strategy: str, representation: str, candidate_count: int, selected_count: int, omitted_count: int, estimated_tokens: int, complete: bool, ) Content-free account of what corpus reduction sent and omitted. from_reduction classmethod from_reduction(reduction: Reduction) -> ContextSummary Project a full reduction onto the response-safe summary. to_dict to_dict() -> dict[str, object] Serialize for the sidecar extension. Run Manifest Facets anyinfer.MANIFEST_FORMAT module-attribute MANIFEST_FORMAT = '1' Manifest format version. Bumped when an existing field's meaning changes, never when a field is added — a reader that ignores unknown keys survives additions, which is the same rule the context envelope follows. anyinfer.RequestFacet dataclass RequestFacet( message_count: int = 0, role_counts: Mapping[str, int] = dict(), char_count: int = 0, estimated_tokens: int | None = None, schema_present: bool = False, schema_name: str | None = None, schema_digest: str | None = None, tool_names: tuple[str, ...] = (), tool_choice: str = "auto", sampling: Mapping[str, Any] = dict(), reasoning: str | None = None, timeout_s: float | None = None, repair_budget: int = 0, metadata_keys: tuple[str, ...] = (), ) The shape and fingerprints of what was asked for, never the payload. Attributes: Name Type Description message_count int How many messages the request carried. role_counts Mapping[str, int] Message count per role, in role order. char_count int Total characters of message text. estimated_tokens int | None Planning-side input estimate, or None when not computed. schema_present bool Whether structured output was requested. schema_name str | None The schema's label, which is a title rather than content. schema_digest str | None SHA-256 of the canonical schema JSON, so two runs can be compared without either one carrying the schema body. tool_names tuple[str, ...] Names of the tools offered, in order. tool_choice str How tool use was constrained. sampling Mapping[str, Any] Sampling controls actually set, omitting the ones left unset. reasoning str | None Requested reasoning effort, when one was asked for. timeout_s float | None Per-attempt wall clock the request carried, when set. repair_budget int Repair round-trips the request allowed. metadata_keys tuple[str, ...] Keys of caller-supplied metadata; values may be anything. anyinfer.RouteFacet dataclass RouteFacet( requested: tuple[str, ...] = (), resolved: str | None = None, considered: tuple[RouteStep, ...] = (), ) Which targets were asked for, which one answered, and what happened between. Attributes: Name Type Description requested tuple[str, ...] The fallback chain as the caller wrote it, unresolved. resolved str | None The target that produced the result, or None when none did. considered tuple[RouteStep, ...] Every target the router touched, in the order it touched them. anyinfer.RouteStep dataclass RouteStep(target: str, outcome: str, reason: str = '') One target the router considered, and what became of it. Attributes: Name Type Description target str The resolved target, as provider:model. outcome str ok, failed, retried, skipped_unhealthy, redirected, or abandoned for a target the route left before it produced a result. reason str Why, in the words the router used — a health gate, a context overflow, a content-policy redirect, or the error that ended it. anyinfer.CapabilityFacet dataclass CapabilityFacet( target: str = "", facts: tuple[SourcedFact, ...] = () ) Every provenance-tagged capability the call actually consumed. Attributes: Name Type Description target str The target these capabilities describe. facts tuple[SourcedFact, ...] One entry per known capability field, provenance intact. anyinfer.SourcedFact dataclass SourcedFact(name: str, value: Any, provenance: str) One capability value the call consumed, with its provenance intact. Provenance is carried verbatim rather than collapsed into a bare value: "the context window was 8192" and "the context window was assumed to be 8192" are different statements, and a manifest that could not tell them apart would be useless for the question it exists to answer. Attributes: Name Type Description name str Which capability this is, e.g. context_window. value Any The value itself, rendered as JSON-safe data. provenance str Where it came from — catalog, discovered, probed, default, or override. anyinfer.AttemptFacet dataclass AttemptFacet( target: str, attempt_number: int = 1, outcome: str = "ok", error: ErrorInfo | None = None, first_token_ms: float | None = None, total_ms: float | None = None, queued_ms: float | None = None, retry_reason: str | None = None, retry_delay_s: float | None = None, paced_s: Mapping[str, float] = dict(), ) One attempt against one target, with the reason it ended as it did. Attributes: Name Type Description target str The resolved target attempted. attempt_number int 1-based count against this target; a fallback restarts it at 1. outcome str How it ended, using the same vocabulary as anyinfer.AttemptRecord. error ErrorInfo | None The failure snapshot, for an attempt that did not succeed. first_token_ms float | None Time to the first content delta, when any arrived. total_ms float | None Full duration of the attempt. queued_ms float | None How long client-side pacing held it before dispatch. retry_reason str | None Why a retry was scheduled after this attempt, when one was. retry_delay_s float | None How long the router waited before retrying. paced_s Mapping[str, float] Seconds this attempt spent waiting on a rate limiter, and why, summed per reason. anyinfer.SchemaFacet dataclass SchemaFacet( requested: bool = False, chosen: str | None = None, used: str | None = None, ladder: tuple[MechanismRung, ...] = (), repair_attempts: int = 0, repairs: tuple[RepairRecord, ...] = (), validated: bool = False, ) What was asked of the structured-output ladder, and what it delivered. Attributes: Name Type Description requested bool Whether the request carried a schema at all. chosen str | None The mechanism the ladder selected before dispatch. used str | None The mechanism the winning attempt actually used. ladder tuple[MechanismRung, ...] Every rung considered, strongest first, with the reason each was rejected. repair_attempts int How many repair round-trips were spent. repairs tuple[RepairRecord, ...] One record per repair, with the validation errors that caused it. validated bool Whether a structured value was finally produced. anyinfer.MechanismRung dataclass MechanismRung( mechanism: str, available: bool, reason: str = "" ) One structured-output rung and why it was or was not selected. anyinfer.CacheFacet dataclass CacheFacet( policy_mode: str | None = None, mechanism: str | None = None, mark_count: int = 0, estimated_cacheable_tokens: int = 0, read_tokens: int | None = None, write_tokens: int | None = None, ) What was planned for the target's prompt cache, and what it reported back. read_tokens and write_tokens are what the provider said; everything else is what was asked of it. The two are kept apart because an intention is not a saving. Attributes: Name Type Description policy_mode str | None The cache mode in force, or None when no policy applied. mechanism str | None How caching was engaged: explicit marks or implicit prefix stability. None means caching was not engaged. mark_count int How many marks were placed; always zero for implicit. estimated_cacheable_tokens int Planning-side size of what the plan tried to cache. read_tokens int | None Prompt tokens the provider reported serving from its cache. write_tokens int | None Prompt tokens the provider reported writing into its cache. anyinfer.ContextFacet dataclass ContextFacet(reductions: tuple[ReductionRecord, ...] = ()) Reductions the request went through before it was sent. Attributes: Name Type Description reductions tuple[ReductionRecord, ...] One record per reduction, in the order they were applied. anyinfer.ReductionRecord dataclass ReductionRecord( strategy: str, representation: str, candidate_count: int = 0, selected_count: int = 0, omitted_count: int = 0, estimated_tokens: int = 0, max_tokens: int = 0, binding_constraints: tuple[str, ...] = (), calls: int = 0, complete: bool = True, ) One context reduction applied on the way to dispatch. Attributes: Name Type Description strategy str The strategy requested, or history for a compacted conversation. representation str The strategy actually applied. candidate_count int Documents or messages offered to the reducer. selected_count int Documents kept at detail fidelity, or messages retained. omitted_count int What was not represented in detail. estimated_tokens int Planning-side estimate of the result. max_tokens int The budget the reduction was held to. binding_constraints tuple[str, ...] Which ceilings excluded at least one candidate. calls int Generation calls the reduction itself spent. complete bool Whether nothing was omitted. anyinfer.RepairRecord dataclass RepairRecord( attempt_number: int, mechanism: str | None = None, errors: tuple[str, ...] = (), ) One schema-repair round trip. Attributes: Name Type Description attempt_number int 1-based repair count within this generation. mechanism str | None The mechanism in force when validation failed. errors tuple[str, ...] The validation messages that triggered the repair. anyinfer.DroppedParameter dataclass DroppedParameter(target: str, parameter: str, reason: str) A requested parameter the target would not honour as asked. Attributes: Name Type Description target str Which target withheld it. parameter str The parameter name, dotted for a field of a compound one. reason str What the target did instead. anyinfer.UsageFacet dataclass UsageFacet( input_tokens: int | None = None, output_tokens: int | None = None, total_tokens: int | None = None, cache_read_tokens: int | None = None, cache_write_tokens: int | None = None, reasoning_tokens: int | None = None, cost_usd: str | None = None, estimated_fields: Mapping[str, str] = dict(), search_units: int | None = None, ) Token accounting and cost, with estimated figures marked as estimated. Attributes: Name Type Description input_tokens int | None Prompt tokens, as counted by the provider. output_tokens int | None Generated tokens. total_tokens int | None Prompt plus completion. cache_read_tokens int | None Prompt tokens served from the provider's cache. cache_write_tokens int | None Prompt tokens written into it. reasoning_tokens int | None Tokens spent on hidden reasoning, where reported. cost_usd str | None Cost as a decimal string, or None when pricing is not trustworthy for this target. Never zero for an unpriced call. estimated_fields Mapping[str, str] Usage fields that were derived rather than reported, each with the method used. search_units int | None Provider-native billed search units (reranking), where reported. anyinfer.TimingFacet dataclass TimingFacet( first_token_ms: float | None = None, total_ms: float | None = None, output_tokens_per_s: float | None = None, phases: Mapping[str, float] = dict(), ) Centrally-measured latency for the winning attempt. Attributes: Name Type Description first_token_ms float | None Time to the first content delta. total_ms float | None Full duration. output_tokens_per_s float | None Decode throughput. phases Mapping[str, float] Provider-reported sub-timings, in milliseconds. anyinfer.PayloadFacet dataclass PayloadFacet( prompt_text: str | None = None, response_text: str | None = None, schema_body: str | None = None, tool_arguments: tuple[str, ...] = (), repair_texts: tuple[str, ...] = (), ) The strings a content-free manifest deliberately leaves out. Populated only when a manifest is built with payloads enabled, and every value here has already passed through the redaction registry. The default manifest carries None in its place, which is what makes "safe to paste into a public issue tracker" a structural property rather than a promise about field contents. Attributes: Name Type Description prompt_text str | None The request's message text, flattened. response_text str | None The final response text. schema_body str | None The JSON Schema the request carried. tool_arguments tuple[str, ...] Arguments of each tool call the model requested, as JSON. repair_texts tuple[str, ...] The responses that failed validation, in repair order. anyinfer.manifest_json_schema manifest_json_schema() -> dict[str, Any] The JSON Schema a serialized manifest validates against. Published so a golden-file workflow has something to check against, and marked pre-1.0 alongside the rest of this API. Readers must ignore unknown keys: the format adds fields without a version bump, and only a change of meaning bumps MANIFEST_FORMAT. Returns: Type Description dict[str, Any] The schema as a plain dictionary. anyinfer.context.ContextSummary dataclass ContextSummary( strategy: str, representation: str, candidate_count: int, selected_count: int, omitted_count: int, estimated_tokens: int, complete: bool, ) Content-free account of what corpus reduction sent and omitted. from_reduction classmethod from_reduction(reduction: Reduction) -> ContextSummary Project a full reduction onto the response-safe summary. to_dict to_dict() -> dict[str, object] Serialize for the sidecar extension. Stream Events anyinfer.StreamEvent module-attribute StreamEvent = ( TextDelta | ReasoningDelta | ToolCallDelta | UsageUpdate | TimingMark | AttemptFailed | StreamEnded ) Any event a consumer may observe. anyinfer.TextDelta dataclass TextDelta(text: str) A fragment of visible answer text. anyinfer.ReasoningDelta dataclass ReasoningDelta(text: str) A fragment of reasoning/thinking text, excluded from the answer text. anyinfer.ToolCall dataclass ToolCall(id: str, name: str, arguments: Mapping[str, Any]) A model's request to invoke a tool. Attributes: Name Type Description id str Provider-assigned call id. Adapters synthesize "call_0", "call_1"… when the provider omits one, so downstream correlation always has a key. name str The tool being called. arguments Mapping[str, Any] Parsed JSON arguments. An unparseable argument payload yields {} and a warning on the Generation. anyinfer.ToolCallDelta dataclass ToolCallDelta( index: int, call_id: str | None, name: str | None, arguments_fragment: str, ) A fragment of a tool call. Fragments are correlated by index — the tool-call slot within the response. Concatenate arguments_fragment per index, then JSON-parse the result. Attributes: Name Type Description index int The tool-call slot within the response this fragment belongs to. call_id str | None Provider-assigned call id, on fragments that carry it. name str | None Name of the tool being called, on fragments that carry it. arguments_fragment str The next piece of this slot's JSON argument text; may be empty. anyinfer.UsageUpdate dataclass UsageUpdate(usage: Usage) A usage report; may arrive mid-stream and more than once. anyinfer.TimingMark dataclass TimingMark(name: TimingMarkName, at_ms: float) A centrally-measured timing point, in milliseconds since attempt start. Attributes: Name Type Description name TimingMarkName Which point on the attempt clock this marks. at_ms float Milliseconds elapsed since the attempt started. anyinfer.TimingMarkName module-attribute TimingMarkName = Literal['attempt_start', 'first_token'] Named points on the attempt clock. anyinfer.AttemptFailed dataclass AttemptFailed(record: AttemptRecord) A target attempt failed; a retry or fallback may follow. Attributes: Name Type Description record AttemptRecord The failed attempt's routing-trail entry: target, outcome, error snapshot, and any timing. anyinfer.StreamEnded dataclass StreamEnded(result: Generation) Terminal event carrying the assembled result. --- # Reference / Embeddings and Reranking Source: https://anyinfer.dev/reference/api/embeddings/ Embeddings and Reranking Two stateless inference operations alongside generation: turning text into vectors, and ranking documents against a query. Both are typed and routed the same way generation is (target resolution, retries, fallback, usage, and telemetry), but neither is a field on GenerationRequest. See the embeddings concept page for the embedding-space safety rule that governs fallback. Embedding anyinfer.EmbeddingRequest dataclass EmbeddingRequest( inputs: tuple[str, ...], input_type: EmbeddingInputIntent | None = None, dimensions: int | None = None, expected_space: EmbeddingSpace | None = None, timeout_s: float | None = None, max_response_bytes: int = DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES, metadata: Mapping[str, str] = dict(), provider_options: Mapping[ str, Mapping[str, Any] ] = dict(), batch: BatchPolicy = BatchPolicy(), retain_raw: bool = False, allow_incompatible_fallback: bool = False, ) A request to embed one or more texts into vectors. Attributes: Name Type Description inputs tuple[str, ...] Texts to embed, in order. Duplicates are preserved exactly — deduplication is never implicit, because it would change reported usage and may change provider-side billing or behavior. input_type EmbeddingInputIntent | None What the embedded text will be used for, when the target model distinguishes it. None means no intent is asserted. dimensions int | None Requested output dimensionality, for models supporting native dimensionality reduction. None means the model's default. expected_space EmbeddingSpace | None An EmbeddingSpace the caller expects the result to match. When set, a successful but incompatible provider response is rejected rather than returned, per the cross-space safety rule. timeout_s float | None Per-attempt wall-clock budget; None means DEFAULT_TIMEOUT_S. max_response_bytes int Hard cap on one provider response body. Defaults to DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES — vector batches are far larger than chat responses. metadata Mapping[str, str] Caller-supplied, opaque request metadata carried through telemetry. provider_options Mapping[str, Mapping[str, Any]] Escape hatch, namespaced by provider id, passed through verbatim to the matching adapter and consulted by no core logic. batch BatchPolicy Core-owned batching policy for this request. retain_raw bool Whether to keep the provider's raw response payload on the result. allow_incompatible_fallback bool Explicit opt-in permitting fallback to a route target that cannot be proven to share the primary target's embedding space. Off by default because wrong-space vectors fail silently when compared; a result served through this opt-in always carries a warning naming both targets. effective_timeout_s property effective_timeout_s: float The timeout that applies, honoring the module default when unset. __post_init__ __post_init__() -> None Reject a request with no inputs. Empty input is a local validation error performing no provider call — it is never sent, never billed, and never retried. Raises: Type Description ValueError If inputs is empty. anyinfer.EmbeddingResult dataclass EmbeddingResult( vectors: tuple[EmbeddingVector, ...], target: ResolvedTarget, space: EmbeddingSpace, usage: Usage, timing: Timing, attempts: tuple[AttemptRecord, ...] = (), warnings: tuple[str, ...] = (), raw: Any | None = None, manifest: Any | None = None, ) The result of one embedding request. Attributes: Name Type Description vectors tuple[EmbeddingVector, ...] Embedding vectors in the exact order of EmbeddingRequest.inputs. target ResolvedTarget The provider and model that actually produced these vectors. space EmbeddingSpace Identity of the vector space these vectors live in. usage Usage Token/billing accounting, normalized across providers. timing Timing Centrally-measured latency for the winning attempt. attempts tuple[AttemptRecord, ...] The full routing trail, including failed and retried attempts. warnings tuple[str, ...] Non-fatal notices accumulated along the way. raw Any | None The provider-native response payload, when the request asked to keep it. manifest Any | None The run manifest for this call, or None when manifests are off. anyinfer.EmbeddingVector dataclass EmbeddingVector(values: tuple[float, ...]) One immutable embedding vector, validated on construction. Attributes: Name Type Description values tuple[float, ...] The vector's components, in order. __post_init__ __post_init__() -> None Reject ragged, non-numeric, boolean, or non-finite vector data. Raises: Type Description ValueError If values is empty, contains a bool (which is a int subclass in Python and would otherwise pass a numeric check silently), or contains a non-finite float (NaN or infinity). __len__ __len__() -> int Number of dimensions. anyinfer.EmbeddingSpace dataclass EmbeddingSpace( provider_id: str, model: str, model_revision: str | None = None, dimensions: int | None = None, input_intent_aware: bool = False, normalized: bool | None = None, compatibility_id: str | None = None, ) Identity of the vector space one embedding call's output lives in. Two embedding results are only safely comparable — for storage, search, or a fallback retry — when they came from the same space. This is the strongest identity AnyInfer can construct from what a provider tells it; it is deliberately not a guess when a provider tells us too little. Attributes: Name Type Description provider_id str The provider that produced the vectors. model str The concrete model id, verbatim as resolved (not an alias). model_revision str | None A pinned revision/snapshot identifier, when the provider or catalog exposes one; None when the model string is the only version signal available. dimensions int | None The vector length actually returned. input_intent_aware bool Whether this model's output depends on the requested EmbeddingInputIntent — if True, a query embedded without the same intent handling as the documents it will be compared against is not safely comparable even within the same model. normalized bool | None Whether the provider states its vectors are unit-normalized; None when undocumented. compatibility_id str | None An application-supplied identifier asserting that this space is interchangeable with another sharing the same id. Never inferred by AnyInfer — a guessed equivalence is exactly what this type exists to refuse to produce. None means no compatibility claim has been made. compatible_with compatible_with(other: EmbeddingSpace) -> bool Whether a vector from other may be safely compared with one from this space. True only when both spaces carry the same caller-asserted compatibility_id, or when provider, model, and revision are all identical. Matching dimensions alone is not sufficient — two different models can share a dimension count while encoding semantically incompatible spaces. anyinfer.EmbeddingCapabilities dataclass EmbeddingCapabilities( dimensions: int | None = None, dimension_choices: tuple[int, ...] = (), max_batch_inputs: int | None = None, max_input_tokens: int | None = None, max_input_bytes: int | None = None, input_intents: tuple[EmbeddingInputIntent, ...] = (), normalized: bool | None = None, ) What an embedding-capable model can do, sourced only from facts a provider states. Every field mirrors the provenance discipline of ModelCapabilities: a field this provider does not document stays None rather than becoming a guess. Attributes: Name Type Description dimensions int | None The vector length this model produces, when fixed. dimension_choices tuple[int, ...] Alternative dimensions this model supports via provider-native dimensionality reduction (e.g. Matryoshka-trained models), when documented. max_batch_inputs int | None Most inputs one request may carry, when the provider states a limit. max_input_tokens int | None Largest single input this model accepts, in tokens. max_input_bytes int | None Largest single input this model accepts, in bytes, when the provider bounds by bytes rather than tokens. input_intents tuple[EmbeddingInputIntent, ...] Which EmbeddingInputIntent values this model distinguishes; empty when the model does not support the concept at all. normalized bool | None Whether output vectors are unit-normalized; None when undocumented. overlay overlay( other: EmbeddingCapabilities, ) -> EmbeddingCapabilities Layer other's known fields over this one; unknown never displaces known. The assembly rule for these records mirrors ModelCapabilities.overlay without per-field provenance: they carry only provider-stated facts, so the later layer (config or catalog over static) wins wherever it actually says something. anyinfer.EmbeddingInputIntent module-attribute EmbeddingInputIntent = Literal[ "query", "document", "classification", "clustering" ] What an embedded text will be used for, when the provider or model distinguishes it. Several embedding models produce measurably better retrieval when a query and the documents it will be compared against are embedded with different instructions even though both pass through the same model. A provider that has no such distinction ignores this field entirely; one that requires it but received None degrades per its own documented default, recorded as a warning rather than silently substituted. Reranking anyinfer.RerankRequest dataclass RerankRequest( query: str, documents: tuple[RerankDocument, ...], top_n: int | None = None, timeout_s: float | None = None, max_response_bytes: int = DEFAULT_MAX_RERANK_RESPONSE_BYTES, metadata: Mapping[str, str] = dict(), provider_options: Mapping[ str, Mapping[str, Any] ] = dict(), batch: BatchPolicy = BatchPolicy(), return_documents: bool = False, retain_raw: bool = False, ) A request to rank documents by relevance to a query. Attributes: Name Type Description query str The query text every document is scored against. documents tuple[RerankDocument, ...] Documents to rank, in the caller's original order. Document ids must be unique within the request; duplicate text under distinct ids is permitted and preserved. top_n int | None Return only the top N ranked items. None returns every document ranked. timeout_s float | None Per-attempt wall-clock budget; None means DEFAULT_TIMEOUT_S. max_response_bytes int Hard cap on one provider response body. metadata Mapping[str, str] Caller-supplied, opaque request metadata carried through telemetry. provider_options Mapping[str, Mapping[str, Any]] Escape hatch, namespaced by provider id. batch BatchPolicy Core-owned batching policy for this request. return_documents bool Whether the result should echo document text back on each RankedItem. Off by default — the caller already has the text it sent. retain_raw bool Whether to keep the provider's raw response payload on the result. effective_timeout_s property effective_timeout_s: float The timeout that applies, honoring the module default when unset. __post_init__ __post_init__() -> None Reject a request with an empty query, no documents, or duplicate document ids. Raises: Type Description ValueError If query is blank, documents is empty, top_n is not positive when set, or two documents share an id. anyinfer.RerankResult dataclass RerankResult( items: tuple[RankedItem, ...], target: ResolvedTarget, usage: Usage, timing: Timing, attempts: tuple[AttemptRecord, ...] = (), warnings: tuple[str, ...] = (), raw: Any | None = None, manifest: Any | None = None, ) The result of one rerank request. Attributes: Name Type Description items tuple[RankedItem, ...] Ranked documents, ordered by descending relevance (or by whatever order the provider certifies as its ranking — always the intended reading order). target ResolvedTarget The provider and model that actually produced this ranking. usage Usage Token/billing accounting, normalized across providers. timing Timing Centrally-measured latency for the winning attempt. attempts tuple[AttemptRecord, ...] The full routing trail, including failed and retried attempts. warnings tuple[str, ...] Non-fatal notices accumulated along the way. raw Any | None The provider-native response payload, when the request asked to keep it. manifest Any | None The run manifest for this call, or None when manifests are off. anyinfer.RerankDocument dataclass RerankDocument( id: str, text: str, metadata: Mapping[str, str] = dict() ) One document offered to a rerank request. Attributes: Name Type Description id str Caller-owned opaque identifier, unique within one request. Never interpreted or generated by AnyInfer. text str The document text sent to the provider. metadata Mapping[str, str] Caller-owned metadata retained locally; never sent to a provider unless a provider option explicitly requests it. anyinfer.RankedItem dataclass RankedItem( index: int, document_id: str, score: float, text: str | None = None, ) One document's position and score in a rerank result. Attributes: Name Type Description index int The document's position in RerankRequest.documents, preserved so callers can recover order or metadata without a lookup. document_id str The caller-supplied id of the ranked document. score float The provider's relevance score. Finite by construction; meaningful only within the result produced by the same target and not comparable across different providers or models. text str | None The document's text, present only when the request asked for it. __post_init__ __post_init__() -> None Reject a non-finite score, a negative index, or a blank document id. Raises: Type Description ValueError If score is NaN/infinite, index is negative, or document_id is empty. anyinfer.RerankCapabilities dataclass RerankCapabilities( max_documents: int | None = None, max_tokens_per_document: int | None = None, max_bytes_per_document: int | None = None, native_top_n: bool = False, ) What a reranking-capable model can do, sourced only from facts a provider states. Attributes: Name Type Description max_documents int | None Most documents one request may carry, when the provider states a limit. max_tokens_per_document int | None Largest single document this model accepts, in tokens. max_bytes_per_document int | None Largest single document this model accepts, in bytes. native_top_n bool Whether the provider accepts RerankRequest.top_n natively rather than the core truncating a full ranking locally. overlay overlay(other: RerankCapabilities) -> RerankCapabilities Layer other's known fields over this one; unknown never displaces known. Shared anyinfer.InferenceOperation module-attribute InferenceOperation = Literal[ "generation", "embedding", "rerank" ] The inference operations AnyInfer models as first-class primitives. anyinfer.BatchPolicy dataclass BatchPolicy( max_concurrency: int = 4, allow_split: bool = True, rerank_cross_batch: bool = False, max_items_override: int | None = None, ) Core-owned policy for splitting a request that exceeds a provider's verified limit. Batching is centralized policy, not adapter behavior — providers disagree on maximum inputs, documents, tokens, and bytes, and an adapter never decides how to split a request; it only ever sees one already-sized wire call. Attributes: Name Type Description max_concurrency int Most internal batches dispatched concurrently for one request. allow_split bool Whether the core may split this request across multiple provider calls at all. False means a request exceeding the resolved limit fails locally rather than being silently divided. rerank_cross_batch bool Whether reranking may be split across documents when the provider offers no documented globally-comparable batch contract. Off by default because concatenating scores from separate rerank calls is not a valid global ordering unless the provider says otherwise. max_items_override int | None Caller-supplied ceiling on items per provider call — inputs for embedding, documents for reranking. Beats any provider-declared limit; useful when a provider misbehaves below its documented maximum, or to enable splitting against a target with no verified limit. None defers to the resolved target's verified capability. __post_init__ __post_init__() -> None Reject a non-positive concurrency bound or item ceiling. Raises: Type Description ValueError If max_concurrency or max_items_override is less than 1. --- # Reference / Portability Diff Tool Source: https://anyinfer.dev/reference/api/compare-diff/ Portability Diff Tool anyinfer.compare_diff: snapshot compare() output for a fixture set, and diff two snapshots structurally. No ranking, scoring, or live provider calls; every function here either calls compare() (itself no-dispatch) or works on plain JSON. See the portability guide for the full walkthrough and the fixture schema. from anyinfer import compare_diff anyinfer.compare_diff.load_fixtures load_fixtures(path: str | Path) -> tuple[Fixture, ...] Parse and validate a fixture file. Raises: Type Description ConfigError The file is missing, not valid JSON, declares an unsupported schema_version, or a fixture entry is malformed. anyinfer.compare_diff.snapshot snapshot( fixtures: Sequence[Fixture], *, client: Client ) -> dict[str, Any] Run compare() over every fixture and serialize the results. No new result data model — this only persists TargetComparison.to_dict()'s existing shape, keyed by fixture id and then by target string, so a diff can address any single (fixture, target) pair directly. Parameters: Name Type Description Default fixtures Sequence[Fixture] Fixtures to snapshot, typically from load_fixtures(). required client Client A configured anyinfer.Client — its provider settings determine which targets actually resolve. required Returns: Type Description dict[str, Any] A JSON-safe mapping: {"schema_version": ..., "fixtures": {id: {target: {...}}}}. anyinfer.compare_diff.diff diff( baseline: Mapping[str, Any], current: Mapping[str, Any] ) -> DiffReport Structurally diff two snapshots produced by snapshot(). Raises: Type Description ConfigError Either mapping is not a valid snapshot document. anyinfer.compare_diff.diff_targets diff_targets( fixture: Fixture, target_a: str, target_b: str, *, client: Client, ) -> DiffReport The ad hoc, no-baseline-file "should I move from A to B" report. Runs compare() live for exactly [target_a, target_b] under fixture and diffs the two resulting comparisons directly — the customer-facing portability report. anyinfer.compare_diff.render_text render_text(report: DiffReport) -> str Human-readable rendering of a DiffReport, one line per entry. anyinfer.compare_diff.Fixture dataclass Fixture( id: str, request: GenerationRequest, targets: tuple[str, ...], ) One request to snapshot against one ordered set of targets. Attributes: Name Type Description id str Stable identifier — the key snapshots and diffs are organized by. Renaming a fixture's id is a breaking change to any baseline snapshot that references it, the same way renaming a test would be. request GenerationRequest The request to compare, already resolved to a GenerationRequest. targets tuple[str, ...] Target strings to compare request against, in the order results are reported (never reordered — compare()'s own ordering discipline). anyinfer.compare_diff.DiffReport dataclass DiffReport(entries: tuple[DiffEntry, ...]) The full result of diffing two snapshots (or two live comparisons). Attributes: Name Type Description entries tuple[DiffEntry, ...] Every difference found, in a stable order (fixture, then target, then field) — never ranked or filtered by significance. is_empty property is_empty: bool Whether the two snapshots reported were identical. anyinfer.compare_diff.DiffEntry dataclass DiffEntry( fixture_id: str, target: str, kind: str, field: str, before: Any, after: Any, summary: str, ) One reported difference between two snapshots. Attributes: Name Type Description fixture_id str Which fixture this entry belongs to. target str Which target this entry belongs to. kind str "added" (present only in the newer snapshot), "removed" (present only in the baseline), or "changed" (present in both with a different value). field str Dotted path within TargetComparison.to_dict()'s shape, e.g. "structured_mechanism" or "dropped.0.name". before Any The baseline value, or None when kind is "added". after Any The current value, or None when kind is "removed". summary str A plain-language line reusing compare()'s own field vocabulary. anyinfer.compare_diff.FIXTURE_SCHEMA_VERSION module-attribute FIXTURE_SCHEMA_VERSION = 1 The fixture file format this module reads and writes. Additive, never breaking: a future version adds fields or fixture kinds, it never repurposes an existing key. --- # Reference / Vector Store Add-On Source: https://anyinfer.dev/reference/api/vector-store/ Vector Store Add-On anyinfer_store: a small-scale, single-process, embedded vector store; a separate installable distribution, never imported by anyinfer core and never a dependency of it. See the vector store guide for the full walkthrough and the explicit, permanent scale boundary this package commits to. from anyinfer_store import VectorStore, query_and_rerank anyinfer_store.VectorStore VectorStore(connection: Connection) A single SQLite file holding one embedding space's worth of vectors. Every entry in one store must share a compatible anyinfer.EmbeddingSpace — checked with EmbeddingSpace.compatible_with, the identical rule anyinfer core's own routing applies for a fallback target — so a query can never silently compare vectors that were never comparable to begin with. The space is fixed the first time an entry is added and stored permanently in the file; open the same file again later and it's already there, no re-declaration needed. space property space: EmbeddingSpace | None The embedding space this store is bound to, or None if still empty. open classmethod open(path: str | Path) -> VectorStore Open (creating if needed) a store at path. close close() -> None Close the underlying SQLite connection. __enter__ __enter__() -> VectorStore Support with VectorStore.open(...) as store:. __exit__ __exit__(*exc_info: object) -> None Close the connection on context-manager exit. add add( entry_id: str, vector: Sequence[float], *, space: EmbeddingSpace, metadata: Mapping[str, str] | None = None, text: str | None = None, ) -> None Insert or replace one vector. Raises: Type Description EmbeddingSpaceMismatchError space is not compatible with this store's already-bound space (see EmbeddingSpace.compatible_with). add_many add_many( entries: Sequence[VectorEntry], *, space: EmbeddingSpace ) -> None Insert or replace several vectors in one transaction. remove remove(entry_id: str) -> None Delete one entry; a no-op if it does not exist. get get(entry_id: str) -> VectorEntry | None Look up one entry by id, or None. count count() -> int How many entries this store holds. query query( vector: Sequence[float], *, space: EmbeddingSpace, top_k: int = 10, metadata_filter: Mapping[str, str] | None = None, ) -> list[QueryResult] Brute-force top-k cosine similarity search. Raises: Type Description EmbeddingSpaceMismatchError space is not compatible with this store's bound space. VectorStoreError The store is empty (no space bound yet). rebuild_index rebuild_index() -> None No-op for the brute-force backend; present so a future approximate-index backend can share this interface without a caller-visible change. compact compact() -> None Reclaim disk space after deletions (VACUUM). export_jsonl export_jsonl(path: str | Path) -> None Write every entry, one JSON object per line, plus a header line with the space. import_jsonl import_jsonl(path: str | Path) -> None Load entries previously written by export_jsonl, into this (possibly already-open, possibly empty) store. anyinfer_store.VectorEntry dataclass VectorEntry( id: str, vector: tuple[float, ...], metadata: Mapping[str, str] = dict(), text: str | None = None, ) One stored vector. Attributes: Name Type Description id str Caller-supplied identifier, unique within one store. vector tuple[float, ...] The embedding vector's components. metadata Mapping[str, str] Small caller-supplied key/value payload, exact-match filterable. text str | None The source text, when the caller chose to keep it — needed for a second-stage rerank pass (anyinfer_store.query_and_rerank), optional otherwise. anyinfer_store.QueryResult dataclass QueryResult(entry: VectorEntry, score: float) One ranked match from VectorStore.query. Attributes: Name Type Description entry VectorEntry The matched entry. score float Cosine similarity to the query vector, in [-1.0, 1.0]. Meaningful only within this store — never compared across stores or embedding spaces. anyinfer_store.query_and_rerank async query_and_rerank( store: VectorStore, query_vector: list[float], query_text: str, *, space: EmbeddingSpace, client: AsyncClient, rerank_target: str, candidate_k: int = 20, top_n: int | None = None, ) -> tuple[RankedItem, ...] Coarse vector search, then a real rerank pass over its candidates. Parameters: Name Type Description Default store VectorStore The store to search. required query_vector list[float] The query's embedding, in space. required query_text str The query's original text — reranking scores text, not vectors. required space EmbeddingSpace The embedding space query_vector was produced in; must be compatible with store's bound space (see VectorStore.query). required client AsyncClient An anyinfer.AsyncClient to dispatch the rerank call through. required rerank_target str The rerank target, e.g. "cohere:rerank-v3.5". required candidate_k int How many coarse vector matches to hand to the reranker. 20 top_n int | None Passed through to AsyncClient.rerank; None returns every candidate, reordered. None Raises: Type Description VectorStoreError A candidate has no stored text — reranking needs text, and this package never invents it. anyinfer_store.SIZE_WARNING_THRESHOLD module-attribute SIZE_WARNING_THRESHOLD = 200000 Entry count past which VectorStore.add/add_many warn that brute-force search may be noticeably slow — a signal, not a hard limit; nothing stops working at this count. anyinfer_store.VectorStoreError Bases: Exception Base class for every error this package raises. anyinfer_store.EmbeddingSpaceMismatchError Bases: VectorStoreError A vector was added or queried against a store bound to a different embedding space. The same cross-space safety rule anyinfer's own routing applies, extended to persistence: a wrong-but-plausible vector comparison fails loudly rather than returning a confident-looking, meaningless result. --- # Reference / Routing Source: https://anyinfer.dev/reference/api/routing/ Routing Retry, fallback, and target resolution: the policy layer that adapters are forbidden from containing. Behavior is described in routing. anyinfer.Route dataclass Route( targets: tuple[Target, ...], retry: Retry = Retry(), health_gate: bool = True, health_ttl_s: float = 30.0, context_window_targets: tuple[Target, ...] = (), content_policy_targets: tuple[Target, ...] = (), ) An ordered fallback chain and the policy applied to it. Beyond the general chain, two failure classes get their own chains because the right next target differs by why the first one failed: a prompt that overflowed one model's context needs a larger model, not another same-sized one, and a content-policy refusal needs a differently-governed provider, not a retry. Attributes: Name Type Description targets tuple[Target, ...] Targets to try in order. retry Retry Retry policy applied per target. health_gate bool Skip targets whose health probe recently failed. health_ttl_s float How long a health failure suppresses a target. context_window_targets tuple[Target, ...] Chain used after a ContextLengthError. Empty means "use targets". content_policy_targets tuple[Target, ...] Chain used after a content-filter refusal. Empty means "use targets". of classmethod of(*targets: Target, retry: Retry | None = None) -> Route Build a route from positional targets. specialized_chain_for specialized_chain_for( error: ProviderError, ) -> tuple[Target, ...] The fallback chain that fits this failure, or () for the general one. coerce classmethod coerce(value: Route | Target | Sequence[Target]) -> Route Accept a route, a single target string, or a sequence of targets. anyinfer.Retry dataclass Retry( max_attempts: int = 2, backoff_base_s: float = 0.5, backoff_max_s: float = 30.0, retry_on: Callable[[ProviderError], bool] | None = None, ) Per-target retry policy. Attributes: Name Type Description max_attempts int Total attempts per target, including the first. backoff_base_s float Base for exponential backoff. backoff_max_s float Ceiling for any single delay. retry_on Callable[[ProviderError], bool] | None Overrides the default predicate. The default declines deterministic failures (auth, context length) and otherwise follows error.retryable. should_retry should_retry(error: ProviderError) -> bool Whether error is worth retrying under this policy. anyinfer.ResolvedTarget dataclass ResolvedTarget( provider_id: str, model: str, via_alias: str | None = None, ) A target after alias and provider-alias resolution. Attributes: Name Type Description provider_id str Normalized id of the provider that will serve the request. model str The model identifier, verbatim as the provider expects it. via_alias str | None The catalog alias this target was resolved from, if any. __str__ __str__() -> str Render as the canonical provider:model spelling. --- # Reference / Capabilities Source: https://anyinfer.dev/reference/api/capabilities/ Capabilities Provenance-tagged model metadata: every value knows whether it was cataloged, discovered, probed, or defaulted. The reasoning is in capabilities and provenance; token estimation and the budget calculator are explained in token estimation and context budgets. anyinfer.ModelCapabilities dataclass ModelCapabilities( context_window: Sourced[int] | None = None, max_output_tokens: Sourced[int] | None = None, features: Sourced[Feature] = Sourced( Feature(0), "default" ), pricing: Sourced[Pricing] | None = None, default_temperature: Sourced[float] | None = None, default_top_p: Sourced[float] | None = None, local: LocalModelInfo | None = None, operations: Sourced[frozenset[InferenceOperation]] | None = None, embedding: EmbeddingCapabilities | None = None, ) What a model can do, as far as we know. Attributes: Name Type Description context_window Sourced[int] | None Maximum tokens of input context, with provenance; None when unknown. max_output_tokens Sourced[int] | None Maximum tokens one response may contain, with provenance; None when unknown. features Sourced[Feature] Which Feature flags the model supports, with provenance. pricing Sourced[Pricing] | None Per-million-token pricing, when known. default_temperature Sourced[float] | None The temperature this provider applies when a request sends none, with provenance; None when the provider does not document one. default_top_p Sourced[float] | None The nucleus-sampling cutoff this provider applies when a request sends none, with provenance; None when undocumented. local LocalModelInfo | None Facts about the local artifact, for locally-run models only. operations Sourced[frozenset[InferenceOperation]] | None Which inference operations this model serves, with provenance; None when unknown. Deliberately not a generation Feature flag — "can embed" is a different operation, not a feature of chat — and deliberately not invented from the provider-level operations set, which says what the adapter speaks, not what one model does. embedding EmbeddingCapabilities | None Vector facts this model states about itself — the dimensions and input ceiling a listing or a pinned catalog row declares. None when nothing model-level is known, which is the honest state for most providers: it is filled from what a model actually reports, never from the provider's documentation about a different model. The two sampling defaults exist so an application can say what "provider default" means instead of only that it is one. They are populated from a provider's own documentation and nowhere else; never probed, never inferred from a sibling provider, never carried over from a model family. A provider whose documentation states no default keeps None indefinitely, and that is the correct final state for it rather than a gap waiting to be filled: an invented number presented beside a provenance tag is precisely the estimate-as-authority this type exists to prevent. overlay overlay(other: ModelCapabilities) -> ModelCapabilities Layer other on top of this, field by field, stronger provenance winning. This is the assembly rule: later layers override earlier ones, but a weaker-provenance value never displaces a stronger one. anyinfer.Pricing dataclass Pricing( input_per_1m: Decimal, output_per_1m: Decimal, cache_read_per_1m: Decimal | None = None, cache_write_per_1m: Decimal | None = None, currency: str = "USD", per_search_unit: Decimal | None = None, ) Per-million-token pricing used to compute cost_usd. Cache rates are optional and default to unknown rather than to the input rate. A provider that discounts cached prompt tokens but whose discount we have not recorded must not be billed as though the discount were zero or as though it were free — an unknown rate leaves cached tokens priced as ordinary input, which is the same answer this library gave before cache accounting existed, and is wrong in only one direction that a caller can reason about. Attributes: Name Type Description input_per_1m Decimal Price per one million prompt tokens. output_per_1m Decimal Price per one million generated tokens. cache_read_per_1m Decimal | None Price per one million prompt tokens served from the provider's cache, or None when the rate is not recorded. cache_write_per_1m Decimal | None Price per one million prompt tokens written into the cache, or None when the rate is not recorded. Several providers charge a premium for a write, so this is not assumed to be a discount. currency str Currency code the prices are quoted in. per_search_unit Decimal | None Price per one billed search unit, for rerank providers that bill by searches rather than tokens. None when the provider does not bill this way or the rate is not recorded — a rerank cost stays unknown rather than being priced through an invented token equivalence. anyinfer.Feature Bases: Flag Capabilities a model may support. Structured-output mechanism selection reads these in the order GRAMMAR > JSON_SCHEMA > JSON_MODE > prompt injection. CACHE_USAGE and CACHE_PLACEMENT are deliberately separate facts: reporting what the prompt cache did is not the same as accepting instructions about where it should apply, and a provider may do either without the other. anyinfer.Mechanism module-attribute Mechanism = Literal[ "grammar", "json_schema", "json_mode", "prompt" ] How structured output was requested of the provider. anyinfer.Sourced dataclass Sourced(value: _T, provenance: Provenance = 'default') Bases: Generic[_T] A capability value paired with its provenance. outranks outranks(other: Sourced[_T] | None) -> bool Whether this value's provenance is at least as strong as other's. anyinfer.Provenance module-attribute Provenance = Literal[ "catalog", "discovered", "probed", "default", "override" ] Where a capability value came from, weakest (default) to strongest (override). anyinfer.Health dataclass Health(ok: bool, detail: str = '') Result of a provider's cheap readiness probe. Attributes: Name Type Description ok bool Whether the provider answered its readiness probe successfully. detail str Short human-readable explanation, most useful when ok is false. anyinfer.DiscoveredModel dataclass DiscoveredModel( id: str, capabilities: ModelCapabilities | None = None ) A model reported by a provider's listing endpoint. capabilities carries only fields the provider actually reported; the capability assembler tags them "discovered". Attributes: Name Type Description id str The model identifier exactly as the provider lists it. capabilities ModelCapabilities | None Capability fields the listing reported; None when the provider lists ids only. anyinfer.LocalModelInfo dataclass LocalModelInfo( artifact_size_bytes: int | None = None, parameter_size: str | None = None, quantization: str | None = None, est_ram_bytes: int | None = None, est_vram_bytes: int | None = None, observed_vram_bytes: int | None = None, ) Facts about a local model artifact, used for tuning and recommendation. Attributes: Name Type Description artifact_size_bytes int | None On-disk size of the model weights. parameter_size str | None Parameter count as the runtime reports it (e.g. "7B"). quantization str | None Quantization scheme of the artifact (e.g. "Q4_K_M"). est_ram_bytes int | None Estimated system memory needed to run the model. est_vram_bytes int | None Estimated GPU memory needed to run the model. observed_vram_bytes int | None GPU memory actually measured in use while the model was loaded, when the runtime reports it. anyinfer.ContextBudget dataclass ContextBudget( context_window: Sourced[int] | None, estimate: RequestEstimate, output_reserve_tokens: int, headroom_tokens: int, pricing: Sourced[Pricing] | None = None, ) A request's estimated size held against a model's known capacity. The verdict is tri-state: fits is True/False when the context window is known, and None when it is not — an unknown capacity is reported as unknown, never guessed. Attributes: Name Type Description context_window Sourced[int] | None The model's context window with its provenance, or None when nothing trustworthy is known. estimate RequestEstimate The per-component input-token estimate. output_reserve_tokens int Tokens reserved for the response. headroom_tokens int Safety margin against estimation error. pricing Sourced[Pricing] | None The model's per-token rates with their provenance, when known. input_allowance_tokens property input_allowance_tokens: int | None Tokens the input may spend, or None when the window is unknown. remaining_tokens property remaining_tokens: int | None Allowance left after the estimated input; negative when over budget. This is the number an app packs context against: keep adding material while it stays positive. fits property fits: bool | None Whether the estimated request fits the allowance; None when unknowable. estimated_cost property estimated_cost: CostEstimate | None A preflight cost range, or None when no trustworthy pricing exists. Estimated money never mixes with reported money: cost_usd is only ever computed from provider-reported usage, and this range is only ever computed from the estimate. anyinfer.TokenEstimate dataclass TokenEstimate(tokens: int, floor: int) A token count carried as a planning estimate and a defensible lower bound. Attributes: Name Type Description tokens int The planning figure, deliberately conservative-high. floor int A lower bound the true count is not realistically below. An exact tokenizer sets floor == tokens. __add__ __add__(other: TokenEstimate) -> TokenEstimate Sum two estimates component-wise. anyinfer.TokenEstimator Bases: Protocol Pluggable token counting. Implementations may be heuristic (the shipped default) or exact (tiktoken, a provider's tokenize endpoint). Exact implementations should return TokenEstimate(n, n) so the gate can act on their counts with full force. estimate estimate(text: str) -> TokenEstimate Estimate the token count of text. anyinfer.HeuristicTokenEstimator dataclass HeuristicTokenEstimator(multiplier: float = 1.0) The dependency-free default: token counts from UTF-8 byte counts. Attributes: Name Type Description multiplier float Calibration factor applied to the planning estimate, for providers whose transport envelope inflates reported prompt tokens beyond the serialized bytes. The floor is never inflated — envelope overhead is not something a lower bound may claim. __post_init__ __post_init__() -> None Reject non-finite or non-positive calibration factors. estimate estimate(text: str) -> TokenEstimate Estimate tokens as ceil(bytes/3), with a bytes//8 floor. anyinfer.RequestEstimate dataclass RequestEstimate( messages: TokenEstimate, tools: TokenEstimate, schema: TokenEstimate, envelope: TokenEstimate = TokenEstimate(0, 0), unpriced_parts: int = 0, ) Content-free size accounting for one request, by component. The breakdown follows the typed request itself: what the caller said, what tools were offered, and what schema was attached — the three things that occupy input tokens on any provider. Attributes: Name Type Description messages TokenEstimate The conversation, including per-message wire-framing overhead. tools TokenEstimate Serialized tool specifications, when any were offered. schema TokenEstimate The structured-output schema, when one was requested. Counted whether the wire carries it natively or the core injects it into the prompt — either way it occupies input tokens. envelope TokenEstimate What the provider's own transport adds around all of the above, from its declared TokenCalibration. Zero for every provider that counts what it was sent, and floor-free always: an envelope correction is believed, not proven. tokens property tokens: int Total planning estimate across all components. floor property floor: int Total lower bound across all components. anyinfer.TokenCalibration dataclass TokenCalibration( multiplier: float = 1.0, overhead_tokens: int = 0 ) How much a provider's own envelope inflates the prompt it is sent. Serialized request bytes are not what every provider counts. Some wrap the caller's messages in a transport of their own before the model ever sees them — a session API that prepends its harness, a tool scaffold, a service-side system preamble, and then bill (and window-check) the inflated total. Estimating such a provider from message bytes alone under-counts every request, and the under-count is systematic rather than noise, so budgets stay optimistic right up to the overflow. A provider therefore declares its own correction, and only the planning figure moves: multiplier scales content that grows with the prompt. overhead_tokens adds what the envelope costs regardless of prompt size. Neither touches the estimate's floor. The floor exists to refuse requests before dispatch, and a lower bound may only claim tokens the provider certainly charges — envelope overhead is a correction we believe, not one we can prove. Attributes: Name Type Description multiplier float Factor applied to the planning estimate of prompt-proportional content. 1.0 means the provider counts what was sent. overhead_tokens int Flat tokens the envelope adds per request, counted once. is_identity property is_identity: bool Whether this calibration leaves an estimate unchanged. __post_init__ __post_init__() -> None Reject calibrations that would corrupt every estimate downstream. Raises: Type Description ValueError If the multiplier is not a positive finite number, or the overhead is negative. anyinfer.build_context_budget build_context_budget( request: GenerationRequest, capabilities: ModelCapabilities | None, *, estimator: TokenEstimator | None = None, calibration: TokenCalibration | None = None, output_reserve_tokens: int | None = None, headroom_tokens: int | None = None, ) -> ContextBudget Compute the context budget for one request against one model's capabilities. Parameters: Name Type Description Default request GenerationRequest The request to size. required capabilities ModelCapabilities | None Assembled capabilities supplying the context window and maximum output size. None means nothing is known — the budget stays tri-state. required estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None calibration TokenCalibration | None The target provider's declared envelope correction, from its descriptor. None means the provider counts what it was sent. None output_reserve_tokens int | None Overrides the derived output reserve. None headroom_tokens int | None Overrides the default clamped headroom. None Returns: Type Description ContextBudget The computed ContextBudget. Raises: Type Description ValueError If an explicit reserve or headroom is negative. anyinfer.estimate_request estimate_request( request: GenerationRequest, *, estimator: TokenEstimator | None = None, calibration: TokenCalibration | None = None, ) -> RequestEstimate Estimate the input tokens a request will occupy. Derived from the typed request rather than hand-fed strings: messages (every content part, plus per-message framing overhead), offered tools, and the schema. Parameters: Name Type Description Default request GenerationRequest The request to size. required estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None calibration TokenCalibration | None The target provider's declared envelope correction. Applied to the planning figure only, and reported as its own component so the breakdown still adds up. None means the identity. None Returns: Type Description RequestEstimate The per-component estimate. anyinfer.check_context_fit check_context_fit( request: GenerationRequest, capabilities: ModelCapabilities | None, *, estimator: TokenEstimator | None = None, calibration: TokenCalibration | None = None, output_reserve_tokens: int | None = None, provider: str | None = None, model: str | None = None, ) -> ContextBudget Build the budget for a request and raise if it provably cannot fit. Parameters: Name Type Description Default request GenerationRequest The GenerationRequest to size. required capabilities ModelCapabilities | None The target's assembled capabilities. required estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None calibration TokenCalibration | None The target provider's declared envelope correction. It never affects the gate's decision — the gate reads the floor, which no calibration moves — but it keeps the returned budget consistent with the one budget() reports. None output_reserve_tokens int | None Overrides the derived output reserve. None provider str | None Provider id, for the error's structured fields. None model str | None Model id, for the error message. None Returns: Type Description ContextBudget The computed ContextBudget when the ContextBudget request may proceed. Raises: Type Description ContextLengthError When the estimate's floor exceeds a trusted-provenance context window. anyinfer.ProbeReport dataclass ProbeReport( target: ResolvedTarget, probes: tuple[FeatureProbe, ...] = (), capabilities: ModelCapabilities | None = None, requests: int = 0, usage: Usage = Usage(), ) Everything one probing run learned, and what it cost. Attributes: Name Type Description target ResolvedTarget The target that was probed. probes tuple[FeatureProbe, ...] One result per feature tested, in the order tested. capabilities ModelCapabilities | None What to record at probed provenance, or None when nothing was settled. Already merged with what was known, since a feature flag is one value and a probe that clears a bit must not clear the others with it. requests int Round trips spent. usage Usage Tokens spent, summed across the probes. summary property summary: str One line naming what was settled. outcome_for outcome_for(feature: Feature) -> ProbeOutcome | None What was established for one feature, or None if it was not tested. anyinfer.FeatureProbe dataclass FeatureProbe( feature: Feature, outcome: ProbeOutcome, detail: str = "", ) The result of testing one feature against one target. Attributes: Name Type Description feature Feature The feature that was tested. outcome ProbeOutcome What the attempt established. detail str One sentence explaining the outcome — the provider's rejection, or what came back instead of what was asked for. conclusive property conclusive: bool Whether this probe settled anything worth recording. anyinfer.ProbeOutcome module-attribute ProbeOutcome = Literal[ "supported", "unsupported", "inconclusive" ] What one probe established, if anything. anyinfer.PROBEABLE_FEATURES module-attribute PROBEABLE_FEATURES: tuple[Feature, ...] = ( Feature.JSON_SCHEMA, Feature.JSON_MODE, Feature.GRAMMAR, Feature.TOOLS, Feature.STREAMING, ) Every feature a probe can settle. Absent by design: REASONING (providers overwhelmingly accept the field and ignore it, so a probe would return inconclusive nearly always), SYSTEM_PROMPT and CACHE_USAGE (no single reply distinguishes honored from ignored), and the numeric bounds — finding a context window by bisection would cost dozens of requests to learn what one catalog entry already says. anyinfer.DEFAULT_PROBE_FEATURES module-attribute DEFAULT_PROBE_FEATURES: tuple[Feature, ...] = ( Feature.JSON_SCHEMA, Feature.JSON_MODE, Feature.TOOLS, Feature.STREAMING, ) What probe() tests when the caller names nothing: the four an OpenAI-compatible endpoint most often misreports. Four requests. GRAMMAR is excluded because the engines that have it declare it accurately, so paying a request to confirm buys nothing. anyinfer.CostEstimate dataclass CostEstimate( low: Decimal, high: Decimal, currency: str = "USD" ) A preflight cost range for one request. Deliberately a range, never one number: the input estimate is two-sided (anyinfer.capabilities.estimate) and the output spend is unknown until the model stops. Kept strictly separate from cost_usd, which is only ever computed from reported usage — estimated and actual money must never be indistinguishable. Attributes: Name Type Description low Decimal Floor input tokens priced, with zero output — the least this can cost. high Decimal Planning-estimate input plus the full output reserve priced — a spend ceiling under the budget's own assumptions. currency str The pricing currency. anyinfer.PricingTable PricingTable(entries: dict[str, tuple[PricingEntry, ...]]) Per-provider model pricing with prefix-aware lookup. providers property providers: tuple[str, ...] Provider ids the table covers, sorted. entries_for entries_for(provider_id: str) -> tuple[PricingEntry, ...] Every entry for one provider, or an empty tuple. lookup lookup( provider_id: str, model: str ) -> Sourced[Pricing] | None Find pricing for a model: exact match first, then longest boundary prefix. Returns: Type Description Sourced[Pricing] | None The pricing tagged catalog provenance, or None when the model has no Sourced[Pricing] | None entry; never a fallback price. from_mapping classmethod from_mapping(data: Any) -> PricingTable Build and validate a table from parsed JSON. Raises: Type Description ConfigError On a malformed document — wrong format version, missing fields, or prices that do not parse as non-negative decimals. anyinfer.load_default_pricing cached load_default_pricing() -> PricingTable Load the pricing table bundled with this release. anyinfer.fetch_pricing fetch_pricing( url: str = DEFAULT_PRICING_URL, *, timeout_s: float = 30.0, transport: Any | None = None, ) -> PricingTable Fetch a maintained pricing table over HTTPS — the explicit freshness opt-in. Nothing in the library calls this implicitly. An application that wants prices newer than its installed release calls it on its own schedule and passes the result to the client's pricing_table. Parameters: Name Type Description Default url str Where to fetch from; defaults to the repo's continuously-updated file. DEFAULT_PRICING_URL timeout_s float Request timeout. 30.0 transport Any | None Test seam — an httpx2 transport. None Returns: Type Description PricingTable The fetched, validated table. Raises: Type Description ConfigError If the fetch fails or the response is not a valid pricing document. Spend Accounting An in-process rollup of what a client spent, and an optional ceiling checked before dispatch. Concepts: cost and spending. anyinfer.SpendLedger SpendLedger(currency: str = 'USD') A thread-safe rollup of one client's observed spend. Subscribe it like any other observer:: ledger = SpendLedger() client = ai.Client(providers, observers=[ledger]) ... print(ledger.totals().cost) Two clients that should share a total are given the same ledger. There is deliberately no process-wide instance: a global would make a total depend on import order, and would silently merge the accounting of two libraries that happen to share a process. Only completed requests are counted. A failed attempt that a retry replaced is not separately visible in the event stream, so its tokens are not in these totals — the figure is "what the successful requests cost", not "everything the provider might bill". Where that distinction matters, compare against the provider's own invoice. on_event on_event(event: TelemetryEvent) -> None Absorb one telemetry event. Fast and non-blocking, as the observer contract requires: this is arithmetic under a lock, with no I/O. record record( target: ResolvedTarget, usage: Usage, *, request_id: str | None = None, ) -> None Fold one completed request into the totals. Parameters: Name Type Description Default target ResolvedTarget What served the request. required usage Usage Its reported usage, with cost already computed by the core. required request_id str | None Correlation id, used to attribute the request to the labels its caller supplied. None Raises: Type Description ValueError If the usage carries a cost in a different currency than this ledger's. Converting would require a rate source, and a converted figure would have no provenance. reserve reserve( request_id: str, estimate: Decimal, ceiling: Decimal | None, ) -> tuple[bool, Decimal, Decimal] Atomically reserve a preflight estimate against a cumulative ceiling. Re-reserving the same request replaces its prior estimate, which lets fallback targets update the bound without double-counting one caller request. release release(request_id: str) -> None Release a reservation that will not be replaced by a completion event. reserved reserved() -> Decimal Total preflight spend currently reserved by in-flight requests. totals totals() -> SpendTotals Everything observed so far. by_target by_target() -> Mapping[str, SpendTotals] Totals per provider:model, in first-seen order. by_label by_label(key: str) -> Mapping[str, SpendTotals] Totals per value of one caller-supplied metadata label. The library never interprets these labels — a tenant id, a feature name, a job id are all the application's vocabulary, carried through untouched. reset reset() -> None Forget everything recorded so far. anyinfer.SpendTotals dataclass SpendTotals( cost: Decimal = Decimal(0), currency: str = "USD", requests: int = 0, unknown_requests: int = 0, input_tokens: int = 0, output_tokens: int = 0, cache_read_tokens: int = 0, ) What was spent, and what could not be priced. unknown_requests is the honest counterpart to cost. A provider whose pricing is absent or untrusted produces no cost at all; never a zero, so a total that reported only cost would quietly understate spend by however many calls it could not price. Both numbers travel together for that reason, and every rendering of one should render the other. Attributes: Name Type Description cost Decimal Summed cost of the requests that could be priced. currency str Currency the costs are in. A ledger refuses to mix currencies rather than converting, because a conversion needs a rate source this library must not have. requests int Completed requests observed. unknown_requests int Completed requests whose cost could not be known. input_tokens int Prompt tokens reported across those requests. output_tokens int Generated tokens reported across those requests. cache_read_tokens int Prompt tokens the providers reported serving from cache. complete property complete: bool Whether every observed request could be priced. plus plus(usage: Usage) -> SpendTotals Fold one request's usage into these totals. to_json to_json() -> dict[str, Any] Serialize for a SpendStore. from_json classmethod from_json(data: Mapping[str, Any]) -> SpendTotals | None Deserialize, returning None for anything unreadable. anyinfer.SpendStore SpendStore(path: Path | str) An optional, caller-owned file of accumulated spend. The library persists nothing on its own. An application that wants a total that survives a restart constructs one of these and points it somewhere — the same contract anyinfer.benchmark.MeasurementStore uses, including its most important property: reads are total. A missing, truncated, or foreign file yields nothing rather than raising, because a cache that can break a program is worse than no cache. path property path: Path Where this store reads and writes. load load() -> Mapping[str, SpendTotals] Every stored bucket, keyed by name. Unreadable content yields nothing. accumulate accumulate( ledger: SpendLedger, *, bucket: str = "total" ) -> None Add a ledger's current totals to the stored bucket, atomically. Parameters: Name Type Description Default ledger SpendLedger The ledger whose totals to fold in. required bucket str Which stored bucket to add to — a process name, a job id, or the default single bucket. 'total' Raises: Type Description ValueError If the ledger's currency differs from the stored bucket's. anyinfer.SpendPolicy dataclass SpendPolicy( max_total_usd: Decimal | None = None, max_request_usd: Decimal | None = None, on_unknown: Literal["allow", "refuse"] = "allow", ) A ceiling on what one client may spend. Off unless supplied. Checked before dispatch, beside the context gate, so a refusal costs nothing. This is the caller's own policy on their own client — it shares no state with any other process and enforces no organization quota, which this library deliberately leaves to the deployment around it. Attributes: Name Type Description max_total_usd Decimal | None Ceiling on this client's cumulative spend. None means no ceiling. max_request_usd Decimal | None Ceiling on any single request's estimated cost. on_unknown Literal['allow', 'refuse'] What to do when a target's cost cannot be estimated, because its pricing is missing or untrusted. allow preserves today's behaviour; refuse is for callers who would rather fail than spend blind. There is no third option that treats unknown as zero — a guard that does that enforces nothing while appearing to. active property active: bool Whether this policy can refuse anything. __post_init__ __post_init__() -> None Reject a ceiling that cannot be enforced. Raises: Type Description ValueError On a negative ceiling or an unknown on_unknown value. Rate Governance Client-side pacing for one provider instance, and the header dialect a provider reports its window in. Both are inert until configured. Concepts: routing and rate limits. anyinfer.RateLimits dataclass RateLimits( max_concurrent: int | None = None, requests_per_minute: float | None = None, min_interval_s: float = 0.0, respect_headers: bool = True, reserve_fraction: float = 0.0, ) Client-side pacing for one provider instance. Inert unless configured. This paces this process's own requests to one provider so an application that fans out does not discover the provider's limits by being throttled by them. It shares no state with any other process, enforces no quota the provider did not state, and never influences which target is chosen — a limiter that picked a different provider because this one was busy would be load balancing, which this library deliberately does not do. With every field left at its default, a request is dispatched exactly as it was before this existed: no permit, no delay, no bookkeeping. Attributes: Name Type Description max_concurrent int | None Most requests in flight at once for this instance. None means unbounded, which is today's behaviour. requests_per_minute float | None Sustained request rate. Enforced as a token bucket, so a burst up to the per-minute allowance is permitted and then paced. min_interval_s float Smallest gap between two dispatches, for providers that object to bursts regardless of the rate. respect_headers bool Whether to slow down when the provider's own rate-limit headers say its window is nearly exhausted. Inert when the provider declares no header dialect, since there is nothing to read. reserve_fraction float Fraction of the provider's stated remaining allowance to leave untouched, between 0 and 1. Matters whenever this process is not the only consumer of the key: stopping at the last request in the window means the other consumer is the one that gets throttled. active property active: bool Whether this policy can delay anything. A bare RateLimits() is active: it means "pace me by what the provider reports", which is the least a caller who asked for governance at all can mean. Opting out is spelled by supplying no limits, not by supplying empty ones — so RateLimits(respect_headers=False) with no bounds is the one inert instance, and it is inert honestly rather than by accident. __post_init__ __post_init__() -> None Reject a limit that cannot be honoured. Raises: Type Description ValueError On a non-positive bound, a negative interval, or a reserve fraction outside the unit interval. anyinfer.RateLimitHeaders dataclass RateLimitHeaders( requests_remaining: str = "", requests_reset: str = "", tokens_remaining: str = "", tokens_reset: str = "", limit_requests: str = "", limit_tokens: str = "", ) Which response headers a provider reports its rate-limit state in. Header names are wire facts and differ per provider, so they are declared on the descriptor and recorded in that provider's contract snapshot; never branched on by provider id in the core. A provider whose dialect cannot be verified from its documentation declares nothing. An empty dialect is not a failure: pacing falls back to whatever bounds the caller configured, which is a smaller promise honestly kept rather than a guessed header name that silently reads None forever. Attributes: Name Type Description requests_remaining str Requests left in the current window. requests_reset str When the request window resets. Read as seconds, or as a duration like "1m30s" for the providers that spell it that way. tokens_remaining str Tokens left in the current window. tokens_reset str When the token window resets, in the same two spellings. limit_requests str The window's full request allowance, when the provider states it. Only needed to turn reserve_fraction into an absolute floor. limit_tokens str The window's full token allowance, for the same reason. declared property declared: bool Whether this provider reports anything worth reading. --- # Reference / Context Reduction Source: https://anyinfer.dev/reference/api/context/ Context Reduction Fit a document corpus to a token budget. The application collects; this subpackage reduces. The reasoning and the strategy tradeoffs are in context reduction; the task-oriented walkthrough is fitting a corpus to a budget. Imported from its own path, like the other optional subsystems: from anyinfer import context Documents and Results anyinfer.context.ContextDocument dataclass ContextDocument( path: str, content: str, sha256: str, pinned: bool = False, language: str | None = None, extract: str = "", ) One document offered to the reducer. Attributes: Name Type Description path str POSIX-style relative path. Doubles as identity and as a ranking signal — a query term matching the path outweighs the same term in the body. content str The document's text. sha256 str Hex digest of content, used for identity in rendered envelopes and for deterministic tie-breaking. pinned bool Sorts before every unpinned document and is selected first. This is how an app says "the user explicitly chose this file". language str | None Language name for rendering and rollups, detected when omitted. extract str A structural summary (signatures, imports, headings) used by the tiered strategy when the whole document does not fit. Empty means none. bytes_length property bytes_length: int UTF-8 byte length of the content. of classmethod of( path: str, content: str, *, pinned: bool = False, language: str | None = None, extract: str | None = None, ) -> ContextDocument Build a document, computing its digest and filling in what was omitted. Parameters: Name Type Description Default path str POSIX-style relative path. required content str The document's text. required pinned bool Whether the document must be included ahead of ranked candidates. False language str | None Overrides language detection. None extract str | None Overrides extraction. Pass "" to opt out of it entirely; omit it to have one derived from the content. None Returns: Type Description ContextDocument The document, with sha256 computed and language/extract derived unless ContextDocument they were supplied. anyinfer.context.Reduction dataclass Reduction( strategy: str, representation: str, documents: tuple[ContextDocument, ...], candidate_count: int, text: str, estimated_tokens: int, max_tokens: int, max_bytes: int, max_documents: int, total_bytes: int, binding_constraints: tuple[str, ...] = (), collapsed_exact: int = 0, collapsed_near: int = 0, compacted_count: int = 0, partial_count: int = 0, carried_over: int = 0, tier_metadata: Mapping[str, Any] | None = None, ) What a reduction produced, and what it cost. Attributes: Name Type Description strategy str The strategy that was requested. auto stays auto even after dispatch, so the caller can see what they asked for. representation str The strategy actually applied — what auto resolved to. documents tuple[ContextDocument, ...] Documents represented at full, compact, or extract fidelity. In tiered this is the set actually rendered in detail, not the ranked prefix. candidate_count int How many documents were offered. text str The rendered envelope. Always present — place it in your own message. estimated_tokens int Planning-side estimate of text. max_tokens int The token budget this reduction was held to. max_bytes int The byte ceiling in force. max_documents int The document-count ceiling in force. total_bytes int UTF-8 byte length of text. binding_constraints tuple[str, ...] Which ceilings excluded at least one document, in the fixed order ("document count", "bytes", "tokens"). Empty means everything fit. collapsed_exact int Documents rendered as a pointer because a byte-identical copy was sent. Lossless. collapsed_near int Documents rendered as a pointer because a similar copy was sent. Their differences are not in the envelope. compacted_count int Documents sent with commentary removed because they would not fit whole. partial_count int Documents represented by only part of their content — the spans packed selected, rather than the whole file. carried_over int Documents this reduction kept because the previous one had them, when previous= was supplied. tier_metadata Mapping[str, Any] | None Strategy-specific detail (tier composition, chunk counts). collapsed_count property collapsed_count: int How many documents were represented by a pointer to another. omitted_count property omitted_count: int How many offered documents are not represented at all. Collapsed duplicates are not omitted: their content reached the model under another path, and the envelope says so. complete property complete: bool Whether every offered document reached the model at full fidelity. Exact collapse preserves completeness — the same bytes were sent, once. Near collapse, compaction, and chunk-level selection do not: each drops content that was offered, and a strategy that sends fragments must not report the same confidence as one that sends files. state state() -> ReductionState Capture what was sent, to hand back as the next turn's previous=. metadata metadata() -> dict[str, Any] The full machine-readable record, for logging or a debug pane. summary summary() -> str A one-line, content-free description of what happened. Safe to show a user or write to a log: counts and ceilings only, never paths or content — a path name can itself be sensitive. event event(*, calls: int = 0) -> ContextReduced Build the telemetry event describing this reduction. anyinfer.context.ReductionState dataclass ReductionState( entries: tuple[tuple[str, str], ...] = (), representation: str = "", ) What an earlier reduction sent, so the next one can send the same thing. Selection is deterministic given the same inputs, but a corpus changes between turns and a re-ranked selection can churn for no reason — swapping one file for an equally ranked other, moving the whole prompt prefix and missing the provider's cache. Hand the previous state back through select(previous=...) and unchanged documents get a rank bonus, so the set stays put unless something real moved it. Attributes: Name Type Description entries tuple[tuple[str, str], ...] (path, sha256) for every document rendered at detail fidelity, in path order. representation str The strategy that produced it, for diagnostics. of classmethod of(reduction: Reduction) -> ReductionState Capture the state of a completed reduction. unchanged unchanged( documents: Sequence[ContextDocument], ) -> frozenset[str] Paths present here and still byte-identical in documents. A path whose content changed is deliberately excluded: carrying it over would move the prompt prefix anyway, so there is nothing to preserve. metadata metadata() -> dict[str, Any] The machine-readable record, content-free apart from paths the caller owns. anyinfer.context.Strategy module-attribute Strategy = Literal[ "auto", "whole", "ranked", "tiered", "packed" ] Names accepted by select's strategy argument. anyinfer.context.RenderOrder module-attribute RenderOrder = Literal['path', 'rank'] Whether selected documents render by stable path order or relevance rank. anyinfer.context.RankCache dataclass RankCache( term_counts: dict[str, Counter[str]] = dict(), document_frequency: Counter[str] = Counter(), document_lengths: dict[str, int] = dict(), total_documents: int = 0, split_identifiers: bool = False, ) Precomputed term statistics for one corpus. Ranking a corpus repeatedly — as an interactive app does on every turn — otherwise re-tokenizes every document each time. Build one with anyinfer.context.rank.build_rank_cache() and pass it back in. Validity is the caller's responsibility: key it on a corpus hash and rebuild when the corpus changes. Passing a cache built from different documents produces undefined ranking rather than an error, which is why the cache is not consulted for document identity. Attributes: Name Type Description term_counts dict[str, Counter[str]] Per-document term frequencies, keyed by document path. document_frequency Counter[str] How many documents contain each term. document_lengths dict[str, int] Total token count per document, for length normalization. total_documents int Corpus size, for the inverse-document-frequency term. split_identifiers bool Which tokenization produced these counts. Ranking checks it and rebuilds rather than scoring a query tokenized one way against statistics gathered the other — a mismatch there produces plausible, wrong ordering. Advanced Settings One record carries every algorithmic choice. The same field names are the context block of the configuration file and the --context-* flags of anyinfer context. Every setting that changes what gets sent is off by default; ContextTuning.recommended() enables the set worth having for a source-code corpus. Setting Default What it changes collapse_duplicates True Render byte-identical documents once near_duplicate_threshold 0.0 Collapse merely similar documents too selection_order "rank" "density" admits by score per token diversity 0.0 Penalize candidates resembling what is already chosen split_identifiers False Tokenize compound identifiers into their parts query_expansion False Pseudo-relevance feedback before ranking salience_weight 0.0 Blend in import-graph centrality compact_fallback False Shorten a document rather than drop it carry_over_bonus 0.0 Keep the previous turn's selection, with previous= chunk_tokens 512 Chunk size for packed and distill rollup_share 0.45 Budget share tiered reserves for its rollup Two orderings deserve a note. "rank" admits documents strongest-first; "density" admits them by score divided by token cost, which packs measurably more relevance into a fixed budget (the classic knapsack result) at the risk of preferring two good small files over one great large one. diversity penalizes each candidate by how much it resembles what is already selected (multiplicatively, value * (1 - diversity * similarity), because the two value scales differ by orders of magnitude), so a budget is not spent on eight files that say the same thing. anyinfer.context.ContextTuning dataclass ContextTuning( collapse_duplicates: bool = True, near_duplicate_threshold: float = 0.0, shingle_size: int = 5, selection_order: SelectionOrder = "rank", diversity: float = 0.0, split_identifiers: bool = False, query_expansion: bool = False, expansion_terms: int = 8, feedback_documents: int = 5, expansion_weight: float = 0.4, salience_weight: float = 0.0, salience_damping: float = 0.85, salience_iterations: int = 20, compact_fallback: bool = False, chunk_tokens: int = DEFAULT_CHUNK_TOKENS, rollup_share: float = 0.45, carry_over_bonus: float = 0.0, ) Advanced settings for corpus reduction. Every field defaults to the behaviour AnyInfer has always had, except collapse_duplicates. Construct one, or start from recommended() and override. Attributes: Name Type Description collapse_duplicates bool Render byte-identical documents once, with the rest as pointer elements. Lossless — the content is still present, and on by default, since sending the same bytes twice helps nobody. near_duplicate_threshold float Jaccard similarity at or above which two documents are treated as duplicates of each other. 0.0 disables near-duplicate detection; 0.9 is a good starting point for vendored or generated siblings. Lossy: the near-duplicate's differences are not sent, so it is off by default. shingle_size int Word count per shingle for near-duplicate comparison. Larger is stricter. selection_order SelectionOrder rank admits documents strongest-first. density admits them by score per token, which fits measurably more relevance into the same budget at the cost of sometimes preferring two good small files to one great large one. diversity float Similarity penalty applied to candidates resembling what is already selected, between 0.0 (pure relevance) and 1.0 (near-pure novelty). Stops a budget being spent on eight files that say the same thing. split_identifiers bool Tokenize resolve_credentials and resolveCredentials as their parts and as the whole, so a query of "resolve credentials" matches the identifier that names it. query_expansion bool Rank once, harvest distinctive terms from the strongest documents, then re-rank with the expanded query. The lexical answer to vocabulary mismatch: it finds "login" from "authentication" whenever the two co-occur anywhere in the corpus. Costs a second ranking pass, no inference. expansion_terms int How many harvested terms to add. feedback_documents int How many top-ranked documents to harvest them from. expansion_weight float Weight of an expansion term relative to an original query term. salience_weight float How much a document's centrality in the corpus's own import graph contributes to its score. The signal is query-independent, so this is what orders a corpus when the query is weak or absent; 0.0 disables the graph pass entirely. salience_damping float Random-restart probability complement for the centrality iteration. The conventional 0.85. salience_iterations int Fixed iteration count. Fixed rather than convergence-tested, because determinism outranks the last decimal place. compact_fallback bool When a document will not fit whole, send it with comments, docstrings, and blank runs removed before giving up on it. Elisions are counted in the rendered element, never silent. chunk_tokens int Target chunk size for packed and for distill. rollup_share float Share of the budget tiered reserves for its module rollup. carry_over_bonus float Rank bonus applied to documents an earlier reduction already sent unchanged, when previous= is supplied. Keeps the selected set — and therefore the rendered prefix — stable across turns so provider prompt caches keep hitting. 0.0 ranks each turn from scratch. ranking_is_default property ranking_is_default: bool Whether ranking behaves exactly as the unconfigured ranker does. Selection consults this to skip the expansion and centrality passes entirely rather than running them with neutral parameters. __post_init__ __post_init__() -> None Reject settings that cannot produce a usable reduction. Raises: Type Description ValueError On an out-of-range or non-finite value, naming the field. recommended classmethod recommended() -> ContextTuning The settings worth turning on for a typical source-code corpus. Near-duplicate collapse at a strict threshold, density-ordered selection with a mild diversity penalty, identifier splitting and query expansion, a light centrality signal, and compact fallback instead of dropping a file outright. Every one of these changes what gets sent, which is why they are a named preset rather than the default. from_mapping classmethod from_mapping(values: Mapping[str, Any]) -> ContextTuning Build settings from a JSON-shaped mapping. Used by the shared configuration loader and the CLI, so a context block in a config file and a --context-* flag mean exactly the same thing. Parameters: Name Type Description Default values Mapping[str, Any] Field names to values. Unknown names are an error rather than being ignored, so a typo does not silently do nothing. required Returns: Type Description ContextTuning The settings. Raises: Type Description ValueError On an unknown key, a wrong type, or an out-of-range value. merged merged(**overrides: Any) -> ContextTuning Return a copy with overrides applied, dropping any that are None. The shape command-line parsing wants: unspecified flags arrive as None and must leave the configured value alone. to_mapping to_mapping() -> dict[str, Any] The settings as a JSON-shaped mapping, for logging or round-tripping. anyinfer.context.SelectionOrder module-attribute SelectionOrder = Literal['rank', 'density'] How the greedy selector orders candidates. anyinfer.context.SELECTION_ORDERS module-attribute SELECTION_ORDERS = ('rank', 'density') Accepted ContextTuning.selection_order values. Selection anyinfer.context.select Corpus selection: the strategies, the result type, and the dispatch rule. Reduction is emulation of a larger context window, and emulation announces itself. Every reduction returns what it kept, what it dropped, what it collapsed, which ceiling bound it, and a content-free summary — plus a ContextReduced telemetry event when an observer is supplied. A silent truncation that looks like a complete answer is the failure mode this module exists to prevent. Everything algorithmic is a setting rather than a constant. ContextTuning decides how duplicates collapse, whether candidates are ordered by relevance or relevance per token, whether near-identical documents are penalized against each other, and what a document degrades to instead of being dropped. The defaults reproduce the plain behaviour exactly, so turning nothing on changes nothing. DEFAULT_MAX_DOCUMENTS module-attribute DEFAULT_MAX_DOCUMENTS = 200 Ceiling on documents in one envelope; a long tail of tiny files helps nobody. DEFAULT_MAX_BYTES module-attribute DEFAULT_MAX_BYTES = 4 * 1024 * 1024 Byte ceiling, enforced independently of tokens because transports cap bytes. VALID_STRATEGIES module-attribute VALID_STRATEGIES = ( "auto", "whole", "ranked", "tiered", "packed", ) Strategy names select() accepts. Strategy module-attribute Strategy = Literal[ "auto", "whole", "ranked", "tiered", "packed" ] Names accepted by select's strategy argument. RenderOrder module-attribute RenderOrder = Literal['path', 'rank'] Whether selected documents render by stable path order or relevance rank. ReductionState dataclass ReductionState( entries: tuple[tuple[str, str], ...] = (), representation: str = "", ) What an earlier reduction sent, so the next one can send the same thing. Selection is deterministic given the same inputs, but a corpus changes between turns and a re-ranked selection can churn for no reason — swapping one file for an equally ranked other, moving the whole prompt prefix and missing the provider's cache. Hand the previous state back through select(previous=...) and unchanged documents get a rank bonus, so the set stays put unless something real moved it. Attributes: Name Type Description entries tuple[tuple[str, str], ...] (path, sha256) for every document rendered at detail fidelity, in path order. representation str The strategy that produced it, for diagnostics. of classmethod of(reduction: Reduction) -> ReductionState Capture the state of a completed reduction. unchanged unchanged( documents: Sequence[ContextDocument], ) -> frozenset[str] Paths present here and still byte-identical in documents. A path whose content changed is deliberately excluded: carrying it over would move the prompt prefix anyway, so there is nothing to preserve. metadata metadata() -> dict[str, Any] The machine-readable record, content-free apart from paths the caller owns. Reduction dataclass Reduction( strategy: str, representation: str, documents: tuple[ContextDocument, ...], candidate_count: int, text: str, estimated_tokens: int, max_tokens: int, max_bytes: int, max_documents: int, total_bytes: int, binding_constraints: tuple[str, ...] = (), collapsed_exact: int = 0, collapsed_near: int = 0, compacted_count: int = 0, partial_count: int = 0, carried_over: int = 0, tier_metadata: Mapping[str, Any] | None = None, ) What a reduction produced, and what it cost. Attributes: Name Type Description strategy str The strategy that was requested. auto stays auto even after dispatch, so the caller can see what they asked for. representation str The strategy actually applied — what auto resolved to. documents tuple[ContextDocument, ...] Documents represented at full, compact, or extract fidelity. In tiered this is the set actually rendered in detail, not the ranked prefix. candidate_count int How many documents were offered. text str The rendered envelope. Always present — place it in your own message. estimated_tokens int Planning-side estimate of text. max_tokens int The token budget this reduction was held to. max_bytes int The byte ceiling in force. max_documents int The document-count ceiling in force. total_bytes int UTF-8 byte length of text. binding_constraints tuple[str, ...] Which ceilings excluded at least one document, in the fixed order ("document count", "bytes", "tokens"). Empty means everything fit. collapsed_exact int Documents rendered as a pointer because a byte-identical copy was sent. Lossless. collapsed_near int Documents rendered as a pointer because a similar copy was sent. Their differences are not in the envelope. compacted_count int Documents sent with commentary removed because they would not fit whole. partial_count int Documents represented by only part of their content — the spans packed selected, rather than the whole file. carried_over int Documents this reduction kept because the previous one had them, when previous= was supplied. tier_metadata Mapping[str, Any] | None Strategy-specific detail (tier composition, chunk counts). collapsed_count property collapsed_count: int How many documents were represented by a pointer to another. omitted_count property omitted_count: int How many offered documents are not represented at all. Collapsed duplicates are not omitted: their content reached the model under another path, and the envelope says so. complete property complete: bool Whether every offered document reached the model at full fidelity. Exact collapse preserves completeness — the same bytes were sent, once. Near collapse, compaction, and chunk-level selection do not: each drops content that was offered, and a strategy that sends fragments must not report the same confidence as one that sends files. state state() -> ReductionState Capture what was sent, to hand back as the next turn's previous=. metadata metadata() -> dict[str, Any] The full machine-readable record, for logging or a debug pane. summary summary() -> str A one-line, content-free description of what happened. Safe to show a user or write to a log: counts and ceilings only, never paths or content — a path name can itself be sensitive. event event(*, calls: int = 0) -> ContextReduced Build the telemetry event describing this reduction. StrategyOutlook dataclass StrategyOutlook( strategy: str, representation: str, selected_count: int, omitted_count: int, collapsed_count: int, compacted_count: int, partial_count: int, estimated_tokens: int, total_bytes: int, binding_constraints: tuple[str, ...], complete: bool, ) What one strategy would produce, costed exactly rather than modelled. Attributes: Name Type Description strategy str The strategy this describes. representation str What it resolved to — identical to strategy here, since plan() never costs auto. selected_count int Documents it would represent at detail fidelity. omitted_count int Documents it would not represent at all. collapsed_count int Documents it would render as a pointer to another. compacted_count int Documents it would shorten rather than drop. partial_count int Documents it would represent by fragments rather than whole. estimated_tokens int Planning-side estimate of the envelope it would render. total_bytes int Byte length of that envelope. binding_constraints tuple[str, ...] Which ceilings would bind. complete bool Whether it would send everything at full fidelity. of classmethod of(strategy: str, reduction: Reduction) -> StrategyOutlook Describe a reduction that was run purely to be measured. metadata metadata() -> dict[str, Any] The machine-readable record. ReductionPlan dataclass ReductionPlan( candidate_count: int, max_tokens: int, options: tuple[StrategyOutlook, ...], distill_chunks: int, distill_calls: int, ) What every strategy would do with this corpus and this budget. A dry run for context preparation, in the same spirit as the pre-dispatch request preflight: choose a strategy from measured outcomes instead of guessing and finding out in the transcript. Costing it spends no inference and touches no network — the deterministic strategies are actually executed and their envelopes measured, then discarded, so the numbers are exact rather than modelled. Attributes: Name Type Description candidate_count int Documents offered. max_tokens int The budget every option was held to. options tuple[StrategyOutlook, ...] One outlook per deterministic strategy, in a fixed order. distill_chunks int How many chunks anyinfer.context.distill would split this corpus into at the configured chunk size. distill_calls int The floor on generation calls distillation would spend — one per chunk plus a single reduce. A corpus whose notes do not fit at once reduces hierarchically and spends more. option option(strategy: str) -> StrategyOutlook | None The outlook for one strategy, or None if it was not costed. best best() -> StrategyOutlook | None The option that gets the most of the corpus to the model. Prefers an option that sends everything at full fidelity; failing that, the one representing the most documents at detail fidelity; and among equals, the one that represents them most faithfully — a whole file beats a summary beats a fragment. It is a recommendation, not a decision: an app that would rather have twelve whole files than four hundred summarized ones should read options and pick for itself. metadata metadata() -> dict[str, Any] The full machine-readable record. summary summary() -> str A one-line, content-free description of the plan. normalize_strategy normalize_strategy(value: str | None) -> str Normalize a strategy name. Parameters: Name Type Description Default value str | None The requested strategy; None or blank means "auto". required Returns: Type Description str The lowercase strategy name. Raises: Type Description ValueError On an unrecognized strategy, listing the valid names. select select( documents: Iterable[ContextDocument], query: str, *, max_tokens: int, strategy: str = "auto", max_documents: int = DEFAULT_MAX_DOCUMENTS, max_bytes: int = DEFAULT_MAX_BYTES, estimator: TokenEstimator | None = None, rank_cache: RankCache | None = None, module_digests: Mapping[str, str] | None = None, render_order: RenderOrder = "path", tuning: ContextTuning | None = None, previous: ReductionState | None = None, observer: Observer | None = None, ranker: SemanticRanker | None = None, ) -> Reduction Reduce a corpus to fit a token budget. Parameters: Name Type Description Default documents Iterable[ContextDocument] The corpus the app has already collected and approved. required query str What the request is about, used for relevance ranking. required max_tokens int The token budget. Normally client.budget(messages, target=...).remaining_tokens — an explicit number, because an unknown context window stays unknown rather than being guessed at. required strategy str auto (default), whole, ranked, tiered, or packed. 'auto' max_documents int Ceiling on documents represented. DEFAULT_MAX_DOCUMENTS max_bytes int Ceiling on envelope bytes. DEFAULT_MAX_BYTES estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None rank_cache RankCache | None Precomputed corpus statistics, for repeated queries. None module_digests Mapping[str, str] | None App-supplied per-module summaries, rendered by tiered. The library never generates these. None render_order RenderOrder path (default) renders selected documents in path order regardless of rank, so consecutive turns over the same corpus share a stable prompt prefix and provider prompt caches hit. rank renders strongest-first. 'path' tuning ContextTuning | None Advanced settings — duplicate collapse, selection order, diversity, query expansion, centrality, compact fallback. Defaults reproduce the plain behaviour exactly. None previous ReductionState | None The state of the last reduction over this corpus. Unchanged documents get tuning.carry_over_bonus so the selected set, and the rendered prefix — stays stable across turns. None observer Observer | None Receives a ContextReduced event describing the outcome. None ranker SemanticRanker | None Caller-supplied semantic scoring (SemanticRanker). When set, its scores replace the lexical ranking for both ordering and admission — one scoring call per reduction. The default stays lexical and offline; build a rerank-backed implementation with anyinfer.semantic_ranker. None Returns: Type Description Reduction The Reduction, whose text is the envelope to place in your own message. Raises: Type Description ValueError On an unknown strategy or a non-positive budget. plan plan( documents: Iterable[ContextDocument], query: str, *, max_tokens: int, max_documents: int = DEFAULT_MAX_DOCUMENTS, max_bytes: int = DEFAULT_MAX_BYTES, estimator: TokenEstimator | None = None, module_digests: Mapping[str, str] | None = None, tuning: ContextTuning | None = None, ) -> ReductionPlan Cost every strategy against this corpus without committing to one. Spends no inference and performs no I/O: each deterministic strategy is run, its envelope measured, and the text discarded. The distillation figures are projections — that is the only strategy whose cost cannot be known without paying it. Parameters: Name Type Description Default documents Iterable[ContextDocument] The corpus. required query str What the request is about. required max_tokens int The budget to hold every option to. required max_documents int Ceiling on documents represented. DEFAULT_MAX_DOCUMENTS max_bytes int Ceiling on envelope bytes. DEFAULT_MAX_BYTES estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None module_digests Mapping[str, str] | None App-supplied module summaries, costed into the tiered option. None tuning ContextTuning | None Advanced settings, applied to every option so the comparison is fair. None Returns: Type Description ReductionPlan The ReductionPlan. Raises: Type Description ValueError On a non-positive budget. anyinfer.context.normalize_strategy normalize_strategy(value: str | None) -> str Normalize a strategy name. Parameters: Name Type Description Default value str | None The requested strategy; None or blank means "auto". required Returns: Type Description str The lowercase strategy name. Raises: Type Description ValueError On an unrecognized strategy, listing the valid names. anyinfer.context.VALID_STRATEGIES module-attribute VALID_STRATEGIES = ( "auto", "whole", "ranked", "tiered", "packed", ) Strategy names select() accepts. anyinfer.context.DEFAULT_MAX_DOCUMENTS module-attribute DEFAULT_MAX_DOCUMENTS = 200 Ceiling on documents in one envelope; a long tail of tiny files helps nobody. anyinfer.context.DEFAULT_MAX_BYTES module-attribute DEFAULT_MAX_BYTES = 4 * 1024 * 1024 Byte ceiling, enforced independently of tokens because transports cap bytes. Planning Cost every strategy before committing to one. Spends no inference and performs no I/O. anyinfer.context.plan plan( documents: Iterable[ContextDocument], query: str, *, max_tokens: int, max_documents: int = DEFAULT_MAX_DOCUMENTS, max_bytes: int = DEFAULT_MAX_BYTES, estimator: TokenEstimator | None = None, module_digests: Mapping[str, str] | None = None, tuning: ContextTuning | None = None, ) -> ReductionPlan Cost every strategy against this corpus without committing to one. Spends no inference and performs no I/O: each deterministic strategy is run, its envelope measured, and the text discarded. The distillation figures are projections — that is the only strategy whose cost cannot be known without paying it. Parameters: Name Type Description Default documents Iterable[ContextDocument] The corpus. required query str What the request is about. required max_tokens int The budget to hold every option to. required max_documents int Ceiling on documents represented. DEFAULT_MAX_DOCUMENTS max_bytes int Ceiling on envelope bytes. DEFAULT_MAX_BYTES estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None module_digests Mapping[str, str] | None App-supplied module summaries, costed into the tiered option. None tuning ContextTuning | None Advanced settings, applied to every option so the comparison is fair. None Returns: Type Description ReductionPlan The ReductionPlan. Raises: Type Description ValueError On a non-positive budget. anyinfer.context.ReductionPlan dataclass ReductionPlan( candidate_count: int, max_tokens: int, options: tuple[StrategyOutlook, ...], distill_chunks: int, distill_calls: int, ) What every strategy would do with this corpus and this budget. A dry run for context preparation, in the same spirit as the pre-dispatch request preflight: choose a strategy from measured outcomes instead of guessing and finding out in the transcript. Costing it spends no inference and touches no network — the deterministic strategies are actually executed and their envelopes measured, then discarded, so the numbers are exact rather than modelled. Attributes: Name Type Description candidate_count int Documents offered. max_tokens int The budget every option was held to. options tuple[StrategyOutlook, ...] One outlook per deterministic strategy, in a fixed order. distill_chunks int How many chunks anyinfer.context.distill would split this corpus into at the configured chunk size. distill_calls int The floor on generation calls distillation would spend — one per chunk plus a single reduce. A corpus whose notes do not fit at once reduces hierarchically and spends more. option option(strategy: str) -> StrategyOutlook | None The outlook for one strategy, or None if it was not costed. best best() -> StrategyOutlook | None The option that gets the most of the corpus to the model. Prefers an option that sends everything at full fidelity; failing that, the one representing the most documents at detail fidelity; and among equals, the one that represents them most faithfully — a whole file beats a summary beats a fragment. It is a recommendation, not a decision: an app that would rather have twelve whole files than four hundred summarized ones should read options and pick for itself. metadata metadata() -> dict[str, Any] The full machine-readable record. summary summary() -> str A one-line, content-free description of the plan. anyinfer.context.StrategyOutlook dataclass StrategyOutlook( strategy: str, representation: str, selected_count: int, omitted_count: int, collapsed_count: int, compacted_count: int, partial_count: int, estimated_tokens: int, total_bytes: int, binding_constraints: tuple[str, ...], complete: bool, ) What one strategy would produce, costed exactly rather than modelled. Attributes: Name Type Description strategy str The strategy this describes. representation str What it resolved to — identical to strategy here, since plan() never costs auto. selected_count int Documents it would represent at detail fidelity. omitted_count int Documents it would not represent at all. collapsed_count int Documents it would render as a pointer to another. compacted_count int Documents it would shorten rather than drop. partial_count int Documents it would represent by fragments rather than whole. estimated_tokens int Planning-side estimate of the envelope it would render. total_bytes int Byte length of that envelope. binding_constraints tuple[str, ...] Which ceilings would bind. complete bool Whether it would send everything at full fidelity. of classmethod of(strategy: str, reduction: Reduction) -> StrategyOutlook Describe a reduction that was run purely to be measured. metadata metadata() -> dict[str, Any] The machine-readable record. Duplicate Collapse anyinfer.context.find_duplicates find_duplicates( documents: Sequence[ContextDocument], *, tuning: ContextTuning = DEFAULT_TUNING, ) -> DuplicateMap Group duplicate documents and choose one representative for each group. Parameters: Name Type Description Default documents Sequence[ContextDocument] The corpus. required tuning ContextTuning Supplies collapse_duplicates, near_duplicate_threshold, and shingle_size. DEFAULT_TUNING Returns: Type Description DuplicateMap The DuplicateMap. Empty when both mechanisms are disabled, when the corpus has DuplicateMap fewer than two documents, or when nothing repeated. anyinfer.context.DuplicateMap dataclass DuplicateMap( canonical: Mapping[str, str] = dict(), exact: frozenset[str] = frozenset(), ) Which documents were collapsed into which. Attributes: Name Type Description canonical Mapping[str, str] Duplicate path to the path that represents it. A path absent from this mapping is itself canonical. exact frozenset[str] Paths collapsed because they were byte-identical, as opposed to merely similar. Rendering distinguishes the two, because one is lossless and the other is not. collapsed_count property collapsed_count: int How many documents are represented by another. __bool__ __bool__() -> bool Whether anything was collapsed at all. is_exact is_exact(path: str) -> bool Whether path was collapsed losslessly. members members(canonical_path: str) -> tuple[str, ...] Paths collapsed into canonical_path, in path order. History Compaction Reduce a conversation rather than a corpus, without breaking tool-call pairing. Call it directly, or hand anyinfer.HistoryPolicy to a client and let every frontend built on that client apply the same rules on the request path. anyinfer.context.compact_history compact_history( messages: Iterable[Message], *, max_tokens: int, estimator: TokenEstimator | None = None, keep_recent: int = DEFAULT_KEEP_RECENT, keep_system: bool = True, observer: Observer | None = None, ) -> HistoryCompaction Shrink a conversation to fit a token budget without invalidating it. Three passes over the unprotected middle, cheapest loss first: tool-result payloads are elided, then long text payloads, then plain messages are dropped outright. Each pass stops the moment the conversation fits, so a transcript that needs one large tool result elided loses exactly that and nothing else. Parameters: Name Type Description Default messages Iterable[Message] The conversation so far. required max_tokens int The budget. Normally client.budget(...).remaining_tokens less whatever the next turn will add — an explicit number, because an unknown window stays unknown. required estimator TokenEstimator | None Token counting strategy; defaults to the byte heuristic. None keep_recent int Trailing messages held at full fidelity. DEFAULT_KEEP_RECENT keep_system bool Whether system messages are protected wherever they appear. Leave this on unless the application's system prompt is genuinely disposable. True observer Observer | None Receives a ContextReduced event describing the outcome. None Returns: Type Description HistoryCompaction The HistoryCompaction. Check fits: a conversation whose protected messages HistoryCompaction alone exceed the budget comes back unchanged and honest rather than mutilated. Raises: Type Description ValueError On a non-positive budget or a negative keep_recent. anyinfer.context.HistoryCompaction dataclass HistoryCompaction( messages: tuple[Message, ...], original_count: int, dropped_count: int, elided_results: int, elided_texts: int, estimated_tokens: int, original_tokens: int, max_tokens: int, fits: bool, ) What compaction produced, and what it cost. Attributes: Name Type Description messages tuple[Message, ...] The compacted conversation, ready to send. original_count int Messages offered. dropped_count int Messages removed entirely. elided_results int Tool results whose payload was replaced by a marker. elided_texts int Text parts whose payload was replaced by a marker. estimated_tokens int Planning-side estimate of the compacted conversation. original_tokens int The same estimate before compaction. max_tokens int The budget it was held to. fits bool Whether the result is within that budget. False means the protected messages alone exceed it, and no further compaction was available — the caller decides what to do, because dropping a system prompt or the current turn is not a decision a library should make quietly. changed property changed: bool Whether anything was dropped or elided at all. complete property complete: bool Whether the conversation reached the model intact. saved_tokens property saved_tokens: int How many planning tokens compaction recovered. metadata metadata() -> dict[str, Any] The full machine-readable record, for logging or a debug pane. summary summary() -> str A one-line, content-free description of what happened. Safe to show a user or write to a log: counts and ceilings only, never message text. event event() -> ContextReduced Build the telemetry event describing this compaction. anyinfer.context.DEFAULT_KEEP_RECENT module-attribute DEFAULT_KEEP_RECENT = 6 Trailing messages held at full fidelity. Six covers a tool call, its result, and the exchange around them — enough that the model is never asked to continue from a turn it cannot see. anyinfer.HistoryPolicy dataclass HistoryPolicy( enabled: bool = True, mode: HistoryMode = "last_resort", keep_recent: int = 6, keep_system: bool = True, ) Opt-in conversation compaction, applied by the client on the request path. A prompt that outgrows the window has two possible answers: send it somewhere with a bigger window, or make it smaller. The router has always owned the first (Route.context_window_targets). This owns the second, at the same layer, so the Python API, the command line, and the OpenAI-compatible frontend all behave the same way, because all three are the same client wearing different skins. Compaction is never silent: it emits a ContextReduced telemetry event, and it is off unless a policy is supplied. Attributes: Name Type Description enabled bool Whether to compact at all. Present so a configuration file can turn the policy off without deleting how it was tuned. mode HistoryMode last_resort compacts only after the route's context-overflow chain is exhausted, so a larger-window model is always preferred to losing history. proactive compacts to fit the resolved target before dispatch, trading that preference for one fewer failed preflight. keep_recent int Trailing messages held at full fidelity. keep_system bool Whether system messages are protected wherever they appear. See anyinfer.context.compact_history for the rules compaction follows and for the same behaviour as a function you call yourself. active property active: bool Whether this policy will actually compact anything. __post_init__ __post_init__() -> None Reject a policy that cannot be applied. Raises: Type Description ValueError On an unknown mode or a negative recent window. Ranking Public so it can be replaced: the shipped ranker is lexical, and an application needing semantic retrieval ranks its own documents and passes the result through. anyinfer.context.rank Lexical relevance ranking. A BM25-style scorer, deliberately lexical and dependency-free: term frequency saturated and length-normalized, weighted by inverse document frequency, plus two signals that matter a great deal in a code or document corpus and nothing in classical IR — a path match outweighs a body match, and well-known anchor files (README, pyproject.toml, ARCHITECTURE) get a small bonus. What this is not. There are no embeddings and no semantic matching. That is a deliberate boundary — see the concept documentation, and the reason ranking is exposed as a function you can replace rather than hidden inside selection. Three optional settings narrow the gap without an index or a model. Identifier splitting tokenizes resolve_credentials as its parts as well as the whole, so a query phrased in words matches an identifier phrased in code. Query expansion ranks once, harvests distinctive terms from the strongest documents, and re-ranks, which does find "login" from "authentication" whenever the two co-occur anywhere in the corpus. And centrality scores a document by its position in the corpus's own import graph, a query-independent signal that is what orders a corpus when the query is weak or absent. Tokenization is ASCII alphanumeric. Ranking is fully deterministic: ties break on path depth, then path, then digest, so the same corpus and query always produce the same order regardless of the order documents were supplied in. STOP_WORDS module-attribute STOP_WORDS = frozenset( { "a", "about", "all", "an", "and", "any", "are", "as", "at", "be", "been", "but", "by", "can", "do", "does", "for", "from", "get", "has", "have", "how", "i", "if", "in", "into", "is", "it", "its", "of", "on", "or", "please", "should", "so", "some", "than", "that", "the", "their", "them", "then", "there", "these", "this", "to", "use", "was", "were", "what", "when", "where", "which", "who", "why", "will", "with", "would", "you", "your", } ) Words carrying no retrieval signal, dropped from queries and documents alike. TERM_SATURATION module-attribute TERM_SATURATION = 1.2 BM25's k1: how fast repeated occurrences of a term stop adding score. LENGTH_NORMALIZATION module-attribute LENGTH_NORMALIZATION = 0.001 Penalty per token of document length, so a long file cannot win on volume alone. PATH_MATCH_WEIGHT module-attribute PATH_MATCH_WEIGHT = 4.0 How much more a query term in the path counts than the same term in the body. Deliberately large: someone asking about credentials almost always means the file named for it, even when a dozen other files mention the word more often. ANCHOR_SCORE module-attribute ANCHOR_SCORE = 0.25 Bonus for files that orient a reader regardless of the query. ANCHOR_NAMES module-attribute ANCHOR_NAMES = frozenset( { "architecture", "changelog", "contributing", "design", "overview", "readme", "cargo.toml", "go.mod", "package.json", "pom.xml", "pyproject.toml", } ) Filenames (and stems) worth a small unconditional boost. SemanticRanker Bases: Protocol Caller-supplied relevance scoring for context reduction. The default ranking is lexical and offline on purpose; this protocol is the opt-in seam for a semantic ranker backed by a rerank model. Scores are keyed by ContextDocument.path — a document absent from the mapping scores 0.0. Scores are only compared against each other within one call, never persisted. Implementations live outside this package (context reduction is a leaf consumer and never imports the client); anyinfer.semantic_ranker builds one from a client and a rerank target. scores scores( documents: Sequence[ContextDocument], query: str ) -> Mapping[str, float] Score every document's relevance to query, keyed by document path. tokenize tokenize( text: str, *, split_identifiers: bool = False ) -> list[str] Split text into lowercase alphanumeric terms, dropping stop words. ASCII-only by design — the tradeoff is stated in the module documentation. Parameters: Name Type Description Default text str What to tokenize. required split_identifiers bool Also emit the parts of compound identifiers. resolveCredentials and resolve_credentials both yield the compound and resolve and credentials, so a query written in words matches an identifier written in code. The compound is kept as well, so an exact match on the full identifier still scores highest. False Returns: Type Description list[str] The terms, in order of appearance. build_rank_cache build_rank_cache( documents: Iterable[ContextDocument], *, split_identifiers: bool = False, ) -> RankCache Precompute term statistics for a corpus. Pass the result to rank() on subsequent queries over the same corpus. The caller owns invalidation; see RankCache. Parameters: Name Type Description Default documents Iterable[ContextDocument] The corpus. required split_identifiers bool Tokenize compound identifiers into their parts as well. Must match the setting ranking will use; rank() rebuilds a cache that disagrees. False Returns: Type Description RankCache The statistics. score_document score_document( document: ContextDocument, query_terms: Mapping[str, float], cache: RankCache, ) -> float Score one document against a query. Parameters: Name Type Description Default document ContextDocument The candidate. required query_terms Mapping[str, float] Query terms mapped to their weights. Counter(tokenize(query)) for a plain query; expand_query produces a weighted one. required cache RankCache Corpus statistics covering this document. required Returns: Type Description float A non-negative relevance score. Zero means nothing matched, which is still a float valid candidate, just an unranked one. query_terms query_terms( query: str, *, tuning: ContextTuning = DEFAULT_TUNING ) -> dict[str, float] Tokenize a query into weighted terms, without expansion. The base a expand_query call starts from, and what score_document wants when no expansion is configured. expand_query expand_query( query: str, documents: Sequence[ContextDocument], *, cache: RankCache, tuning: ContextTuning = DEFAULT_TUNING, ) -> dict[str, float] Add distinctive terms from the strongest documents to the query. Pseudo-relevance feedback: rank once against the query as written, take the top documents on faith, and harvest the terms that make them distinctive — high frequency within that set, low frequency across the corpus. Those terms join the query at a reduced weight and everything is ranked again. This is the lexical answer to vocabulary mismatch. It has no index and no model, and it finds a file that says "login" from a query that says "authentication" whenever some document in the corpus uses both. It also inherits the classic failure mode: if the top documents are wrong, expansion makes them wronger, which is why expansion_weight defaults well below one. Parameters: Name Type Description Default query str The query as written. required documents Sequence[ContextDocument] The corpus. required cache RankCache Statistics for that corpus. required tuning ContextTuning Supplies expansion_terms, feedback_documents, and expansion_weight. DEFAULT_TUNING Returns: Type Description dict[str, float] Term weights, with the original query terms at full weight. Returns the dict[str, float] unexpanded terms when expansion is disabled, the query is empty, or the corpus dict[str, float] has nothing to harvest. salience salience( documents: Sequence[ContextDocument], *, tuning: ContextTuning = DEFAULT_TUNING, ) -> dict[str, float] Score documents by their centrality in the corpus's own import graph. An edge runs from a document to every document whose filename stem it imports. The stationary distribution of a damped random walk over those edges answers "what does this corpus depend on?", which is query-independent, and therefore what orders a corpus when the query is weak or missing entirely. Ranking with an empty query otherwise falls through to the path tie-break, which is arbitrary. Parameters: Name Type Description Default documents Sequence[ContextDocument] The corpus. required tuning ContextTuning Supplies salience_damping and salience_iterations. DEFAULT_TUNING Returns: Type Description dict[str, float] Path to a score in [0, 1], normalized so the most central document scores dict[str, float] one. An empty mapping when the corpus has no resolvable edges at all, so callers dict[str, float] can skip the blend entirely. rank rank( documents: Sequence[ContextDocument], query: str, *, rank_cache: RankCache | None = None, tuning: ContextTuning = DEFAULT_TUNING, carry_over: Iterable[str] = (), ) -> list[ContextDocument] Order documents by relevance, pinned ones first. Parameters: Name Type Description Default documents Sequence[ContextDocument] The corpus. required query str What the app is asking about. An empty query ranks everything at zero, leaving the deterministic tie-break as the order — unless tuning.salience_weight is set, which is exactly what that setting is for. required rank_cache RankCache | None Precomputed statistics for this corpus; built on the fly when absent, and rebuilt when its tokenization disagrees with tuning. None tuning ContextTuning Advanced settings. Defaults reproduce the plain lexical ranker. DEFAULT_TUNING carry_over Iterable[str] Paths an earlier reduction already sent unchanged. Each receives tuning.carry_over_bonus, which keeps a turn's selection, and therefore its rendered prefix — stable enough for a prompt cache to hit. () Returns: Type Description list[ContextDocument] A new list, most relevant first. Ordering is total and deterministic: every list[ContextDocument] pinned document precedes every unpinned one, then higher score, then shallower list[ContextDocument] path, then path, then digest. scores_for scores_for( documents: Sequence[ContextDocument], query: str, *, cache: RankCache, tuning: ContextTuning = DEFAULT_TUNING, ) -> dict[str, float] Relevance scores keyed by path, for selectors that need magnitudes not order. rank() returns an ordering, which is all a greedy fill needs. Density ordering and the diversity penalty need the numbers themselves. anyinfer.context.SemanticRanker Bases: Protocol Caller-supplied relevance scoring for context reduction. The default ranking is lexical and offline on purpose; this protocol is the opt-in seam for a semantic ranker backed by a rerank model. Scores are keyed by ContextDocument.path — a document absent from the mapping scores 0.0. Scores are only compared against each other within one call, never persisted. Implementations live outside this package (context reduction is a leaf consumer and never imports the client); anyinfer.semantic_ranker builds one from a client and a rerank target. scores scores( documents: Sequence[ContextDocument], query: str ) -> Mapping[str, float] Score every document's relevance to query, keyed by document path. anyinfer.semantic_ranker semantic_ranker( client: Client, target: Target, *, batch: BatchPolicy | None = None, ) -> SemanticRanker Build a SemanticRanker for anyinfer.context.select from a rerank target. One rerank call per reduction, spending real provider usage — which is exactly why context reduction does not do this by default. Reduction is synchronous, so this takes the synchronous Client. Parameters: Name Type Description Default client Client An open synchronous client configured with the target's provider. required target Target A rerank-capable target, e.g. "cohere:rerank-v3.5". required batch BatchPolicy | None Batching policy for corpora larger than the provider's document limit. Splitting a rerank produces chunk-local scores, so enabling rerank_cross_batch here trades global comparability for coverage — the result's warning says so. None Returns: Type Description SemanticRanker An object satisfying the SemanticRanker protocol. anyinfer.context.build_rank_cache build_rank_cache( documents: Iterable[ContextDocument], *, split_identifiers: bool = False, ) -> RankCache Precompute term statistics for a corpus. Pass the result to rank() on subsequent queries over the same corpus. The caller owns invalidation; see RankCache. Parameters: Name Type Description Default documents Iterable[ContextDocument] The corpus. required split_identifiers bool Tokenize compound identifiers into their parts as well. Must match the setting ranking will use; rank() rebuilds a cache that disagrees. False Returns: Type Description RankCache The statistics. anyinfer.context.tokenize tokenize( text: str, *, split_identifiers: bool = False ) -> list[str] Split text into lowercase alphanumeric terms, dropping stop words. ASCII-only by design — the tradeoff is stated in the module documentation. Parameters: Name Type Description Default text str What to tokenize. required split_identifiers bool Also emit the parts of compound identifiers. resolveCredentials and resolve_credentials both yield the compound and resolve and credentials, so a query written in words matches an identifier written in code. The compound is kept as well, so an exact match on the full identifier still scores highest. False Returns: Type Description list[str] The terms, in order of appearance. anyinfer.context.expand_query expand_query( query: str, documents: Sequence[ContextDocument], *, cache: RankCache, tuning: ContextTuning = DEFAULT_TUNING, ) -> dict[str, float] Add distinctive terms from the strongest documents to the query. Pseudo-relevance feedback: rank once against the query as written, take the top documents on faith, and harvest the terms that make them distinctive — high frequency within that set, low frequency across the corpus. Those terms join the query at a reduced weight and everything is ranked again. This is the lexical answer to vocabulary mismatch. It has no index and no model, and it finds a file that says "login" from a query that says "authentication" whenever some document in the corpus uses both. It also inherits the classic failure mode: if the top documents are wrong, expansion makes them wronger, which is why expansion_weight defaults well below one. Parameters: Name Type Description Default query str The query as written. required documents Sequence[ContextDocument] The corpus. required cache RankCache Statistics for that corpus. required tuning ContextTuning Supplies expansion_terms, feedback_documents, and expansion_weight. DEFAULT_TUNING Returns: Type Description dict[str, float] Term weights, with the original query terms at full weight. Returns the dict[str, float] unexpanded terms when expansion is disabled, the query is empty, or the corpus dict[str, float] has nothing to harvest. anyinfer.context.salience salience( documents: Sequence[ContextDocument], *, tuning: ContextTuning = DEFAULT_TUNING, ) -> dict[str, float] Score documents by their centrality in the corpus's own import graph. An edge runs from a document to every document whose filename stem it imports. The stationary distribution of a damped random walk over those edges answers "what does this corpus depend on?", which is query-independent, and therefore what orders a corpus when the query is weak or missing entirely. Ranking with an empty query otherwise falls through to the path tie-break, which is arbitrary. Parameters: Name Type Description Default documents Sequence[ContextDocument] The corpus. required tuning ContextTuning Supplies salience_damping and salience_iterations. DEFAULT_TUNING Returns: Type Description dict[str, float] Path to a score in [0, 1], normalized so the most central document scores dict[str, float] one. An empty mapping when the corpus has no resolvable edges at all, so callers dict[str, float] can skip the blend entirely. Structure and Tiers anyinfer.context.detect_language detect_language(path: str) -> str | None Infer a language from a path's suffix, or None when it is ambiguous. anyinfer.context.structural_extract structural_extract( content: str, *, language: str | None ) -> str Reduce a document to its declarations, imports, and headings. Parameters: Name Type Description Default content str The document text. required language str | None The detected language. None yields no extract — guessing patterns for an unknown language produces noise, not a summary. required Returns: Type Description str The extract, or "" when none could be produced. Files under str SMALL_FILE_VERBATIM_BYTES are returned whole: they are already their own str summary. anyinfer.context.imported_names imported_names( content: str, *, language: str | None = None ) -> tuple[str, ...] Names a document imports, as bare identifiers. The corpus's own dependency graph is derivable from this: a name here that matches another document's filename stem is an edge. Used for query-independent centrality ranking, where "which files does everything else depend on?" is the whole question. Parameters: Name Type Description Default content str The document text. required language str | None Accepted for symmetry with the rest of this module and to allow future per-language refinement; the current extraction is language-agnostic. None Returns: Type Description str Distinct identifiers, in first-appearance order. Path separators, dots, and ... :: are all split, so from ..context.rank import score yields tuple[str, ...] context, rank, and score. anyinfer.context.is_generated_path is_generated_path(path: str) -> bool Whether a path looks machine-generated or vendored. Offered as a public helper because the decision belongs at collection time — the library never walks a filesystem, but an app deciding what to collect needs the same heuristic AnyInfer would have applied. anyinfer.context.module_surfaces module_surfaces( documents: Sequence[ContextDocument], *, depth: int = 2 ) -> dict[str, str] Group a corpus into modules and render each one's surface text. Offered as a public helper for the app-side digest recipe: generate a summary per module with your own client, cache it keyed on the surface's digest, and hand the results back as module_digests. Deterministic, so the cache key is stable. Parameters: Name Type Description Default documents Sequence[ContextDocument] The corpus. required depth int Path-prefix depth at which to group. 2 Returns: Type Description dict[str, str] Module path to concatenated extract (or content) text, in path order. anyinfer.context.DEFAULT_ROLLUP_SHARE module-attribute DEFAULT_ROLLUP_SHARE = 0.45 Share of the token and byte budget reserved for the module rollup. The default for ContextTuning.rollup_share, which is what actually applies. Compaction The fidelity between a structural extract and a whole file. anyinfer.context.compact_source compact_source( content: str, *, language: str | None = None, path: str | None = None, ) -> CompactSource Remove commentary and blank runs from a document. Parameters: Name Type Description Default content str The document text. required language str | None The language, from anyinfer.context.detect_language. Inferred from path when omitted. None path str | None Used to infer the language when one was not supplied. None Returns: Type Description CompactSource The CompactSource. An unknown language, or one with no commentary syntax, CompactSource yields the input with blank runs collapsed and nothing else touched — a safe CompactSource no-op is always better than a guess at foreign syntax. anyinfer.context.CompactSource dataclass CompactSource( text: str, original_lines: int, elided_lines: int ) A document with its commentary removed. Attributes: Name Type Description text str The compacted content. original_lines int Line count before compaction. elided_lines int How many lines were removed. Zero means compaction found nothing to drop, and text equals the input. is_reduced property is_reduced: bool Whether compaction actually removed anything. anyinfer.context.supports_compaction supports_compaction(language: str | None) -> bool Whether compaction knows how to shorten this language. Parameters: Name Type Description Default language str | None A language name from anyinfer.context.detect_language. required Returns: Type Description bool Whether compact_source can do better than returning its input. Chunking anyinfer.context.Chunk dataclass Chunk( document: ContextDocument, text: str, index: int, start_line: int, end_line: int, ) One span of a document. Attributes: Name Type Description document ContextDocument The document this came from. text str The span's text. index int Position within the document, from zero. start_line int First line of the span, 1-based and inclusive. end_line int Last line of the span, 1-based and inclusive. anyinfer.context.split_document split_document( document: ContextDocument, *, chunk_tokens: int = DEFAULT_CHUNK_TOKENS, ) -> list[Chunk] Split a document into boundary-aware chunks with line spans. Prefers the last blank line within budget, falls back to the last line break, and hard-cuts only when neither lands past a quarter of the budget. Parameters: Name Type Description Default document ContextDocument The document to split. required chunk_tokens int Target chunk size in planning tokens. DEFAULT_CHUNK_TOKENS Returns: Type Description list[Chunk] Chunks in document order. A document shorter than one chunk yields exactly one. anyinfer.context.DEFAULT_CHUNK_TOKENS module-attribute DEFAULT_CHUNK_TOKENS = 512 Target chunk size for sub-document splitting, shared by packed and distill. Large enough for a whole function, small enough to pack several. The canonical home for this default; anyinfer.context.pack imports it rather than redefining it, since a chunk size chosen for packed's splitter is the same number ContextTuning.chunk_tokens means for distill. Distillation The only reduction that spends inference. See distill a corpus for the cookbook. anyinfer.context.distill The distill strategy: map/reduce a corpus that will never fit. The other strategies decide what to drop. This one reads everything and writes something shorter: each chunk is summarized against the query (the map phase), then the notes are synthesized into one answer (the reduce phase). It is separated from anyinfer.context.select by construction, because it is the one strategy that spends money. It takes your client, issues real generation calls, and reports the count and aggregate usage so the multiplier is never a surprise. Two properties distinguish this from a naive map/reduce. Reduction is hierarchical: if the map notes together exceed the target's window, they are reduced in batches and the batch summaries reduced again, rather than being sent in one overflowing request. And a deterministic reducer can replace the reduce call entirely, so an application that merges structurally pays for the map phase only. Prompts here are mechanical scaffolding — "here is chunk 3 of 9, take notes" — not application prose. You own the question; override map_instructions and reduce_instructions to own the framing too. DEFAULT_CONCURRENCY module-attribute DEFAULT_CONCURRENCY = 4 Map calls in flight at once. Bounded because a fan-out is someone's rate limit. SupportsGenerate Bases: Protocol The slice of an async client distill() needs. A structural protocol rather than an import, so this subpackage never depends on the client — anyinfer.AsyncClient satisfies it as-is. generate async generate( messages: Any, *, target: str, **kwargs: Any ) -> Generation Generate one result. budget budget( messages: Any, *, target: str, **kwargs: Any ) -> ContextBudget Compute a context budget without issuing a request. SupportsGenerateSync Bases: Protocol The synchronous mirror of SupportsGenerate, satisfied by anyinfer.Client. generate generate( messages: Any, *, target: str, **kwargs: Any ) -> Generation Generate one result. budget budget( messages: Any, *, target: str, **kwargs: Any ) -> ContextBudget Compute a context budget without issuing a request. Distillation dataclass Distillation( text: str, chunk_count: int, calls: int, usage: Usage, reduce_depth: int = 1, notes: tuple[str, ...] = (), ) What a distillation produced, and what it cost. Attributes: Name Type Description text str The synthesized answer. chunk_count int How many chunks the source was split into. calls int Total generation calls spent, map and reduce together. This is the multiplier over a single request. usage Usage Merged usage across every call, including cost when providers report it. reduce_depth int 1 for a single-pass reduce; higher when notes were reduced in batches and the batch summaries reduced again. notes tuple[str, ...] The intermediate map outputs. Payload-bearing — excluded from repr and never placed in telemetry. summary summary() -> str A one-line, content-free description of the run. event event(*, max_tokens: int) -> ContextReduced Build the telemetry event describing this distillation. distill async distill( source: str | Iterable[ContextDocument], query: str, *, client: SupportsGenerate, target: str, max_output_tokens: int = 1024, chunk_tokens: int | None = None, concurrency: int = DEFAULT_CONCURRENCY, map_instructions: str | None = None, reduce_instructions: str | None = None, reducer: Callable[[Sequence[str]], str] | None = None, observer: Observer | None = None, ) -> Distillation Summarize a corpus larger than the window by mapping and reducing over it. Parameters: Name Type Description Default source str | Iterable[ContextDocument] Raw text, or documents. Documents split per document, because a document boundary is a natural chunk boundary. required query str What the summary should answer. required client SupportsGenerate Anything satisfying SupportsGenerate — normally an anyinfer.AsyncClient. required target str Where to send the calls. required max_output_tokens int Ceiling on the final answer. 1024 chunk_tokens int | None Chunk size. Derived from the target's remaining budget when omitted. None concurrency int Map calls in flight at once. DEFAULT_CONCURRENCY map_instructions str | None Replaces the default note-taking instruction. None reduce_instructions str | None Replaces the default synthesis instruction. None reducer Callable[[Sequence[str]], str] | None Merge the notes deterministically instead of with a reduce call. Saves every reduce call, and makes the merge reproducible. None observer Observer | None Receives a ContextReduced event when the run finishes. None Returns: Type Description Distillation The Distillation. Raises: Type Description ConfigError When chunk_tokens is omitted and the target's context window is unknown. An unknown window stays unknown — the caller chooses the number. distill_sync distill_sync( source: str | Iterable[ContextDocument], query: str, *, client: SupportsGenerateSync, target: str, max_output_tokens: int = 1024, chunk_tokens: int | None = None, map_instructions: str | None = None, reduce_instructions: str | None = None, reducer: Callable[[Sequence[str]], str] | None = None, observer: Observer | None = None, ) -> Distillation Run distill() sequentially against a synchronous client. Chunks are processed one at a time: concurrency is the async path's feature, and a sync caller that wants it should use distill() with an anyinfer.AsyncClient. Args and returns are as distill(), minus concurrency. Raises: Type Description ConfigError When chunk_tokens is omitted and the window is unknown. anyinfer.context.distill_sync distill_sync( source: str | Iterable[ContextDocument], query: str, *, client: SupportsGenerateSync, target: str, max_output_tokens: int = 1024, chunk_tokens: int | None = None, map_instructions: str | None = None, reduce_instructions: str | None = None, reducer: Callable[[Sequence[str]], str] | None = None, observer: Observer | None = None, ) -> Distillation Run distill() sequentially against a synchronous client. Chunks are processed one at a time: concurrency is the async path's feature, and a sync caller that wants it should use distill() with an anyinfer.AsyncClient. Args and returns are as distill(), minus concurrency. Raises: Type Description ConfigError When chunk_tokens is omitted and the window is unknown. anyinfer.context.Distillation dataclass Distillation( text: str, chunk_count: int, calls: int, usage: Usage, reduce_depth: int = 1, notes: tuple[str, ...] = (), ) What a distillation produced, and what it cost. Attributes: Name Type Description text str The synthesized answer. chunk_count int How many chunks the source was split into. calls int Total generation calls spent, map and reduce together. This is the multiplier over a single request. usage Usage Merged usage across every call, including cost when providers report it. reduce_depth int 1 for a single-pass reduce; higher when notes were reduced in batches and the batch summaries reduced again. notes tuple[str, ...] The intermediate map outputs. Payload-bearing — excluded from repr and never placed in telemetry. summary summary() -> str A one-line, content-free description of the run. event event(*, max_tokens: int) -> ContextReduced Build the telemetry event describing this distillation. anyinfer.context.SupportsGenerate Bases: Protocol The slice of an async client distill() needs. A structural protocol rather than an import, so this subpackage never depends on the client — anyinfer.AsyncClient satisfies it as-is. generate async generate( messages: Any, *, target: str, **kwargs: Any ) -> Generation Generate one result. budget budget( messages: Any, *, target: str, **kwargs: Any ) -> ContextBudget Compute a context budget without issuing a request. Rendering The envelope format, exposed for applications that parse reduced context back out of stored transcripts. anyinfer.context.render_corpus render_corpus(blocks: Iterable[str]) -> str Wrap rendered blocks in the corpus element. anyinfer.context.render_file_block render_file_block(document: ContextDocument) -> str Render a whole document. anyinfer.context.render_extract_block render_extract_block(document: ContextDocument) -> str Render a document's structural extract. anyinfer.context.render_chunk_block render_chunk_block( document: ContextDocument, text: str, start_line: int, end_line: int, ) -> str Render one contiguous span of a document, with its line range. anyinfer.context.render_compact_block render_compact_block( document: ContextDocument, text: str, *, elided_lines: int, ) -> str Render a document with its commentary removed, saying how much was removed. A distinct element from render_file_block on purpose: a reader must be able to tell a whole file from a shortened one, and elided_lines makes the shortening a number rather than an impression. anyinfer.context.render_duplicate_block render_duplicate_block( path: str, canonical: str, *, identical: bool ) -> str Render a pointer from a collapsed document to the one that represents it. identical="true" means byte-for-byte, and nothing was lost. "false" means the documents were merely similar above the configured threshold, and this one's differences are not in the envelope — a real loss of fidelity, stated rather than implied. anyinfer.context.ENVELOPE_FORMAT module-attribute ENVELOPE_FORMAT = 1 Version stamped on every rendered wrapper. Bumped when an existing element's meaning changes, not when a new one is added: a reader that ignores unknown elements keeps working across additions, which is the point of declaring the version at all. --- # Reference / Telemetry and Redaction Source: https://anyinfer.dev/reference/api/telemetry/ Telemetry and Redaction Typed in-process events, payload-free by default, plus the redaction registry that keeps secrets out of everything. Concepts: telemetry · credentials and redaction. Observing anyinfer.Observer Bases: Protocol A telemetry sink. Implementations receive every event the client emits. Keep on_event fast and non-blocking; queue work elsewhere if it might be slow. on_event on_event(event: TelemetryEvent) -> None Handle one telemetry event. anyinfer.TelemetryEvent module-attribute TelemetryEvent = ( RequestStarted | ArenaCompleted | TargetResolved | AttemptStarted | FirstToken | AttemptCompleted | RetryScheduled | FallbackTriggered | RepairAttempted | RequestCompleted | RequestFailed | ParameterDropped | UsageEstimated | ServerLifecycle | DownloadProgress | ContextReduced | ProviderDiagnostic | CachePlanned | RateLimitWaited | RateLimitObserved ) Any event an observer may receive. Request Lifecycle Events anyinfer.RequestStarted dataclass RequestStarted( request_id: str, targets: tuple[Target, ...], metadata: Mapping[str, str] = dict(), prompt_text: str | None = None, operation: InferenceOperation = "generation", ) An inference request entered the router. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. targets tuple[Target, ...] The fallback chain as requested, in the order the router will try it. metadata Mapping[str, str] Caller-supplied labels from the request, passed through untouched. prompt_text str | None The prompt text; None unless the receiving observer registered with payloads=True. Generation only — embedding inputs and rerank documents are never carried on events. operation InferenceOperation Which inference operation this request is. Defaults to "generation" so pre-existing consumers observe no change; the embed and rerank dispatchers stamp their own value. anyinfer.TargetResolved dataclass TargetResolved(request_id: str, target: ResolvedTarget) A target string resolved to a concrete provider and model. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The concrete provider and model the target string resolved to. anyinfer.AttemptStarted dataclass AttemptStarted( request_id: str, target: ResolvedTarget, attempt_number: int, ) An attempt against one resolved target began. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target being attempted. attempt_number int 1-based count of attempts against this target; retries increment it, and falling back to a new target restarts it at 1. anyinfer.FirstToken dataclass FirstToken( request_id: str, target: ResolvedTarget, at_ms: float ) The first content delta arrived — the centrally-measured TTFT. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target that produced the token. at_ms float Milliseconds from attempt start to the first content delta. anyinfer.AttemptCompleted dataclass AttemptCompleted( request_id: str, target: ResolvedTarget, usage: Usage, timing: Timing, finish_reason: str, ) An attempt finished successfully. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target that served the attempt. usage Usage Token usage for the attempt, merged across every report the provider sent. timing Timing Centrally-measured attempt timings, comparable across providers. finish_reason str Normalized reason generation stopped, e.g. stop or length. anyinfer.RetryScheduled dataclass RetryScheduled( request_id: str, target: ResolvedTarget, attempt_number: int, delay_s: float, error: ErrorInfo, ) A retryable failure will be retried against the same target after a delay. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target that will be retried. attempt_number int The 1-based attempt that just failed; the retry is the next one. delay_s float Seconds the router sleeps before the retry. error ErrorInfo Snapshot of the retryable failure. anyinfer.FallbackTriggered dataclass FallbackTriggered( request_id: str, from_target: ResolvedTarget, to_target: Target, error: ErrorInfo | None = None, ) A target was abandoned; the router advanced to the next in the chain. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. from_target ResolvedTarget The resolved target that was abandoned. to_target Target The next target string the router will try, not yet resolved. error ErrorInfo | None Snapshot of the failure that caused the abandonment, or None when the switch was a content-policy redirect rather than an error. anyinfer.RepairAttempted dataclass RepairAttempted( request_id: str, target: ResolvedTarget, attempt_number: int, mechanism: Mechanism | None, errors: tuple[str, ...] = (), raw_text: str | None = None, ) A schema violation triggered a repair re-prompt. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target being re-prompted. attempt_number int 1-based count of repair attempts within this generation. mechanism Mechanism | None The structured-output mechanism in force when validation failed, when one was chosen. errors tuple[str, ...] The schema-validation messages that triggered the repair. raw_text str | None The response text that failed validation; None unless the receiving observer registered with payloads=True. anyinfer.ParameterDropped dataclass ParameterDropped( request_id: str, target: ResolvedTarget, parameter: str, reason: str, ) A requested parameter was not honored as asked, because the target cannot. Dropping a parameter silently is how a caller ends up debugging why temperature=0 had no effect. Every drop is observable instead. The same applies to a parameter honored only in part — a repair budget clamped to a provider's ceiling is reported here too, since a budget quietly reduced from three to one is no more discoverable than one ignored outright. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target the parameter was withheld from. parameter str Name of the request parameter that was not honored, dotted for a field of a compound one (repair.max_attempts). reason str Human-readable explanation of what the target did instead. anyinfer.ProviderDiagnostic dataclass ProviderDiagnostic( target: ResolvedTarget | None, diagnostic: Diagnostic, request_id: str | None = None, ) A provider reported something about its own runtime. Emitted after an attempt for providers that declare reports_diagnostics, and whenever diagnostics() is called directly. The same text also lands on Generation.warnings, so a caller reading only results still sees it; this event is for observers that want it correlated with the request that hit it. Attributes: Name Type Description target ResolvedTarget | None The resolved target whose runtime is being described, or None when the diagnostic was collected outside a request. diagnostic Diagnostic What the provider reported. request_id str | None Correlation id, or None when collected outside a request — in which case the OTel bridge records it as a standalone span. anyinfer.ContextReduced dataclass ContextReduced( strategy: str, representation: str, candidate_count: int, selected_count: int, omitted_count: int, estimated_tokens: int, max_tokens: int, binding_constraints: tuple[str, ...] = (), calls: int = 0, ) Context was reduced to fit a budget. Emitted for a reduced document corpus and for a compacted message history alike — both emulate a larger context window, and emulation is observable rather than silent. Content-free by construction: counts and ceilings only, never paths, document text, or message text — a path name can itself be sensitive. Attributes: Name Type Description strategy str The strategy requested (auto stays auto), or history for a compacted conversation. representation str The strategy actually applied. candidate_count int Documents, or messages — offered to the reducer. selected_count int Documents represented at detail fidelity, or messages kept. omitted_count int Documents not represented in detail, or messages dropped. estimated_tokens int Planning-side estimate of the rendered envelope, or of the compacted conversation. max_tokens int The budget the reduction was held to. binding_constraints tuple[str, ...] Which ceilings excluded at least one document. calls int Generation calls spent; non-zero only for distill. anyinfer.ArenaCompleted dataclass ArenaCompleted( request_id: str, target_count: int, strategy: str, agreement: int | None, calls: int, memoized_tool_calls: int, synthesized: bool, ) A bounded multi-target arena finished, without carrying answer content. anyinfer.CachePlanned dataclass CachePlanned( request_id: str, target: ResolvedTarget, mechanism: str, mark_count: int, estimated_cacheable_tokens: int, ) The core decided how to engage a target's prompt cache. Emitted only when a policy was in force and the target offered a mechanism. A policy that found nothing to use reports a ParameterDropped instead, because the interesting fact there is the degradation, not the plan. Content-free: counts and mechanism only. What was marked is a position, never text. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target the plan applies to. mechanism str explicit for per-segment marks, implicit for prefix stability. mark_count int How many marks were placed; always zero for implicit. estimated_cacheable_tokens int Planning-side size of what the plan tries to cache. An intention, not a saving — realized savings come only from the provider's own reported usage. anyinfer.RateLimitWaited dataclass RateLimitWaited( request_id: str, provider_id: str, waited_s: float, reason: Literal[ "concurrency", "interval", "provider-headers" ], target: ResolvedTarget | None = None, ) A request was held back by client-side pacing before it was dispatched. Emitted so a paced request never looks like a slow provider. The same wait also lands in the attempt's timing.phases["queued_ms"], because latency a caller cannot attribute is a support ticket. Attributes: Name Type Description request_id str Correlation id of the waiting request; empty when the wait happened outside a tracked generation, as on a model listing. provider_id str The provider instance whose limiter did the waiting. waited_s float How long the request was held. reason Literal['concurrency', 'interval', 'provider-headers'] concurrency for an in-flight bound, interval for a configured rate or minimum gap, provider-headers for a window the provider itself reported. target ResolvedTarget | None The resolved target, when the wait belongs to a generation attempt. anyinfer.RateLimitObserved dataclass RateLimitObserved( provider_id: str, requests_remaining: int | None = None, tokens_remaining: int | None = None, resets_in_s: float | None = None, ) A provider reported its rate-limit state on a response. Emitted at most once per response, and only when the provider declared a header dialect and actually populated it. Purely what the provider said — this library adds no estimate of its own, and a field the provider left out stays None rather than becoming a guess. Attributes: Name Type Description provider_id str The provider instance that reported. requests_remaining int | None Requests left in the current window, when stated. tokens_remaining int | None Tokens left in the current window, when stated. resets_in_s float | None Seconds until the window resets, when stated in a form that can be read as a duration. anyinfer.UsageEstimated dataclass UsageEstimated( request_id: str, target: ResolvedTarget, field_name: str, method: str, ) A usage figure was derived rather than reported by the provider. Estimated and reported numbers must never be indistinguishable downstream; this event is what marks the difference for observers. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target the estimate applies to. field_name str The usage field that was estimated, e.g. input_tokens. method str How the estimate was derived. anyinfer.RequestCompleted dataclass RequestCompleted( request_id: str, target: ResolvedTarget, usage: Usage, timing: Timing, repair_attempts: int = 0, response_text: str | None = None, ) A request produced a result. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. target ResolvedTarget The resolved target that produced the result. usage Usage Final token usage for the request. timing Timing Final centrally-measured timings for the request. repair_attempts int How many repair re-prompts were needed; 0 means the first response validated. response_text str | None The final response text; None unless the receiving observer registered with payloads=True. anyinfer.RequestFailed dataclass RequestFailed(request_id: str, error: ErrorInfo) A request exhausted its route without a result. Attributes: Name Type Description request_id str Correlation id shared by every event this request emits. error ErrorInfo Snapshot of the terminal failure, after every target and retry was spent. Local Subsystem Events anyinfer.DownloadProgress dataclass DownloadProgress( artifact_id: str, downloaded_bytes: int, total_bytes: int | None, done: bool = False, phase: str = "", file_index: int = 0, file_count: int = 0, filename: str = "", session_bytes: int = 0, ) Progress of a model acquisition. downloaded_bytes and total_bytes are aggregate across the whole acquisition, which is what their names have always implied. Earlier builds reported them per file, so a sharded artifact restarted the counter at zero on every shard with no way for an observer to tell. The remaining fields carry the per-file detail that the aggregate figures deliberately no longer mix in. Attributes: Name Type Description artifact_id str The artifact or catalog variant being acquired. downloaded_bytes int Bytes present across every file, including bytes that were already on disk before this run, so resuming reports the resumed position. total_bytes int | None Total expected bytes, or None when a size is genuinely unknown. done bool Whether the acquisition finished. phase str Which stage emitted this, when the acquisition engine supplied one. file_index int 1-based index of the file that most recently advanced. file_count int How many files this acquisition covers. filename str Name of that file. session_bytes int Bytes this run actually transferred, as opposed to resumed. anyinfer.ServerLifecycle dataclass ServerLifecycle( server_id: str, state: Literal[ "starting", "ready", "stopping", "stopped", "crashed", ], detail: str = "", ) A supervised local server changed state. Attributes: Name Type Description server_id str Identifies the supervised server — the key of the model it serves. state Literal['starting', 'ready', 'stopping', 'stopped', 'crashed'] The state the server just entered. detail str Human-readable context, such as a stop reason or a tail of crash output; empty when there is nothing to add. Redaction anyinfer.RedactionRegistry RedactionRegistry() A thread-safe set of secrets to strip from outbound strings. register register(secret: str | None) -> None Register a secret for redaction. Values shorter than MIN_SECRET_LEN are ignored: redacting them would corrupt unrelated text far more often than it would protect anything. redact redact(text: str) -> str Replace every registered secret in text with REDACTED. clear clear() -> None Forget all registered secrets. Intended for tests. __len__ __len__() -> int Number of registered secrets. anyinfer.register_secret register_secret(secret: str | None) -> None Register a secret with the process-wide registry. anyinfer.redact redact(text: str) -> str Redact registered secrets from text using the process-wide registry. OpenTelemetry Export The optional [otel] extra maps these events onto OpenTelemetry spans and metrics. Guide: observability. anyinfer.otel.install install( client: Any, *, record_payloads: bool = False ) -> OTelObserver Attach an OTelObserver to a client. Parameters: Name Type Description Default client Any An AsyncClient or Client. required record_payloads bool Attach prompt and response text to spans. False Returns: Type Description OTelObserver The observer, so it can be detached later. anyinfer.otel.OTelObserver OTelObserver( *, tracer: Any = None, meter: Any = None, record_payloads: bool = False, ) Maps AnyInfer telemetry events onto OpenTelemetry spans and metrics. One span per request, with attempts as span events. Requests and attempts are correlated by request_id, so a fallback chain reads as a single trace rather than several disconnected ones. Parameters: Name Type Description Default tracer Any An OpenTelemetry tracer. Defaults to one from the global provider. None meter Any An OpenTelemetry meter. Defaults to one from the global provider. None record_payloads bool Attach prompt and response text to spans. Off by default, matching the payload-free default of the event contract itself. Subscribe with payloads=True as well for this to have any effect. False Raises: Type Description ConfigError If opentelemetry-api is not installed. on_event on_event(event: TelemetryEvent) -> None Handle one telemetry event. Never raises: the dispatcher isolates observer failures, but a telemetry bridge that can break a generation would be a poor trade regardless. anyinfer.otel.GEN_AI module-attribute GEN_AI = 'gen_ai' Prefix of the GenAI semantic-convention attribute namespace. --- # Reference / Registry, Catalog, Credentials Source: https://anyinfer.dev/reference/api/registry/ Registry, Catalog, and Credentials How providers describe themselves (frozen descriptors, declarative setup specs), how targets and aliases resolve, and how credential references become secrets. Concepts: targets and aliases · credentials. Provider Registry anyinfer.ProviderRegistry ProviderRegistry( *, load_builtins: bool = True, load_entry_points: bool = True, ) Maps provider ids and aliases to descriptors, rejecting collisions. Built-in providers are registered on first use. Entry-point providers are discovered lazily on the first lookup that misses, so installing a provider package is enough to make it resolvable. register register( descriptor: ProviderDescriptor, *, replace: bool = False ) -> None Register a descriptor. Parameters: Name Type Description Default descriptor ProviderDescriptor The provider to register. required replace bool Allow replacing an existing registration of the same id. Aliases are still checked against other providers. False Raises: Type Description ConfigError On a duplicate id or an alias already claimed by another provider. unregister unregister(provider_id: str) -> None Remove a provider and its aliases. Unknown ids are ignored. resolve_alias resolve_alias(name: str) -> str Resolve a provider name or alias to a canonical provider id. Raises: Type Description ConfigError If no provider claims the name. get get(provider_id: str) -> ProviderDescriptor Look up a descriptor by id or alias. Raises: Type Description ConfigError If no provider claims the name. has has(provider_id: str) -> bool Whether a provider with this id or alias is registered. known_ids known_ids() -> tuple[str, ...] Every registered canonical provider id, sorted. __iter__ __iter__() -> Iterator[ProviderDescriptor] Iterate descriptors in canonical-id order. plugin_issues plugin_issues() -> tuple[PluginLoadIssue, ...] Third-party entry points that did not become usable providers. Empty when every installed provider package loaded, which is the ordinary case. This call itself triggers discovery if it has not already run, so callers get a complete answer without touching the registry first. anyinfer.ProviderDescriptor dataclass ProviderDescriptor( id: str, display_name: str, factory: AdapterFactory, aliases: tuple[str, ...] = (), locality: Literal[ "hosted", "local", "remote" ] = "hosted", default_base_url: str | None = None, requires_base_url: bool = False, setup: ProviderSetupSpec = ProviderSetupSpec(), reasoning_translator: ReasoningTranslator = _no_reasoning, static_capabilities: Mapping[ str, ModelCapabilities ] = dict(), default_capabilities: ModelCapabilities = ModelCapabilities(), operations: frozenset[InferenceOperation] = frozenset( {"generation"} ), static_embedding_capabilities: Mapping[ str, EmbeddingCapabilities ] = dict(), static_rerank_capabilities: Mapping[ str, RerankCapabilities ] = dict(), token_calibration: TokenCalibration = TokenCalibration(), rate_limit_headers: RateLimitHeaders = RateLimitHeaders(), governs_own_transport: bool = False, supports_sessions: bool = False, model_puller: ModelPuller | None = None, model_inventory: Literal[ "available", "installed", "served" ] = "served", uses_catalog: bool = False, reports_diagnostics: bool = False, cache_mechanism: CacheMechanism | None = None, cache_max_marks: int = 0, cache_min_tokens: int = 0, grammar_needs_prompt_injection: bool = False, max_repair_attempts: int | None = None, ignored_parameters: tuple[str, ...] = (), derived_from: str | None = None, ) Declarative facts about a provider and how to instantiate its adapter. id instance-attribute id: str Canonical provider id — the provider half of a provider:model target. Normalized for lookup: lowercased, stripped, underscores to hyphens. display_name instance-attribute display_name: str Human-readable name for UIs and error messages. factory instance-attribute factory: AdapterFactory Builds this provider's adapter instance from its resolved configuration. aliases class-attribute instance-attribute aliases: tuple[str, ...] = () Alternative names that resolve to this provider; each must be globally unique across the registry. locality class-attribute instance-attribute locality: Literal['hosted', 'local', 'remote'] = 'hosted' Where inference physically happens. local means "on this machine" and carries two consequences — genuine zero pricing, and hardware detection that describes the right computer. remote is the third case a descriptor can never state on its own: an engine that is normally local, reached over a network. A client downgrades local to remote when the configured base URL is not loopback, because stamping zero cost on someone else's metered proxy, or sizing models against the wrong machine's RAM, are both silent wrong answers. default_base_url class-attribute instance-attribute default_base_url: str | None = None Endpoint used when settings supply no base URL; None when there is no sensible default, as with per-tenant or supervised endpoints. requires_base_url class-attribute instance-attribute requires_base_url: bool = False Whether the adapter cannot be built without a configured base URL. setup class-attribute instance-attribute setup: ProviderSetupSpec = ProviderSetupSpec() Declarative description of the configuration this provider needs, which is what a config UI renders. reasoning_translator class-attribute instance-attribute reasoning_translator: ReasoningTranslator = _no_reasoning Maps normalized reasoning effort onto this provider's wire fields. The default translates every effort to nothing, for providers without a reasoning control. static_capabilities class-attribute instance-attribute static_capabilities: Mapping[str, ModelCapabilities] = ( field(default_factory=dict) ) Per-model capabilities declared ahead of time, keyed by model id; layered over default_capabilities when capabilities are assembled. default_capabilities class-attribute instance-attribute default_capabilities: ModelCapabilities = ( ModelCapabilities() ) Capabilities assumed for any model without a more specific source. operations class-attribute instance-attribute operations: frozenset[InferenceOperation] = frozenset( {"generation"} ) Which inference operations this provider's adapter implements. Defaults to generation-only, which is every existing adapter's actual behavior — this field is purely additive. A provider declaring "embedding" must build an adapter satisfying anyinfer.providers.base.EmbedsText; the client checks this when the adapter is first constructed rather than trusting the declaration blindly. static_embedding_capabilities class-attribute instance-attribute static_embedding_capabilities: Mapping[ str, EmbeddingCapabilities ] = field(default_factory=dict) Per-model embedding capabilities declared ahead of time, keyed by model id. Populated only for models the provider actually embeds with; empty for a generation-only provider. static_rerank_capabilities class-attribute instance-attribute static_rerank_capabilities: Mapping[ str, RerankCapabilities ] = field(default_factory=dict) Per-model rerank capabilities declared ahead of time, keyed by model id. Populated only for models the provider actually reranks with; empty for a generation-only provider. token_calibration class-attribute instance-attribute token_calibration: TokenCalibration = TokenCalibration() How much this provider's transport inflates the prompt it is billed for. Declared here rather than measured per request because it is a property of the provider's envelope, not of any one call: a session API that wraps the caller's messages in its own harness charges that harness on every request. The default is the identity — the provider counts what it was sent, and only a provider with evidence of a systematic gap should declare otherwise. rate_limit_headers class-attribute instance-attribute rate_limit_headers: RateLimitHeaders = RateLimitHeaders() Which response headers this provider reports its rate-limit state in. Empty by default, which means client-side pacing for this provider can only honour the bounds its caller configured. Declaring a dialect is what lets pacing anticipate the provider's own window instead — and, like every other wire fact here, it belongs in the provider's contract snapshot with a verified date. governs_own_transport class-attribute instance-attribute governs_own_transport: bool = False Whether this provider builds its own transport rather than taking the core's. True for adapters that talk through a vendor SDK or an interactive session instead of an httpx2 client the core constructed. The core cannot wrap what it did not build, so such a provider gets concurrency pacing only, applied around the call, and reports a dropped parameter if the caller asked for header-driven pacing it cannot perform. supports_sessions class-attribute instance-attribute supports_sessions: bool = False Whether the provider can keep state between requests — a session API, or keep-alive model residency, rather than treating every request as independent. model_puller class-attribute instance-attribute model_puller: ModelPuller | None = None How this provider is told to make a model available, or None when it cannot be. For engines that keep their own model store, registry, and downloader — Ollama — the useful operation is not download these weights but make yourself ready. The implementation lives in anyinfer.local.services and is merely pointed at from here, both because acquisition never belongs in an adapter and because a declared hook keeps "which providers can do this" answerable from the registry rather than from a chain of engine checks in the core. Weights fetched this way land in the engine's store under the engine's own name. Nothing is written to AnyInfer's model store and nothing is indexed there, so locate_model() will not find them — they are not ours to find. model_inventory class-attribute instance-attribute model_inventory: Literal[ "available", "installed", "served" ] = "served" What list_models() means for model-management UIs. available is a catalog of things that could be run, installed is a provider-owned on-disk store, and served is the set an already-running engine exposes. The distinction prevents an application from presenting every catalog entry as though it were already installed. uses_catalog class-attribute instance-attribute uses_catalog: bool = False Whether the core should supply its active catalog to this adapter. Declared here so catalog composition is a provider fact, not a provider-id branch in client construction. Supervised engines use it to resolve model references while ordinary protocol adapters leave it false. reports_diagnostics class-attribute instance-attribute reports_diagnostics: bool = False Whether this provider's adapter implements SupportsDiagnostics. Declared rather than probed for, so "which providers can tell me about their runtime" is answerable from the registry alone. The core only calls diagnostics() on a provider that advertises it here. cache_mechanism class-attribute instance-attribute cache_mechanism: CacheMechanism | None = None How this provider's prompt cache is engaged, or None when it offers nothing. explicit means the wire format accepts per-segment cache marks and the adapter knows how to spell one. implicit means the provider caches stable prefixes by itself, so there is nothing to send and the core's whole duty is to leave the prefix alone. Declared here rather than inferred, because "does this provider cache" is a protocol fact recorded in its contract snapshot, not something to probe for. cache_max_marks class-attribute instance-attribute cache_max_marks: int = 0 Most explicit cache marks this provider accepts per request; 0 when it takes none. Exceeding a provider's ceiling is an error on some APIs and silently ignored on others, so the core clamps to this and reports the clamp. cache_min_tokens class-attribute instance-attribute cache_min_tokens: int = 0 Smallest segment this provider will actually cache, in tokens. Below its own floor a provider bills a cache write and then never serves a read from it, so a mark placed there costs money and saves none. 0 means the provider states no floor. grammar_needs_prompt_injection class-attribute instance-attribute grammar_needs_prompt_injection: bool = False Whether grammar mode also requires the schema in the prompt. True for engines that compile the schema to a decoding grammar without conditioning the model on it (llama.cpp, and Ollama's format): the grammar guarantees well-formed JSON but not meaningful JSON unless the model was told the shape. max_repair_attempts class-attribute instance-attribute max_repair_attempts: int | None = None The most schema-repair round trips this provider may be asked for, or None for no provider-imposed ceiling. The repair budget is the caller's to set, and for almost every provider it should stay that way. A few cannot honor it: a provider whose every request is slow, interactively authenticated, or metered per conversation turn makes a second repair attempt cost far more than the malformed answer is worth, and one that keeps server-side conversation state is unlikely to answer a re-ask differently anyway. Such a provider says so here, and the core clamps to it — visibly, as a ParameterDropped event, since a budget quietly reduced from three to one is exactly the kind of degradation this library refuses to perform in silence. ignored_parameters class-attribute instance-attribute ignored_parameters: tuple[str, ...] = () Request parameters this provider accepts and silently discards. Distinct from "rejects with an error" and from "supported": a silently-ignored parameter looks like success while doing nothing, so the core reports it as a ParameterDropped event instead of letting it pass unnoticed. derived_from class-attribute instance-attribute derived_from: str | None = None The engine this descriptor is an instance of, when it is one. An application that configures two Azure tenants or two OpenAI-compatible endpoints gives each instance its own id (work-azure, ollama-local); each becomes a descriptor derived from the underlying engine's, differing only in identity. None means this descriptor is an engine, which is the ordinary case. identifiers property identifiers: tuple[str, ...] The id plus every alias, all normalized. anyinfer.ProviderSetupSpec dataclass ProviderSetupSpec( fields: tuple[SetupField, ...] = (), model_selection: Literal[ "discover-or-manual", "manual-only" ] = "discover-or-manual", host_shorthand: HostShorthand | None = None, any_of: tuple[tuple[str, ...], ...] = (), requirement_note: str = "", ) Everything a config UI needs to configure a provider without knowing which it is. fields class-attribute instance-attribute fields: tuple[SetupField, ...] = () The provider's configurable fields, in the order a UI should present them. model_selection class-attribute instance-attribute model_selection: Literal[ "discover-or-manual", "manual-only" ] = "discover-or-manual" Whether a UI may offer models discovered from the endpoint, or must let the user type a model id because the provider cannot enumerate what it serves. host_shorthand class-attribute instance-attribute host_shorthand: HostShorthand | None = None Expansion rule applied when a bare hostname is entered as the base URL, or None when only full URLs make sense for this provider. any_of class-attribute instance-attribute any_of: tuple[tuple[str, ...], ...] = () Groups of field keys of which at least one must be supplied. Some providers accept a choice of credential rather than a fixed one — Anthropic takes either an API key or a claude.ai OAuth token. That is a constraint over a group, so no per-field required flag can express it: marking both required demands both, and marking neither required lets an unconfigured instance save cleanly. Each inner tuple names the keys in one such group. requirement_note class-attribute instance-attribute requirement_note: str = '' One line explaining the spec's requirements, shown beneath the fields. Carried here rather than assembled by the UI because only the provider knows why its any_of groups exist; a generated sentence would say what is required without ever saying what the alternatives mean. essential_fields property essential_fields: tuple[SetupField, ...] The fields a UI should put in front of the user, in declared order. Everything the provider cannot supply a sensible answer for: credentials, and the endpoints and identifiers that vary per account. This is the short list an application prompts for. advanced_fields property advanced_fields: tuple[SetupField, ...] The fields that already have a standard value, in declared order. Offer them — a base URL override is what makes a proxy or a mirror usable — but offer them folded away, since changing one is the rare case rather than the setup path. __post_init__ __post_init__() -> None Reject a spec that hides a field a user cannot skip. Marking a field both mandatory and advanced asks a UI to do two contradictory things, and the resolution it usually picks — honor the disclosure — produces the one failure mode worth designing against: a save that refuses, naming a field that is not on screen. Caught here, at import time, so it is a provider-authoring error rather than a user's dead end. unsatisfied_groups unsatisfied_groups( values: Mapping[str, str], ) -> tuple[tuple[str, ...], ...] Return the any_of groups no value satisfies. Empty when every group has at least one non-blank value, which is the case a UI needs to allow a save. label_for label_for(key: str) -> str The declared label for a field key, falling back to the key itself. anyinfer.SetupField dataclass SetupField( key: str, label: str, kind: SetupFieldKind, required: bool = False, help_text: str = "", placeholder: str = "", env_var: str = "", advanced: bool = False, default_value: str = "", choices: tuple[str, ...] = (), ) One configurable field a provider needs, described declaratively. key instance-attribute key: str Machine name the entered value is saved under in provider settings — a well-known key such as api_key or api_version, or a provider-specific options entry. label instance-attribute label: str Human-readable name a UI shows for the field. kind instance-attribute kind: SetupFieldKind Semantic role of the field, which tells a UI how to render and validate it. required class-attribute instance-attribute required: bool = False Whether saving needs a non-blank value for this field on its own. Either-or alternatives between fields are expressed via ProviderSetupSpec.any_of instead. help_text class-attribute instance-attribute help_text: str = '' Explanatory sentence shown alongside the field; empty when the label suffices. placeholder class-attribute instance-attribute placeholder: str = '' Example value a UI shows in the empty editor. Declared per field because the right example is provider knowledge: the environment variable an Anthropic key conventionally lives in is not the one an OpenAI key does, and a UI that guesses picks one provider's convention and is wrong for all the others. Empty means the UI falls back to whatever generic hint suits the kind. env_var class-attribute instance-attribute env_var: str = '' Environment variable this field is conventionally supplied from, if any. The machine-readable half of what placeholder says in prose. A placeholder reading "env://ANTHROPIC_API_KEY or a literal key" is a UI hint; parsing it back out to learn which variable to look for is guessing at free text. Declared here, "is this provider already usable on this machine?" becomes a lookup rather than a regex, which is what anyinfer.local.discovery and a config UI's "we found this in your environment" both need. The bare variable name, never the env:// reference form — the scheme is the credential resolver's spelling, and storing it here would make every consumer strip it. Empty when the provider has no convention, which is the case for a generic OpenAI-compatible endpoint and for every provider whose credential is not an environment variable at all. advanced class-attribute instance-attribute advanced: bool = False Whether this field has a standard value that is right for almost every user. The split this expresses is prominence, not optionality. required already says "saving fails without a value"; plenty of fields are neither required nor worth showing — an Ollama base URL, an Anthropic API version, a Bedrock signing profile. Presented as equals they read as five questions where there is really one, and every consuming application then has to rediscover which is which from prose help text. So the provider says it here: advanced fields are the ones a UI may fold behind a disclosure, leaving the fields a user genuinely has to answer in front of them. A required field is never advanced — hiding something that blocks saving is exactly the trap this exists to avoid, and ProviderSetupSpec rejects that combination. default_value class-attribute instance-attribute default_value: str = '' The value the provider falls back to when this field is left blank. What a UI shows as "standard: …" beside a hidden field, so folding one away never hides what it will do. Empty when the field has no default — a credential has none, and neither does an endpoint the user must supply. A UI should render this rather than pre-filling the editor with it: a saved copy of today's default is a value frozen at the moment someone opened a dialog, and it keeps overriding the real default long after that default has moved on. choices class-attribute instance-attribute choices: tuple[str, ...] = () The accepted values, for a choice field. A bounded enum typed into a free-text box is a value that validates at request time rather than at configuration time, which turns a typo into a runtime error somewhere else entirely. Declaring the set here lets a UI offer it and lets a non-UI caller check a stored value without knowing which provider it belongs to. Empty for every other kind — a field whose values a provider cannot enumerate has nothing to put here, and SetupField rejects the two contradictory combinations. __post_init__ __post_init__() -> None Reject a field whose declared choices and kind disagree. A choice with no alternatives renders as an empty dropdown, and choices on a free-text field are a constraint no UI will apply. Both are provider-authoring errors, caught at import time rather than at the moment someone opens a dialog. anyinfer.HostShorthand dataclass HostShorthand(scheme: str, default_port: int) Expansion rule for bare hostnames, e.g. myserver → http://myserver:11434. scheme instance-attribute scheme: str URL scheme prepended when expanding a bare host, e.g. http. default_port instance-attribute default_port: int Port appended when the bare host does not name one. expand expand(host: str) -> str Expand a bare host into a full base URL, leaving full URLs untouched. anyinfer.default_registry module-attribute default_registry = ProviderRegistry() The process-wide provider registry used when a client is given no other. Deliberately not named registry: a module-level name equal to the module's own name shadows the module itself on the package, which breaks introspection, documentation generation, and patching in tests. anyinfer.providers.builtin_descriptors builtin_descriptors() -> Iterator[ProviderDescriptor] Yield every built-in provider descriptor. A module that fails to import — typically because its optional extra is absent — is skipped rather than breaking discovery for the others. The resulting "unknown provider" error, raised only if that provider is actually requested, carries the install hint. Catalog Two shapes over one body of data: the alias ladder ("just give me a good default") and the logical model table ("let me browse and pick"). Catalog.with_alias_target bridges them. See the model catalog. anyinfer.Catalog dataclass Catalog( aliases: Mapping[str, AliasEntry] = dict(), artifacts: Mapping[str, GgufArtifact] = dict(), models: Mapping[str, ModelEntry] = dict(), default_alias: str = "medium", format_version: int = FORMAT_VERSION, ) A parsed alias catalog. Attributes: Name Type Description aliases Mapping[str, AliasEntry] The tier ladder, keyed by lowercase alias name. artifacts Mapping[str, GgufArtifact] Pinned GGUF artifacts, keyed by artifact id. Includes artifacts derived from the model table's GGUF variants as well as explicitly declared ones. models Mapping[str, ModelEntry] The logical model table, keyed by model id. default_alias str The alias resolution falls back to when a caller does not pick one. format_version int Schema version of the parsed document. has_alias has_alias(name: str) -> bool Whether an alias exists (case-insensitively). alias alias(name: str) -> AliasEntry Look up an alias. Raises: Type Description ConfigError If the alias is unknown. targets_for_alias targets_for_alias(name: str) -> Mapping[str, TargetEntry] Every provider realization of an alias. artifact artifact(artifact_id: str) -> GgufArtifact Look up a GGUF artifact. Raises: Type Description ConfigError If the artifact is unknown. alias_names alias_names() -> tuple[str, ...] Every alias name, sorted. model model(model_id: str) -> ModelEntry Look up a logical model. Raises: Type Description ConfigError If the model is unknown. models_for models_for( provider_id: str | None = None, *, best_at: str | None = None, kind: ModelKind | None = None, ) -> tuple[ModelEntry, ...] Logical models filtered by serving channel, category, and kind, id-ordered. Parameters: Name Type Description Default provider_id str | None Keep only models this channel can serve. None best_at str | None Keep only models carrying this category tag. None kind ModelKind | None Keep only rows of this kind. None — the default — keeps every kind, because a caller browsing "what can this machine run" wants the embedding models too; a caller filling a chat picker passes "generation" rather than relying on a narrowing it never asked for. None with_alias_target with_alias_target( alias: str, provider_id: str, model_id: str ) -> Catalog Point one alias's provider target at a catalog model. This is the bridge between browsing and the tier ladder: an app implements "use my catalog pick as medium" in one call, and the result resolves through the ordinary alias machinery with no resolver changes. Raises: Type Description ConfigError If the alias or model is unknown, or the model has no artifact for the named provider. overlay overlay(other: Catalog) -> Catalog Merge other on top of this catalog, entry by entry. Application entries win over bundled ones at the alias, artifact, and model level — an overridden alias replaces the bundled one wholesale rather than merging its target map, so an app can remove a provider from a tier it does not want used. The same wholesale rule applies per model id. from_mapping classmethod from_mapping(data: Mapping[str, Any]) -> Catalog Parse a catalog document. Raises: Type Description ConfigError On an unsupported format version or malformed entries. from_files classmethod from_files(*paths: Path) -> Catalog Load several catalog documents and overlay them left to right. The bundled catalog is split this way on purpose: default.json stays a small, human-editable alias policy file while models.json is machine-maintained data with its own refresh cadence. Raises: Type Description ConfigError If any file is missing, malformed, or empty of catalogs. anyinfer.ModelEntry dataclass ModelEntry( id: str, kind: ModelKind = "generation", family: str = "", display_name: str = "", parameter_size: str | None = None, quantization: str | None = None, context_window: int | None = None, license: str = "", best_at: tuple[str, ...] = (), est_file_bytes: int | None = None, est_ram_bytes: int | None = None, est_vram_bytes: int | None = None, last_verified: str = "", source: str = "", variants: tuple[ModelVariant, ...] = (), ollama: OllamaChannel | None = None, embedding: EmbeddingCapabilities | None = None, description: str = "", ) One logical model in the catalog. Attributes: Name Type Description id str Stable catalog id ("qwen2.5-7b-instruct"). kind ModelKind "generation" (the default, and every entry written before embeddings existed) or "embedding". See ModelKind. family str Model family, for grouping in a UI. display_name str Human-facing name. parameter_size str | None Parameter class ("7B"), keying the KV-cache cost table. quantization str | None The default quantization the headline estimates assume. context_window int | None Native context length, when known. license str License id; gated against the download allowlist. best_at tuple[str, ...] Categories from BEST_AT. est_file_bytes int | None Download size of the default variant. est_ram_bytes int | None Memory needed on the CPU-only path. est_vram_bytes int | None Memory needed when fully offloaded. last_verified str ISO date the entry was actually checked against upstream. source str URL of the upstream repository or registry page. variants tuple[ModelVariant, ...] The quantization ladder, best quality first is not assumed — sort by ModelVariant.quality_rank. ollama OllamaChannel | None The Ollama channel, when the model is published there. embedding EmbeddingCapabilities | None Vector facts on kind="embedding" rows only, as the same record the client reads — the catalog states only dimensions and max_input_tokens, the two an upstream model card actually publishes. Everything else about an embedding model is either provider-specific (batch ceilings) or measurable rather than declared (normalization, which probe_embedding() observes), so the catalog does not guess at it. description str Free text for display. name property name: str Display name, falling back to the id. is_embedding property is_embedding: bool Whether this row describes an embedding model rather than a chat model. channels property channels: tuple[str, ...] Provider ids that can serve this model, sorted. gguf_artifact_id property gguf_artifact_id: str | None The artifact id of this model's headline GGUF variant, when there is one. "Headline" means the quantization the entry's own memory estimates describe — the rung a browsing user is being shown; not the highest rung in the repository. Picking the largest would quietly hand someone a Q8_0 download after they read a Q4_K_M size. variants_for variants_for( engine: str | None = None, ) -> tuple[ModelVariant, ...] Variants for one engine, best quality first. variant variant(variant_id: str) -> ModelVariant Look up one variant. Raises: Type Description ConfigError If the variant is unknown. matches_best_at matches_best_at(category: str | None) -> bool Whether this entry carries a category tag (case-insensitively). anyinfer.ModelVariant dataclass ModelVariant( id: str, engine: str = "llama.cpp", kind: str = "gguf", quantization: str = "", quality_rank: int = 0, est_file_bytes: int | None = None, est_ram_bytes: int | None = None, est_vram_bytes: int | None = None, min_compute_capability: str | None = None, source: SourceRef = SourceRef(), artifact_id: str | None = None, ) One (model, quantization, engine) rung of a model's ladder. A variant carries its own source reference, because a quantized vLLM variant is usually a different repository while a quantized GGUF variant is usually a different file in the same repository. One schema covers both only because the reference is per variant rather than per model. Attributes: Name Type Description id str Stable variant id, unique within the catalog. engine str "llama.cpp" or "vllm". kind str "gguf" or "hf_repo". quantization str The quantization this rung ships ("Q4_K_M", "awq"). quality_rank int Ladder position; higher is better quality. est_file_bytes int | None On-disk size of the weights. est_ram_bytes int | None Memory needed on the CPU-only path. est_vram_bytes int | None Memory needed when fully offloaded. min_compute_capability str | None NVIDIA compute capability this variant's kernels need, as a string ("8.9"). None means no gate. source SourceRef Where the bytes come from. artifact_id str | None For GGUF variants, the id under which the derived GgufArtifact is registered, so alias targets and the llama.cpp adapter can reference it. is_pinned property is_pinned: bool Whether every declared file carries a revision and a digest. anyinfer.OllamaChannel dataclass OllamaChannel(tag: str, digest: str | None = None) How a logical model is packaged in the Ollama registry. We never download these — the daemon owns its own blob store. The tag is what we recommend, and digest is what drift checking compares against, because registry tags are mutable by design. anyinfer.BEST_AT module-attribute BEST_AT: frozenset[str] = frozenset( { "agentic", "code-completion", "coding", "drafting", "embeddings", "general-chat", "long-context", "low-resource", "math", "multilingual", "rag", "reasoning", "tool-use", "vision", } ) The closed vocabulary of "best at" categories. Closed on purpose: a free-text tag set drifts into synonyms nobody can filter on. Adding a category is a deliberate edit here, and the catalog validator enforces the set. anyinfer.ModelKind module-attribute ModelKind = str What a catalog row is for: "generation" or "embedding". The two share every acquisition mechanism — a GGUF is a GGUF, and the resolver, digests, and download machinery never care what the weights compute. What differs is interpretation, and it differs enough that guessing is wrong: an embedding model has no KV cache sized for a chat context, no quality ladder a user is trading off against throughput, and no place in the small/medium/large tier system, which answers "how big a chat model should I run". One field distinguishes them so the pieces that genuinely differ can ask, rather than a second table duplicating the pieces that do not. anyinfer.MODEL_KINDS module-attribute MODEL_KINDS: frozenset[str] = frozenset( {"generation", "embedding"} ) The closed vocabulary of ModelEntry.kind, enforced by the parser and the validator. anyinfer.load_default_catalog load_default_catalog() -> Catalog Load the catalog bundled with this AnyInfer build. Two documents, overlaid: default.json carries the hand-edited alias policy, and models.json carries the machine-maintained logical model table with its own refresh cadence — the same split the bundled pricing table uses. Browsing the Local Catalog What Client.local_catalog() returns: every catalog model annotated with whether it fits, and why. anyinfer.CatalogView dataclass CatalogView( entries: tuple[CatalogEntryFit, ...] = (), hardware: HardwareProfile | None = None, hardware_source: HardwareSource = "unavailable", backend: Backend | None = None, notes: tuple[str, ...] = (), ) A filtered, fit-annotated view of the local model catalog. Attributes: Name Type Description entries tuple[CatalogEntryFit, ...] Models, best-fit-first. hardware HardwareProfile | None The profile fits were judged against, when there was one. hardware_source HardwareSource "detected" (probed this machine), "provided" (the caller supplied specs), or "unavailable" — the cue to collect a remote host's specs from the user and call again. backend Backend | None The llama.cpp runtime variant that would actually drive these, when one is installed. notes tuple[str, ...] View-level remarks, such as the runtime a machine should install. runnable property runnable: tuple[CatalogEntryFit, ...] Only the entries this machine can plausibly run. __len__ __len__() -> int How many entries the view holds. __iter__ __iter__() -> Any Iterate the entries, best fit first. anyinfer.CatalogEntryFit dataclass CatalogEntryFit( model: ModelEntry, fit: ModelFit, channels: tuple[str, ...] = (), ) One catalog model, judged against a machine. Attributes: Name Type Description model ModelEntry The catalog entry. fit ModelFit How it fits, with reasons. channels tuple[str, ...] Provider ids that can serve it. id property id: str The model id. name property name: str The display name. Credentials anyinfer.CredentialResolver Bases: Protocol Resolves credential references of one scheme. handles handles(reference: str) -> bool Whether this resolver recognizes reference. resolve resolve(reference: str) -> str Resolve reference to a secret. Raises: Type Description CredentialError If the reference is recognized but cannot be resolved. anyinfer.ResolverChain ResolverChain(resolvers: list[CredentialResolver]) Tries each resolver in order, returning the first match's result. The chain; not the individual resolvers — is responsible for registering resolved secrets for redaction, so a third-party resolver cannot forget to. add add( resolver: CredentialResolver, *, first: bool = True ) -> None Register an additional resolver, by default ahead of the built-ins. resolve resolve(reference: str | None) -> str | None Resolve a credential reference. Parameters: Name Type Description Default reference str | None The reference string, or None for "no credential configured". required Returns: Type Description str | None The resolved secret, or None when reference is None or empty. Raises: Type Description CredentialError If no resolver handles the reference, or resolution fails. anyinfer.default_resolver default_resolver() -> ResolverChain Build the standard resolver chain: keyring, env, then literal. Literal is last because it accepts anything; the scheme-specific resolvers must get first refusal. --- # Reference / Configuration Source: https://anyinfer.dev/reference/api/configuration/ Configuration API The versioned JSON loader and what it produces: load_config and loads_config parse and validate a file into an AnyInferConfig, and dump_config and dumps_config write the same format back. The prose reference for the file format lives in Shared configuration. anyinfer.CONFIG_FORMAT_VERSION module-attribute CONFIG_FORMAT_VERSION = 1 The configuration format version written and understood by this release. anyinfer.MAX_CONFIG_BYTES module-attribute MAX_CONFIG_BYTES = 1024 * 1024 Maximum accepted configuration size. anyinfer.AnyInferConfig dataclass AnyInferConfig( providers: tuple[ProviderSettings, ...] = (), route: Route | None = None, format_version: int = CONFIG_FORMAT_VERSION, context: ContextTuning = DEFAULT_TUNING, history: HistoryPolicy | None = None, cache: CachePolicy | None = None, arena: ArenaPolicy | None = None, arenas: Mapping[str, ArenaPolicy] = dict(), mcp: tuple[MCPServer, ...] = (), operation_routes: Mapping[str, Route] = dict(), ) Validated configuration shared by every AnyInfer integration surface. Pass providers and route directly to Client or AsyncClient. The same object is used by the command-line runner and OpenAI-compatible sidecar. Attributes: Name Type Description providers tuple[ProviderSettings, ...] Configured provider instances, in declaration order. route Route | None Default fallback route, when one was configured. format_version int Parsed file-format version. context ContextTuning Advanced context-reduction settings from the optional context block. Pass to anyinfer.context.select as tuning=. Defaults reproduce the library's plain behaviour, so a file without the block behaves as before. history HistoryPolicy | None Conversation-compaction policy from the optional history block, or None when the file does not ask for one. Pass to Client or AsyncClient as history=; every frontend built on that client then behaves identically. cache CachePolicy | None Prompt-cache placement from the optional cache block, or None when the file does not ask for one. Pass to Client or AsyncClient as cache=. Absent means no placement — caching changes what a provider bills, so it is never turned on by a file that did not name it. mcp tuple[MCPServer, ...] Model Context Protocol servers described by the optional mcp block. These are inert descriptions: loading a file never spawns a process or opens a socket. Pass them to anyinfer.mcp.MCPToolset.connect when tools are wanted. operation_routes Mapping[str, Route] Per-operation default routes from the optional operation_routes block, keyed "embedding"/"rerank". An embedding route can never be selected for generation or vice versa — generation's default stays default_route. Pass to Client or AsyncClient as operation_routes=. anyinfer.load_config load_config( path: str | Path, *, registry: ProviderRegistry | None = None, ) -> AnyInferConfig Read and validate an AnyInfer JSON configuration file. Parameters: Name Type Description Default path str | Path File to read. required registry ProviderRegistry | None Provider registry used to validate setup-field names. Defaults to the process-wide registry, including installed third-party providers. None Raises: Type Description ConfigError If the file cannot be read or does not match the shared format. anyinfer.loads_config loads_config( text: str, *, source: str = "", registry: ProviderRegistry | None = None, ) -> AnyInferConfig Parse and validate AnyInfer configuration from a JSON string. Parameters: Name Type Description Default text str UTF-8 JSON text. required source str Human-readable source name included in validation errors. '' registry ProviderRegistry | None Provider registry used to validate setup-field names. None Raises: Type Description ConfigError If the text is too large, invalid JSON, or has invalid fields. anyinfer.dumps_config dumps_config( config: AnyInferConfig, *, comments: bool = False ) -> str Render a configuration as the JSON text loads_config accepts. The other half of the shared format. Three frontends could read this file and none could write it, which left every example of the format as prose and left anyinfer init with no way to produce one but string templating. Round-tripping is the contract: loads_config(dumps_config(c)) == c for every configuration the loader accepts. What that costs is verbosity in one place — a provider instance carrying an opt-in policy emits that policy even when every field in it is standard, because an omitted block and a default-valued block mean different things to the loader and only one of them is what the caller had. Credential values are written exactly as configured. References such as env:// and credential:// remain references, while a literal credential remains literal. This function never resolves a reference, but callers must still review configurations that they constructed with literal secrets before writing or committing them. Discovery and anyinfer init produce references so their generated files contain no key material. Parameters: Name Type Description Default config AnyInferConfig The configuration to render. required comments bool Write a leading COMMENT_KEY note explaining what the file is. Still JSON, and still accepted by the loader. False Returns: Type Description str UTF-8 JSON text, two-space indented, ending in a newline. Raises: Type Description ConfigError If a provider's options or headers hold a value JSON cannot represent. Settings built in Python may carry anything; a file cannot. anyinfer.dump_config dump_config( config: AnyInferConfig, path: str | Path, *, force: bool = False, ) -> None Write a configuration to a file, refusing to replace one that exists. Destructive-by-default is not acceptable for a file a user may have hand-tuned, and a configuration file is exactly that kind of file. Overwriting is available and has to be asked for. Parameters: Name Type Description Default config AnyInferConfig The configuration to write. required path str | Path Where to write it. Parent directories must already exist. required force bool Replace an existing file instead of refusing. False Raises: Type Description ConfigError If the path exists and force is false, if the configuration cannot be rendered, or if the file cannot be written. --- # Reference / Local Inference Source: https://anyinfer.dev/reference/api/local/ Local Inference The anyinfer.local subsystem: hardware detection, backend selection, runtime acquisition, tuning, fit classification, model acquisition and storage, server supervision, and hardware→tier recommendation. Concepts: the local subsystem · the model catalog · the model catalog · guides: run a model locally · run a model locally. from anyinfer import local Hardware anyinfer.local.detect detect(*, use_cache: bool = True) -> HardwareProfile Detect this machine's hardware. Never raises: anything that could not be determined becomes a warning and a None field. Callers treat the result as advice, not as fact. Parameters: Name Type Description Default use_cache bool Read and write the disk cache. Overridden by CACHE_BYPASS_ENV and CACHE_REFRESH_ENV. True Returns: Type Description HardwareProfile The detected profile. anyinfer.local.HardwareProfile dataclass HardwareProfile( os_name: str, arch: str, total_ram_bytes: int | None = None, available_ram_bytes: int | None = None, cpu_name: str | None = None, physical_cores: int | None = None, logical_cores: int | None = None, accelerators: tuple[Accelerator, ...] = (), warnings: tuple[str, ...] = (), detected_at: float = 0.0, ) What we could learn about this machine. Every field may be None: absence means "not determined", never "zero". Attributes: Name Type Description os_name str "windows", "linux", "darwin", or the raw platform string. arch str Machine architecture as reported by platform.machine(). total_ram_bytes int | None Physical RAM. available_ram_bytes int | None RAM currently free, when the platform reports it. cpu_name str | None Processor model string. physical_cores int | None Physical core count, preferred for thread tuning. logical_cores int | None Logical processor count. accelerators tuple[Accelerator, ...] Detected accelerators, strongest first. warnings tuple[str, ...] Everything that could not be determined, and why. detected_at float Unix timestamp of the probe, for cache display. primary_accelerator property primary_accelerator: Accelerator | None The accelerator a server should target, or None for CPU-only. has_accelerator property has_accelerator: bool Whether any non-CPU accelerator was detected. total_vram_bytes property total_vram_bytes: int | None Total memory of the primary accelerator, when known. user_supplied property user_supplied: bool Whether this profile came from from_user_input rather than a probe. from_user_input classmethod from_user_input( *, ram_gb: float | None = None, vram_gb: float | None = None, accelerator: AcceleratorKind | None = None, accelerator_name: str | None = None, compute_capability: str | None = None, os_name: str = "", arch: str = "", ) -> HardwareProfile Build a profile from specs a person supplied, in familiar units. The remote-Ollama case: local probing describes the wrong machine, and no Ollama API reports its host's specs, so asking the user is the only honest source. Values arrive in gigabytes because that is what a user reads off a spec sheet; anything left out stays None and keeps its "not determined" meaning. The profile is marked as self-reported in warnings, so any advice derived from it can say so. to_json to_json() -> dict[str, Any] Serialize for the disk cache. from_json classmethod from_json(data: dict[str, Any]) -> HardwareProfile Deserialize from the disk cache. anyinfer.local.Accelerator dataclass Accelerator( kind: AcceleratorKind, name: str | None = None, total_vram_bytes: int | None = None, free_vram_bytes: int | None = None, unified_memory: bool = False, compute_capability: str | None = None, driver_version: str | None = None, ) One detected accelerator. Attributes: Name Type Description kind AcceleratorKind Which runtime family can drive it. name str | None Human-readable device name, when reported. total_vram_bytes int | None Total device memory, or None when unknown. free_vram_bytes int | None Free device memory at probe time, or None. unified_memory bool True when device memory is shared with system RAM (Apple Silicon), which makes VRAM budgeting a different calculation entirely. compute_capability str | None NVIDIA compute capability as reported by the driver ("8.9"), or None. Quantized kernels gate on this — FP8 needs 8.9, the Marlin GPTQ kernel needs 8.0, and an unknown capability must exclude a gated variant rather than optimistically permit it. driver_version str | None Vendor driver version string, when reported. Used to check that a downloadable CUDA runtime's toolkit version is supported before installing it. compute_capability_value property compute_capability_value: float | None compute_capability as a comparable number, or None when unparseable. driver_major property driver_major: int | None The major component of driver_version, or None. anyinfer.local.AcceleratorKind module-attribute AcceleratorKind = Literal[ "cuda", "rocm", "metal", "vulkan", "cpu" ] Accelerator families we can detect and target. anyinfer.local.probe_signature probe_signature() -> str Fingerprint the probe tooling, so the cache invalidates when it changes. Keyed on the resolved path and mtime of each probe executable plus the interpreter's platform: installing a GPU driver, or moving to different hardware, changes this. anyinfer.local.cache_path cache_path() -> Path Where the detection cache lives. anyinfer.local.CACHE_BYPASS_ENV module-attribute CACHE_BYPASS_ENV = 'ANYINFER_HARDWARE_CACHE_BYPASS' Set to skip the cache entirely (read and write). anyinfer.local.CACHE_REFRESH_ENV module-attribute CACHE_REFRESH_ENV = 'ANYINFER_HARDWARE_CACHE_REFRESH' Set to ignore a cached result and re-probe, then rewrite the cache. Resource Sampling and Storage Lightweight host metrics used by benchmarks and local-capacity reporting. anyinfer.local.ResourceSample dataclass ResourceSample( cpu_percent: float | None = None, ram_percent: float | None = None, gpu_percent: float | None = None, vram_percent: float | None = None, ram_used_bytes: int | None = None, vram_used_bytes: int | None = None, ) One instantaneous host-utilization observation. Percentages range from 0 to 100. None means the platform did not expose a safe, dependency-free reading. anyinfer.local.SystemSampler SystemSampler() Stateful sampler for CPU, RAM, GPU, and VRAM utilization. sample sample() -> ResourceSample Read one sample, leaving unsupported values unknown. anyinfer.local.StorageProfile dataclass StorageProfile( path: str, total_bytes: int | None = None, free_bytes: int | None = None, ) Capacity facts for the filesystem holding a path. anyinfer.local.storage_profile storage_profile(path: Path | str) -> StorageProfile Return capacity/free-space facts for path without performing a speed test. Backends anyinfer.local.available_backends available_backends( *, search_paths: list[Path] | None = None, hardware: HardwareProfile | None = None, runtime_root: Path | None = None, include_runtime_root: bool = True, ) -> list[Backend] Find installed llama-server binaries, best first. Three sources, in descending order of how much they can be trusted: Manifest-validated variants under the well-known runtime root. The manifest states the backend, so nothing is inferred. Caller-supplied directories, where the backend is guessed from the directory name — a convention, not a fact. PATH, where the backend is guessed from the hardware, because a binary on PATH says nothing at all about what it was compiled against. The distinction is recorded in Backend.detail rather than hidden, so a surprising selection can be explained. Parameters: Name Type Description Default search_paths list[Path] | None Extra directories to search, each expected to hold a runtime variant named after its backend (.../cuda/llama-server). None hardware HardwareProfile | None Detected hardware, used to label what a found binary can actually drive. None runtime_root Path | None Override the well-known runtime root. None include_runtime_root bool Search the well-known runtime root at all. True Returns: Type Description list[Backend] Usable backends, ranked. Empty when no binary is found at all. anyinfer.local.select_backend select_backend( hardware: HardwareProfile, *, preferred: AcceleratorKind | None = None, search_paths: list[Path] | None = None, runtime_root: Path | None = None, include_runtime_root: bool = True, ) -> Backend | None Pick the requested, or best, backend this machine can actually use. A CUDA build on a machine with no NVIDIA device is useless, so the selection is the intersection of what is installed and what the hardware can drive. When the best drivable variant is not the best variant the hardware could theoretically use — a Vulkan build on an NVIDIA card because the CUDA add-on is not installed — the returned Backend.detail says so, which is what turns a silent degradation into a discoverable recommendation. anyinfer.local.Backend dataclass Backend( kind: AcceleratorKind, binary: Path, rank: int = 0, detail: str = "", ) One usable llama.cpp runtime variant. Attributes: Name Type Description kind AcceleratorKind The acceleration family this build targets. binary Path Path to its llama-server executable. rank int Preference score from BACKEND_RANK. detail str Why this backend was or was not selected. anyinfer.local.BACKEND_RANK module-attribute BACKEND_RANK: dict[AcceleratorKind, int] = { "cuda": 30, "metal": 25, "rocm": 22, "vulkan": 20, "cpu": 10, } Preference order across runtime variants; higher wins. Runtime Variants AnyInfer ships no llama.cpp binaries. These fetch, validate, and select them; CUDA is an explicit opt-in, never installed on a user's behalf. anyinfer.local.install_runtime install_runtime( kind: AcceleratorKind | None = None, *, hardware: HardwareProfile | None = None, root: Path | None = None, table: RuntimeTable | None = None, progress: ProgressCallback | None = None, client: Client | None = None, force: bool = False, ) -> InstallReport Fetch, verify, and unpack a llama-server runtime variant. Parameters: Name Type Description Default kind AcceleratorKind | None Which backend to install. None picks default_runtime_kind, which is never CUDA. None hardware HardwareProfile | None Detected hardware, needed for the default choice and the CUDA gate. None root Path | None Runtime root; defaults to runtime_root(). None table RuntimeTable | None Pinned artifact table; defaults to the bundled one. None progress ProgressCallback | None Download progress callback. None client Client | None An httpx2.Client, for tests or custom transports. None force bool Install CUDA even when the precondition checks object. Never skips digest verification — only the hardware gate. False Returns: Type Description InstallReport A report naming the executable to launch. Raises: Type Description LocalRuntimeError If no build exists for this platform and backend, if a CUDA precondition fails without force, or if the archive fails verification or cannot be unpacked. anyinfer.local.installed_runtimes installed_runtimes( root: Path | None = None, *, build: str | None = None ) -> list[RuntimeManifest] Every validated runtime variant under the runtime root. anyinfer.local.remove_runtime remove_runtime( kind: AcceleratorKind, *, root: Path | None = None ) -> bool Delete an installed runtime variant. Returns whether anything was removed. anyinfer.local.default_runtime_kind default_runtime_kind( hardware: HardwareProfile | None, ) -> AcceleratorKind Which variant to install when the caller expresses no preference. Never CUDA. The vendor-neutral small builds cover every GPU well enough to be useful immediately, and a several-hundred-megabyte download is a decision a user makes, not one a library makes on their behalf: Apple Silicon → Metal, which needs no vendor runtime and is the native path. Intel Mac → CPU; llama.cpp's Metal backend targets Apple Silicon. Windows or Linux with any GPU → Vulkan, which drives NVIDIA, AMD, and Intel alike. Anything else → CPU. anyinfer.local.check_cuda_preconditions check_cuda_preconditions( hardware: HardwareProfile, table: RuntimeTable ) -> tuple[tuple[str, ...], tuple[str, ...]] Return (blocking_reasons, warnings) for installing the CUDA add-on. The pinned build implies a CUDA toolkit version, which implies a minimum driver and a minimum compute capability. Checking them up front turns "your GPU is too old" into a clear refusal before a 400 MB download instead of an incomprehensible crash at load. anyinfer.local.install_hint install_hint( hardware: HardwareProfile | None, table: RuntimeTable | None = None, ) -> str A one-line suggestion of which runtime this machine should install. anyinfer.local.load_runtime_table load_runtime_table( path: Path | None = None, ) -> RuntimeTable Load the pinned runtime table. Raises: Type Description LocalRuntimeError If the bundled table is unreadable or malformed. Unlike the probes, this is real data shipped with the package: a broken table is a build defect, not a property of the user's machine. anyinfer.local.runtime_root runtime_root() -> Path Where runtime variants are installed. Follows the same per-OS data-dir convention as the model directory, so a user who knows where one lives can find the other. Overridable with ANYINFER_RUNTIME_DIR. anyinfer.local.RuntimeTable dataclass RuntimeTable( build: str, generated: str = "", release_url: str = "", cuda_toolkit: str = "", min_cuda_driver_major: int = 0, min_compute_capability: float = 0.0, warn_below_vram_bytes: int = 0, artifacts: tuple[RuntimeArtifact, ...] = (), ) The pinned set of fetchable runtime builds. Attributes: Name Type Description build str The llama.cpp release tag every variant here comes from. generated str When the table was pinned. release_url str The upstream release page. cuda_toolkit str CUDA version the pinned CUDA build links against. min_cuda_driver_major int Driver major version that toolkit requires. min_compute_capability float Lowest GPU compute capability the build supports. warn_below_vram_bytes int Below this, CUDA works but is not worth the download. artifacts tuple[RuntimeArtifact, ...] Every pinned variant. for_platform for_platform( key: str | None = None, ) -> tuple[RuntimeArtifact, ...] Variants available for one platform, best backend first. artifact artifact( backend: str, *, key: str | None = None ) -> RuntimeArtifact | None One variant by backend, for this platform. anyinfer.local.RuntimeArtifact dataclass RuntimeArtifact( platform: str, backend: AcceleratorKind, filename: str, url: str, sha256: str, size_bytes: int | None = None, companions: tuple[RuntimeArtifact, ...] = (), ) One downloadable runtime archive. Attributes: Name Type Description platform str Platform key this build targets ("win32-amd64"). backend AcceleratorKind Acceleration family the build was compiled for. filename str Archive file name. url str Where to fetch it. sha256 str Expected digest of the archive. size_bytes int | None Expected archive size. companions tuple[RuntimeArtifact, ...] Extra archives unpacked into the same directory — the CUDA runtime libraries ship separately from the llama.cpp build. total_bytes property total_bytes: int Bytes to transfer including companions. anyinfer.local.RuntimeManifest dataclass RuntimeManifest( backend: AcceleratorKind, build: str, architecture: str, executable: Path, directory: Path, ) A validated runtime.json describing an installed variant. Attributes: Name Type Description backend AcceleratorKind The acceleration family this build targets. build str The llama.cpp build id it was cut from. architecture str The machine architecture it runs on. executable Path Absolute path to llama-server. directory Path The variant directory. anyinfer.local.InstallReport dataclass InstallReport( backend: AcceleratorKind, build: str, directory: Path, executable: Path, downloaded_bytes: int = 0, reused: bool = False, warnings: tuple[str, ...] = (), ) The outcome of installing a runtime variant. Attributes: Name Type Description backend AcceleratorKind Which variant was installed. build str The build id it came from. directory Path Where it landed. executable Path The llama-server inside it. downloaded_bytes int Bytes actually transferred (zero when everything was cached). reused bool Whether an already-valid install was kept. warnings tuple[str, ...] Non-blocking notes, including CUDA precondition warnings. Tuning anyinfer.local.plan_server plan_server( hardware: HardwareProfile, model: TuningInputs, *, posture: Posture = "balanced", ) -> ServerPlan Derive a server plan from hardware, model facts, and posture. Parameters: Name Type Description Default hardware HardwareProfile The detected profile. Unknown fields make the plan more conservative, never more optimistic. required model TuningInputs What is known about the model being served. required posture Posture How much of the machine to commit. 'balanced' Returns: Type Description ServerPlan A plan whose memory estimate the caller can check before spawning. anyinfer.local.TuningInputs dataclass TuningInputs( artifact_size_bytes: int | None = None, parameter_size: str | None = None, max_context: int | None = None, requested_context: int | None = None, ) What the tuner needs to know about the model being served. Attributes: Name Type Description artifact_size_bytes int | None On-disk size of the weights; they must be resident too. parameter_size str | None Parameter class ("7B"), keying the KV-cost table. max_context int | None Upper bound from the model itself, when known. requested_context int | None An explicit context the caller wants, overriding the ladder. anyinfer.local.ServerPlan dataclass ServerPlan( context_size: int, parallel: int = 1, threads: int = 4, batch_size: int = 512, ubatch_size: int = 128, gpu_layers: int = 0, cache_type_k: str = "f16", cache_type_v: str = "f16", flash_attention: bool = False, estimated_kv_bytes: int = 0, estimated_total_bytes: int = 0, posture: Posture = "balanced", rationale: tuple[str, ...] = (), projector_path: str | None = None, embeddings: bool = False, ) A concrete, explainable llama-server configuration. Attributes: Name Type Description context_size int --ctx-size, the total context across all slots. parallel int --parallel, concurrent request slots. threads int --threads for CPU work. batch_size int --batch-size. ubatch_size int --ubatch-size. gpu_layers int --n-gpu-layers; 0 means CPU-only. cache_type_k str --cache-type-k. cache_type_v str --cache-type-v. flash_attention bool Whether to request flash attention. estimated_kv_bytes int Predicted KV-cache footprint, for admission control. estimated_total_bytes int Weights plus KV cache. posture Posture The posture this plan was derived under. rationale tuple[str, ...] Human-readable notes explaining the choices. embeddings bool --embeddings. Live-verified 2026-08-14: llama-server refuses every embedding request with a 501 ("This server does not support embeddings. Start it with --embeddings") unless this was set at startup — it cannot be toggled on an already-running server, so a plan's embedding intent must be decided before the process is spawned, not after. context_per_slot property context_per_slot: int Usable context for a single request. server_arguments server_arguments( model_path: str, *, host: str, port: int ) -> list[str] Render the plan as llama-server CLI arguments. --jinja is always on: without it llama-server cannot apply a model's chat template, and tool calling silently does not work at all. anyinfer.local.Posture module-attribute Posture = Literal['conservative', 'balanced', 'aggressive'] How much of the machine the user is willing to spend on inference. anyinfer.local.CONTEXT_LADDER module-attribute CONTEXT_LADDER: tuple[int, ...] = ( 8192, 16384, 32768, 65536, ) Context sizes to consider, smallest first. The largest that fits wins. anyinfer.local.kv_bytes_per_token kv_bytes_per_token( parameter_size: str | None, cache_type: str ) -> int Estimate KV-cache bytes per token for a model class and cache precision. Fit and Variant Selection Whether a model will run on this machine, and which quantization to acquire for it. Both are advisory and both explain themselves. anyinfer.local.classify_fit classify_fit( entry: SizedEntry, hardware: HardwareProfile | None, *, posture: Posture = "balanced", backend: Backend | None = None, ) -> ModelFit Classify one catalog entry against a machine. Parameters: Name Type Description Default entry SizedEntry The catalog model, with its stored memory estimates. required hardware HardwareProfile | None The profile to budget against. None — the remote-host case — always yields unknown, because guessing someone else's machine is not advice. required posture Posture How much of the machine to commit; matches the tuner's postures. 'balanced' backend Backend | None The runtime variant that would actually drive this. Used only to surface the upgrade path when a faster one is available but not installed. None Returns: Type Description ModelFit A fit level with reasons. Never raises. anyinfer.local.ModelFit dataclass ModelFit( level: FitLevel, reasons: tuple[str, ...] = (), headroom_bytes: int | None = None, ) How a model relates to a machine's memory. Attributes: Name Type Description level FitLevel The classification. reasons tuple[str, ...] Human-readable notes, mirroring ServerPlan.rationale. Always non-empty. headroom_bytes int | None Budget minus requirement for the level that was chosen; negative when nothing fit, None when the numbers were unknown. runnable property runnable: bool Whether this machine can plausibly run the model at all. rank property rank: int Sort key for best-fit-first ordering; higher is better. anyinfer.local.FitLevel module-attribute FitLevel = Literal['gpu', 'cpu', 'tight', 'no', 'unknown'] How well a model fits: fully offloaded, CPU-resident, marginal, impossible, or unknown. anyinfer.local.memory_budget memory_budget( hardware: HardwareProfile, *, posture: Posture = "balanced", ) -> tuple[int | None, int | None] Return (vram_budget, ram_budget) in bytes for a posture. None means "not determinable". Free memory is preferred over total when the platform reported it, because a device already hosting a desktop compositor does not have its nameplate VRAM available. Unified memory reports no separate VRAM budget: it is the RAM budget, and counting it twice is how a plan overcommits an Apple Silicon machine. anyinfer.local.sort_by_fit sort_by_fit( pairs: Sequence[tuple[_EntryT, ModelFit]], ) -> list[tuple[_EntryT, ModelFit]] Order entries best-fit-first, then by descending headroom, then by id. Ties broken deterministically so the same catalog and the same machine always produce the same listing — a browsing UI that reshuffles between calls is unusable. Generic in the entry type so a caller keeps whatever it put in, rather than having its rows widened to the protocol. anyinfer.local.SizedEntry Bases: Protocol The subset of a catalog model this module needs. id property id: str Catalog model id. parameter_size property parameter_size: str | None Parameter class ("7B"), when stated. est_ram_bytes property est_ram_bytes: int | None Memory needed on the CPU-only path. est_vram_bytes property est_vram_bytes: int | None Memory needed when fully offloaded. anyinfer.local.select_variant select_variant( variants: Sequence[SelectableVariant], hardware: HardwareProfile | None, *, engine: str | None = None, parameter_size: str | None = None, backend: Backend | None = None, prefs: VariantPrefs | None = None, ) -> VariantChoice | None Choose the best quantization this machine can actually run. Parameters: Name Type Description Default variants Sequence[SelectableVariant] The model's ladder, in any order. required hardware HardwareProfile | None The machine to budget against. None yields None: guessing a quantization for an unknown machine is exactly the kind of confident wrong answer this module exists to avoid. required engine str | None Restrict to one engine's variants. None parameter_size str | None Parameter class, for the KV-cache cost. None backend Backend | None The runtime that would drive this, used to surface an upgrade path. None prefs VariantPrefs | None Selection preferences. None Returns: Type Description VariantChoice | None The chosen rung, or None when nothing acceptable fits. Use VariantChoice | None evaluate_variants when the rejection reasons matter too. anyinfer.local.evaluate_variants evaluate_variants( variants: Sequence[SelectableVariant], hardware: HardwareProfile | None, *, engine: str | None = None, parameter_size: str | None = None, backend: Backend | None = None, prefs: VariantPrefs | None = None, ) -> tuple[ VariantChoice | None, tuple[tuple[str, str], ...] ] Choose a rung and return why every other rung was passed over. The rejections matter even — especially, when nothing was chosen: "no quantization fits" is only useful advice if it comes with the numbers behind it. select_variant is the convenience wrapper for callers that only need the choice. Returns: Type Description VariantChoice | None (choice, rejections). choice is None when nothing acceptable fits, tuple[tuple[str, str], ...] which is a real answer, not a failure — the caller should then offer a smaller tuple[VariantChoice | None, tuple[tuple[str, str], ...]] model. anyinfer.local.VariantChoice dataclass VariantChoice( variant_id: str, quantization: str, fit: ModelFit, engine: str = "llama.cpp", est_file_bytes: int | None = None, reasons: tuple[str, ...] = (), rejected: tuple[tuple[str, str], ...] = (), tensor_parallel_size: int = 1, gpu_memory_utilization: float | None = None, ) The chosen rung, and why the others were not. Attributes: Name Type Description variant_id str The chosen variant. quantization str What it ships. fit ModelFit How it fits this machine. engine str Which engine it targets. est_file_bytes int | None What it will cost to download. reasons tuple[str, ...] Why this rung, and why not the next one up. rejected tuple[tuple[str, str], ...] (variant_id, why not) for every rung that was passed over. tensor_parallel_size int How many devices a vLLM launch should span. gpu_memory_utilization float | None The utilization the budget assumed. anyinfer.local.VariantPrefs dataclass VariantPrefs( posture: Posture = "balanced", context: int = _DEFAULT_CONTEXT, allow_low_quality: bool = False, allow_multi_gpu: bool = False, gpu_memory_utilization: float = 0.9, max_download_bytes: int | None = None, ) How aggressively to choose. Attributes: Name Type Description posture Posture Memory posture, matching the tuner's. context int Context length to budget the KV cache for. allow_low_quality bool Permit rungs below Q4_K_M. allow_multi_gpu bool Let vLLM sum VRAM across identical devices, and emit a tensor_parallel_size hint. llama.cpp never sums by default, because its split is layer-wise and much easier to get wrong. gpu_memory_utilization float vLLM's fraction of device memory to plan against. max_download_bytes int | None Refuse variants larger than this, whatever fits in memory. Acquisition and the Model Store Getting weights onto this disk and finding them again. Model acquisition lives here, never in a provider adapter. anyinfer.local.acquire Getting model weights onto this disk: plan → preflight → fetch → verify → place. One engine for both artifact shapes. A GGUF variant is a file set whose handle is the first shard; a Hugging Face snapshot is a directory whose handle is the directory. The file list and the handle differ; the machinery — resume, digest verification, locking, cancellation, and progress accounting — does not, and those are exactly the things that are expensive to get right twice. Progress is reported for the whole acquisition, not per file. A sharded artifact whose byte counter restarts at zero on every shard is worse than no progress bar, because a user cannot tell a restart from a stall. AcquisitionProgress therefore carries aggregate totals, counts bytes that were already on disk, and knows its full size before the first byte arrives — the payoff for pinning sizes in the catalog. AcquisitionPhase module-attribute AcquisitionPhase = Literal[ "resolving", "planning", "downloading", "verifying", "placing", "done", ] Where an acquisition is. Every transition produces a callback, unthrottled. ProgressSink module-attribute ProgressSink = Callable[[AcquisitionProgress], None] Receives AcquisitionProgress. See its docstring for the threading contract. AcquisitionProgress dataclass AcquisitionProgress( model_id: str, variant_id: str, phase: AcquisitionPhase, file_index: int = 0, file_count: int = 0, filename: str = "", file_downloaded_bytes: int = 0, file_total_bytes: int | None = None, total_downloaded_bytes: int = 0, total_bytes: int | None = None, total_is_estimate: bool = False, session_bytes: int = 0, bytes_per_second: float | None = None, eta_seconds: float | None = None, ) One progress report for a whole acquisition. The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run. Attributes: Name Type Description model_id str The catalog model being acquired. variant_id str The variant being acquired. phase AcquisitionPhase Which stage this report came from. file_index int 1-based index of the file that most recently advanced. file_count int How many files this acquisition covers. filename str Name of the file that most recently advanced. file_downloaded_bytes int Bytes present for that file, including a resumed prefix. file_total_bytes int | None Expected size of that file, when known. total_downloaded_bytes int Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%. total_bytes int | None Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes. total_is_estimate bool True when any file's size came from a guess rather than a pinned or reported figure. session_bytes int Bytes this run actually transferred, which is what rate and ETA are derived from. bytes_per_second float | None Transfer rate, or None until there is a real sample. eta_seconds float | None Seconds remaining, or None on the same condition. A wildly wrong ETA in the first second is worse than no ETA. fraction property fraction: float | None Completion as a fraction, or None when the total is unknown. AcquisitionPlan dataclass AcquisitionPlan( entry_id: str, model_id: str, variant_id: str, kind: str, engine: str, quantization: str, directory: str, handle: str, files: tuple[RemoteFile, ...], already_have_bytes: int = 0, total_bytes: int | None = None, warnings: tuple[str, ...] = (), satisfied: bool = False, revision: str | None = None, repo: str | None = None, license: str = "", ) What an acquisition will do, before it does any of it. Attributes: Name Type Description entry_id str The store entry that will be written. model_id str The catalog model. variant_id str The catalog variant. kind str "gguf" or "hf_repo". engine str Which engine the result is for. quantization str What will be on disk. directory str Store-relative destination. handle str Store-relative engine handle. files tuple[RemoteFile, ...] Every file, resolved. already_have_bytes int Bytes already on disk — verified files plus resumable .part prefixes. total_bytes int | None Total size, or None when any file's size is unknown. warnings tuple[str, ...] Notes from resolution. satisfied bool True when everything is already present and verified. remaining_bytes property remaining_bytes: int | None Bytes still to transfer, or None when the total is unknown. total_is_estimate property total_is_estimate: bool Whether any file's size was unknown, making the total a floor. AcquisitionReport dataclass AcquisitionReport( plan: AcquisitionPlan, entry: StoreEntry | None = None, downloaded_bytes: int = 0, reused: bool = False, cancelled: bool = False, dry_run: bool = False, warnings: tuple[str, ...] = (), ) The outcome of an acquisition. Attributes: Name Type Description plan AcquisitionPlan What was planned. entry StoreEntry | None The registered store entry, or None for a dry run or a cancellation. downloaded_bytes int Bytes transferred by this run. reused bool True when nothing had to be transferred. cancelled bool True when the caller stopped it. Partial transfers are kept. dry_run bool True when nothing was written. warnings tuple[str, ...] Everything the caller should know. path property path: Path | None The engine handle, when one was registered. AcquisitionRequest dataclass AcquisitionRequest( ref: SourceRef, model_id: str, variant_id: str = "", kind: str = "gguf", engine: str = "llama.cpp", quantization: str = "", license: str = "", token: str | None = None, max_concurrent_files: int = 3, allow_unverified: bool = False, enforce_license: bool = False, launch_hints: Mapping[str, Any] = dict(), ) Everything one acquisition needs. Attributes: Name Type Description ref SourceRef Where the bytes come from. model_id str Catalog model id, for the index and for progress reports. variant_id str Catalog variant id. kind str "gguf" or "hf_repo". engine str Which engine the result is for. quantization str What will be on disk. license str License id, checked when enforce_license is set. token str | None Credential for the source, when it needs one. max_concurrent_files int How many transfers may run at once. allow_unverified bool Accept files no digest can check. enforce_license bool Refuse licenses outside the allowlist. launch_hints Mapping[str, Any] Advisory engine arguments to attach to the result. plan_acquisition async plan_acquisition( request: AcquisitionRequest, *, store: ModelStore | None = None, client: AsyncClient | None = None, ) -> AcquisitionPlan Resolve a source and work out exactly what would be transferred. Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards. acquire async acquire( request: AcquisitionRequest, *, store: ModelStore | None = None, client: AsyncClient | None = None, progress: ProgressSink | None = None, plan: AcquisitionPlan | None = None, dry_run: bool = False, cancel_check: Callable[[], bool] | None = None, ) -> AcquisitionReport Acquire a model variant into the store. Parameters: Name Type Description Default request AcquisitionRequest What to acquire and from where. required store ModelStore | None The destination store; defaults to the standard root. None client AsyncClient | None An httpx2.AsyncClient, for tests or custom transports. None progress ProgressSink | None Aggregate progress sink. None plan AcquisitionPlan | None A plan from plan_acquisition, to avoid resolving twice. None dry_run bool Resolve and report sizes without writing anything. False cancel_check Callable[[], bool] | None Polled between chunks; returning True stops the acquisition cooperatively. Task cancellation works too and is the primary mechanism — this exists for the synchronous facade. None Returns: Type Description AcquisitionReport The report, naming the registered entry. Raises: Type Description LocalRuntimeError On a transfer failure, a digest mismatch, or insufficient disk. acquire_sync acquire_sync( request: AcquisitionRequest, *, store: ModelStore | None = None, progress: ProgressSink | None = None, dry_run: bool = False, cancel_check: Callable[[], bool] | None = None, ) -> AcquisitionReport Blocking wrapper around acquire, for callers with no event loop. Raises: Type Description RuntimeError If called from inside a running event loop, where it would deadlock. Use acquire there. launch_hints_for launch_hints_for( entry: StoreEntry, *, path: Path, context_size: int | None = None, gpu_layers: int | None = None, tensor_parallel_size: int | None = None, gpu_memory_utilization: float | None = None, ) -> dict[str, Any] Build the advisory engine arguments that accompany a located model. Data, not process control. These are keys a caller — the llama.cpp supervisor, a future vLLM launcher, or a user pasting a command line — turns into arguments. Producing them from numbers already computed is translation; launching is not, and nothing here starts a process. anyinfer.local.acquire_sync acquire_sync( request: AcquisitionRequest, *, store: ModelStore | None = None, progress: ProgressSink | None = None, dry_run: bool = False, cancel_check: Callable[[], bool] | None = None, ) -> AcquisitionReport Blocking wrapper around acquire, for callers with no event loop. Raises: Type Description RuntimeError If called from inside a running event loop, where it would deadlock. Use acquire there. anyinfer.local.plan_acquisition async plan_acquisition( request: AcquisitionRequest, *, store: ModelStore | None = None, client: AsyncClient | None = None, ) -> AcquisitionPlan Resolve a source and work out exactly what would be transferred. Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards. anyinfer.local.AcquisitionPlan dataclass AcquisitionPlan( entry_id: str, model_id: str, variant_id: str, kind: str, engine: str, quantization: str, directory: str, handle: str, files: tuple[RemoteFile, ...], already_have_bytes: int = 0, total_bytes: int | None = None, warnings: tuple[str, ...] = (), satisfied: bool = False, revision: str | None = None, repo: str | None = None, license: str = "", ) What an acquisition will do, before it does any of it. Attributes: Name Type Description entry_id str The store entry that will be written. model_id str The catalog model. variant_id str The catalog variant. kind str "gguf" or "hf_repo". engine str Which engine the result is for. quantization str What will be on disk. directory str Store-relative destination. handle str Store-relative engine handle. files tuple[RemoteFile, ...] Every file, resolved. already_have_bytes int Bytes already on disk — verified files plus resumable .part prefixes. total_bytes int | None Total size, or None when any file's size is unknown. warnings tuple[str, ...] Notes from resolution. satisfied bool True when everything is already present and verified. remaining_bytes property remaining_bytes: int | None Bytes still to transfer, or None when the total is unknown. total_is_estimate property total_is_estimate: bool Whether any file's size was unknown, making the total a floor. anyinfer.local.AcquisitionProgress dataclass AcquisitionProgress( model_id: str, variant_id: str, phase: AcquisitionPhase, file_index: int = 0, file_count: int = 0, filename: str = "", file_downloaded_bytes: int = 0, file_total_bytes: int | None = None, total_downloaded_bytes: int = 0, total_bytes: int | None = None, total_is_estimate: bool = False, session_bytes: int = 0, bytes_per_second: float | None = None, eta_seconds: float | None = None, ) One progress report for a whole acquisition. The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run. Attributes: Name Type Description model_id str The catalog model being acquired. variant_id str The variant being acquired. phase AcquisitionPhase Which stage this report came from. file_index int 1-based index of the file that most recently advanced. file_count int How many files this acquisition covers. filename str Name of the file that most recently advanced. file_downloaded_bytes int Bytes present for that file, including a resumed prefix. file_total_bytes int | None Expected size of that file, when known. total_downloaded_bytes int Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%. total_bytes int | None Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes. total_is_estimate bool True when any file's size came from a guess rather than a pinned or reported figure. session_bytes int Bytes this run actually transferred, which is what rate and ETA are derived from. bytes_per_second float | None Transfer rate, or None until there is a real sample. eta_seconds float | None Seconds remaining, or None on the same condition. A wildly wrong ETA in the first second is worse than no ETA. fraction property fraction: float | None Completion as a fraction, or None when the total is unknown. anyinfer.local.AcquisitionReport dataclass AcquisitionReport( plan: AcquisitionPlan, entry: StoreEntry | None = None, downloaded_bytes: int = 0, reused: bool = False, cancelled: bool = False, dry_run: bool = False, warnings: tuple[str, ...] = (), ) The outcome of an acquisition. Attributes: Name Type Description plan AcquisitionPlan What was planned. entry StoreEntry | None The registered store entry, or None for a dry run or a cancellation. downloaded_bytes int Bytes transferred by this run. reused bool True when nothing had to be transferred. cancelled bool True when the caller stopped it. Partial transfers are kept. dry_run bool True when nothing was written. warnings tuple[str, ...] Everything the caller should know. path property path: Path | None The engine handle, when one was registered. anyinfer.local.AcquisitionPhase module-attribute AcquisitionPhase = Literal[ "resolving", "planning", "downloading", "verifying", "placing", "done", ] Where an acquisition is. Every transition produces a callback, unthrottled. anyinfer.local.ProgressSink module-attribute ProgressSink = Callable[[AcquisitionProgress], None] Receives AcquisitionProgress. See its docstring for the threading contract. anyinfer.local.ModelStore ModelStore(root: Path | None = None) A directory of acquired models, with an index over it. Not thread-safe by construction; correctness across processes comes from the same cooperative file lock downloads use, taken around every index mutation. root property root: Path The store root. index_path property index_path: Path Where the index document lives. staging_dir staging_dir(entry_id: str) -> Path Where an in-progress acquisition writes its .part files. lock_path lock_path(entry_id: str) -> Path The cross-process lock guarding one entry. entry_dir entry_dir(entry: StoreEntry) -> Path Absolute path to an entry's directory. resolve_within resolve_within(entry: StoreEntry, relative: str) -> Path Resolve a path inside an entry, refusing anything that escapes it. Called for every file before it is opened. Names come from a remote API, so this is the containment gate, applied after resolution so a symlink cannot step outside. Raises: Type Description ConfigError If the path is unsafe or resolves outside the entry directory. load_index load_index() -> dict[str, StoreEntry] Read the index, tolerating absence and corruption. A store whose index cannot be parsed reports as empty rather than raising: the files are still there, rebuild_index can recover them, and refusing to work at all because a cache file got truncated would be the wrong trade. register register(entry: StoreEntry) -> StoreEntry Add or replace an index entry. Called only after every file has been verified: a half-complete multi-file set is never registered, which is what makes locate trustworthy. unregister unregister(entry_id: str) -> StoreEntry | None Drop an index entry without touching files. rebuild_index rebuild_index() -> list[StoreEntry] Drop entries whose files are gone, and re-stat the rest. The recovery path for a user who deleted a directory by hand. It does not invent entries for unknown directories — an unrecognized tree could be anything, and registering it would be a claim about bytes nobody checked. list_installed list_installed() -> list[StoreEntry] Every registered entry, id-ordered. get get(entry_id: str) -> StoreEntry | None One entry by id. find find( model_id: str, *, variant_id: str | None = None, quantization: str | None = None, engine: str | None = None, ) -> StoreEntry | None The best registered entry matching a model and optional constraints. "Best" is the most recently installed match, so re-acquiring at a different quantization changes what a bare model id resolves to, which is what a user who just downloaded something expects. locate locate( model_id: str, *, variant_id: str | None = None, quantization: str | None = None, engine: str | None = None, verify: bool = False, launch_hints: Mapping[str, Any] | None = None, ) -> ResolvedModel | None Find a stored model and return a path an engine can be launched against. No network I/O, ever. Verification is the deliberate exception to "always check": hashing forty gigabytes on every request would be absurd, so the rule is verify on install and on adoption, then on lookup compare size and mtime against the index and re-hash only on a mismatch. verify=True forces a full re-hash. Returns: Type Description ResolvedModel | None The located model, or None when it is absent or fails its check. check check( entry: StoreEntry, *, deep: bool = False ) -> tuple[str, ...] Report what is wrong with a stored entry, cheaply by default. The shallow check compares size and mtime against the index; the deep check re-hashes. Either way an empty result means "as installed". disk_usage disk_usage() -> int Total bytes the registered entries occupy, excluding external ones. remove remove(entry_id: str) -> RemovalReport Delete an entry's files and unregister it. An external entry — one adopted from somebody else's cache — is only unregistered. Deleting files this store never wrote would be overstepping. clear_staging clear_staging(entry_id: str) -> None Remove an entry's staging directory and its partial transfers. adopt_legacy_flat adopt_legacy_flat( artifacts: Sequence[Any], ) -> list[StoreEntry] Register pre-existing flat-layout GGUF files without moving or re-fetching them. Earlier builds wrote /.gguf with no revision in the path. Those files are perfectly good bytes a user paid bandwidth for, so the store adopts them where they lie, but only after verifying each one against its catalog hash, so adoption is never a lie about what is on disk. Parameters: Name Type Description Default artifacts Sequence[Any] Pinned catalog artifacts to look for, each with id and files. required Returns: Type Description list[StoreEntry] Newly registered entries. adopt_external adopt_external( directory: Path, *, entry_id: str, model_id: str, variant_id: str, kind: str = "hf_repo", engine: str = "vllm", quantization: str = "", source: Mapping[str, Any] | None = None, expected: Mapping[str, str] | None = None, ) -> StoreEntry | None Register a directory this store does not own, if every file checks out. The Hugging Face cache case. We do not adopt another library's layout for our own writes — it is their private implementation detail, but re-downloading forty gigabytes the user already has is user-hostile. Adopted entries are marked StoreEntry.external, are never written to, and are never deleted by remove. Parameters: Name Type Description Default directory Path The existing snapshot directory. required entry_id str Id to register it under. required model_id str Catalog model this realizes. required variant_id str Catalog variant this realizes. required kind str Artifact kind. 'hf_repo' engine str Engine the variant targets. 'vllm' quantization str What is on disk. '' source Mapping[str, Any] | None Provenance to record. None expected Mapping[str, str] | None Per-relative-path sha256 that every file must match. Adoption is refused outright when this is empty — an unverified adoption is a guess. None Returns: Type Description StoreEntry | None The registered entry, or None when verification failed. anyinfer.local.StoreEntry dataclass StoreEntry( id: str, kind: str = "gguf", model_id: str = "", variant_id: str = "", quantization: str = "", engine: str = "llama.cpp", source: Mapping[str, Any] = dict(), directory: str = "", handle: str = "", files: tuple[StoredFile, ...] = (), license: str = "", installed_at: float = 0.0, last_used_at: float = 0.0, external: bool = False, warnings: tuple[str, ...] = (), ) One acquired model in the store. Attributes: Name Type Description id str Stable entry id, also the lock and staging name. kind str "gguf" (a file set) or "hf_repo" (a directory snapshot). model_id str The catalog model this realizes, when it came from the catalog. variant_id str The catalog variant, when it came from the catalog. quantization str The quantization on disk. engine str Which engine this variant is for. source Mapping[str, Any] How it was acquired, including the resolved immutable revision. directory str Where the files live, relative to the store root. handle str The path an engine is pointed at, relative to the store root — the first shard for GGUF, the directory for a snapshot. files tuple[StoredFile, ...] Every file, with digests. license str License id recorded at acquisition. installed_at float Unix timestamp of successful registration. last_used_at float Unix timestamp of the most recent ModelStore.locate. external bool True when the bytes are owned by something else (an adopted Hugging Face cache). Removal only unregisters an external entry; it never deletes. warnings tuple[str, ...] Anything the user should know — unverified files, most importantly. total_bytes property total_bytes: int Bytes this entry occupies. verified property verified: bool Whether every file was verified against a digest at install time. to_json to_json() -> dict[str, Any] Serialize for the index. from_json classmethod from_json(data: Mapping[str, Any]) -> StoreEntry Parse one index entry. anyinfer.local.ResolvedModel dataclass ResolvedModel( entry_id: str, kind: str, path: Path, quantization: str | None = None, engine: str = "llama.cpp", verified: bool = False, warnings: tuple[str, ...] = (), launch_hints: Mapping[str, Any] = dict(), ) A located model, ready to launch an engine against. Attributes: Name Type Description entry_id str The store entry. kind str "gguf" or "hf_repo". path Path The engine handle — a file for GGUF, a directory for a snapshot. quantization str | None What is actually on disk. engine str Which engine this variant is for. verified bool Whether every file was verified. warnings tuple[str, ...] Notes carried from the entry. launch_hints Mapping[str, Any] Engine-shaped arguments a caller can turn into a command line. Advisory data, not process control: this module locates weights, it does not start servers. anyinfer.local.RemovalReport dataclass RemovalReport( entry_id: str, removed: bool = False, freed_bytes: int = 0, external: bool = False, ) The outcome of removing an entry. Attributes: Name Type Description entry_id str What was removed. removed bool Whether an entry was found and unregistered. freed_bytes int Bytes reclaimed; zero for an external entry, which is only unregistered. external bool Whether the files were left alone because something else owns them. Sources Where weights come from. Adding an internal mirror is a resolver, not a dependency. anyinfer.local.SourceRef dataclass SourceRef( resolver: str = "huggingface", repo: str | None = None, revision: str | None = None, files: tuple[str, ...] = (), digests: Mapping[str, str] = dict(), sizes: Mapping[str, int] = dict(), roles: Mapping[str, str] = dict(), urls: tuple[str, ...] = (), include: tuple[str, ...] = (), exclude: tuple[str, ...] = (), path: str | None = None, ) A declarative pointer at a set of remote (or already-local) files. Attributes: Name Type Description resolver str Which resolver understands this reference. repo str | None Repository id, for repository-shaped resolvers ("Qwen/Qwen2.5-7B-Instruct"). revision str | None Branch, tag, or — preferably — an immutable commit sha. files tuple[str, ...] Explicit file list. Empty means "whatever the include globs match". digests Mapping[str, str] Per-file expected sha256, when the catalog pinned them. sizes Mapping[str, int] Per-file expected byte counts, when the catalog pinned them. roles Mapping[str, str] Optional per-file roles for companion artifacts such as a vision projector. urls tuple[str, ...] Direct download URLs, for the url resolver. include tuple[str, ...] Glob patterns selecting files from a repository listing. exclude tuple[str, ...] Glob patterns removing files the include list matched. path str | None An existing on-disk location, for the local resolver. to_json to_json() -> dict[str, object] Serialize for the store index. anyinfer.local.SourceResolver Bases: Protocol Turns a SourceRef into a ResolvedArtifact. scheme instance-attribute scheme: str The SourceRef.resolver value this implementation answers to. resolve async resolve( ref: SourceRef, *, token: str | None = None, client: Any | None = None, ) -> ResolvedArtifact Expand a reference into a concrete file list. client is an optional httpx2.AsyncClient the caller already owns. Passing it keeps resolution and the subsequent transfer on one connection pool, and is what makes a resolver testable against a mock transport. anyinfer.local.ResolvedArtifact dataclass ResolvedArtifact( resolver: str, files: tuple[RemoteFile, ...] = (), repo: str | None = None, revision: str | None = None, warnings: tuple[str, ...] = (), ) A concrete, ordered file list ready for acquisition. Attributes: Name Type Description resolver str Which resolver produced this. files tuple[RemoteFile, ...] Every file to fetch, in acquisition order. repo str | None The repository this came from, when applicable. revision str | None The immutable revision this resolved to, when the resolver could determine one. A branch name is always resolved to a commit before use. warnings tuple[str, ...] Anything the caller should know — unverifiable files, skipped pickle weights, and so on. total_bytes property total_bytes: int | None Sum of every file size, or None when any one is unknown. anyinfer.local.RemoteFile dataclass RemoteFile( path: str, url: str, size_bytes: int | None = None, digest: str = "", digest_kind: DigestKind = "none", ) One file a resolver decided belongs to an artifact. Attributes: Name Type Description path str Destination path relative to the entry directory, POSIX-separated. url str Where to fetch it. size_bytes int | None Expected size, or None when genuinely unknown. digest str Expected digest, lowercase hex. digest_kind DigestKind How to compute digest. "none" means unverifiable. filename property filename: str The final path component. Artifacts and Downloads anyinfer.local.GgufArtifact dataclass GgufArtifact( id: str, files: tuple[GgufFile, ...], license: str = "", description: str = "", parameter_size: str | None = None, quantization: str | None = None, est_ram_bytes: int | None = None, est_vram_bytes: int | None = None, embedding: EmbeddingCapabilities | None = None, ) A pinned, verifiable local model artifact. Attributes: Name Type Description id str Stable artifact id, the handle alias targets and the llama.cpp adapter use. files tuple[GgufFile, ...] Every file the artifact comprises — one entry, or several for a sharded model. license str License id; checked against the download allowlist for application-supplied entries. description str Free text for display. parameter_size str | None Parameter class ("7B"), when known. quantization str | None Quantization of the shipped weights ("Q4_K_M"), when known. est_ram_bytes int | None Estimated memory needed on the CPU-only path. est_vram_bytes int | None Estimated memory needed when fully offloaded. embedding EmbeddingCapabilities | None Vector facts, when the artifact's weights are an embedding model rather than a chat model. Its presence is what marks the artifact as one llama-server must be started with --embeddings to serve. total_size_bytes property total_size_bytes: int | None Sum of every file's size, or None when any is unknown. is_sharded property is_sharded: bool Whether this artifact spans multiple files. projector property projector: GgufFile | None Pinned multimodal projector file, when this model has one. __post_init__ __post_init__() -> None Require model weights first and at most one projector companion. anyinfer.local.GgufFile dataclass GgufFile( filename: str, url: str, sha256: str = "", size_bytes: int | None = None, role: Literal["model", "projector"] = "model", ) One file of a (possibly sharded) GGUF artifact. Attributes: Name Type Description filename str The name the file is stored under in the model directory. url str Pinned download URL. sha256 str Expected content hash. Empty means the file cannot be verified, and the downloader warns instead of checking. size_bytes int | None Expected size, when known; feeds download progress totals. anyinfer.local.download_artifact download_artifact( artifact: GgufArtifact, *, model_dir: Path | None = None, progress: ProgressCallback | None = None, client: Client | None = None, enforce_license: bool = False, ) -> DownloadReport Ensure every file of an artifact is present and verified. Parameters: Name Type Description Default artifact GgufArtifact The pinned catalog entry. required model_dir Path | None Destination directory; defaults to default_model_dir(). None progress ProgressCallback | None Called as bytes arrive. None client Client | None An httpx2.Client to use, for tests or custom transports. None enforce_license bool Reject artifacts whose license is not in ALLOWED_LICENSES. Applied to application-supplied entries. False Returns: Type Description DownloadReport A report naming the on-disk files. Raises: Type Description LocalRuntimeError On a hash mismatch, a transfer failure, or a rejected license. anyinfer.local.iter_missing iter_missing( artifacts: Iterable[GgufArtifact], model_dir: Path | None = None, ) -> list[GgufArtifact] Which artifacts are absent or fail verification. anyinfer.local.verify_file verify_file(path: Path, expected_sha256: str) -> bool Whether a file matches its expected hash. An artifact with no recorded hash cannot be verified; its mere existence is accepted, and the caller is warned. anyinfer.local.artifact_paths artifact_paths( artifact: GgufArtifact, model_dir: Path | None = None ) -> tuple[Path, ...] Where an artifact's files would live on disk. anyinfer.local.default_model_dir default_model_dir() -> Path Where downloaded artifacts live by default. anyinfer.local.DownloadReport dataclass DownloadReport( artifact_id: str, paths: tuple[Path, ...], downloaded_bytes: int = 0, reused: bool = False, warnings: tuple[str, ...] = (), ) The outcome of ensuring an artifact is present. Attributes: Name Type Description artifact_id str The artifact this report describes. paths tuple[Path, ...] Where each of the artifact's files lives on disk, in declaration order. downloaded_bytes int Bytes actually transferred; zero when everything was reused. reused bool Whether every file was already present and verified, so nothing was fetched. warnings tuple[str, ...] Non-fatal notes — files with no recorded hash, or files that failed verification and were re-downloaded. primary_path property primary_path: Path The file to hand to llama-server (the first shard of a sharded artifact). anyinfer.local.ProgressCallback module-attribute ProgressCallback = Callable[[str, int, int | None], None] (artifact_id, downloaded_bytes, total_bytes_or_None). anyinfer.local.ALLOWED_LICENSES module-attribute ALLOWED_LICENSES = frozenset( { "apache-2.0", "falcon-llm-2.0", "gemma-terms", "llama-3.1-community", "llama-3.2-community", "llama-3.3-community", "mit", "openrail-m", } ) Licenses permitted for catalog entries, compared case-insensitively. The bundled catalog is curated; entries an application adds are checked so that a convenience feature cannot quietly redistribute weights under terms the user has not seen. Non-commercial and research-only terms are deliberately absent — an application that has accepted those terms adds the model through a catalog overlay, which is an explicit act. anyinfer.local.license_allowed license_allowed(license_id: str) -> bool Whether a license id is in ALLOWED_LICENSES, ignoring case. Server Supervision anyinfer.local.ServerSupervisor ServerSupervisor( *, binary: Path | str = "llama-server", hardware: HardwareProfile | None = None, runtime_backend: AcceleratorKind | None = None, idle_ttl_s: float | None = 900.0, max_resident: int = 1, allow_remote_exposure: bool = False, host: str = LOOPBACK_HOST, health_timeout_s: float = _HEALTH_TIMEOUT_S, on_lifecycle: LifecycleCallback | None = None, ) Owns the llama-server processes for one client. Parameters: Name Type Description Default binary Path | str Path to the llama-server executable. 'llama-server' hardware HardwareProfile | None Detected hardware, used for VRAM admission control. None runtime_backend AcceleratorKind | None A required installed backend family, or None to select the best runtime the detected hardware can drive. None idle_ttl_s float | None Unload a server after this long with no active streams. None keeps servers until the supervisor closes. 900.0 max_resident int How many servers may run at once. Exceeding it evicts the least-recently-used idle server. 1 allow_remote_exposure bool Bind a non-loopback address. Off by default: a local model server is loopback-only unless deliberately exposed. False on_lifecycle LifecycleCallback | None Called with lifecycle events. None resident_models property resident_models: tuple[str, ...] Model keys with a running server. resident_plans property resident_plans: Mapping[str, ServerPlan] The launch plan each running server was started with. What the tuner decided, which is not always what the caller assumed: a plan that offloaded no layers explains a local model running an order of magnitude slower than the same weights did on the same machine last week. acquire async acquire( model_key: str, model_path: Path, plan: ServerPlan, *, persist: bool = False, ) -> ManagedServer Get a ready server for a model, starting or reusing one. Blocks until the server answers its health probe. Concurrent callers requesting different models are serialized, so loads never overlap. Raises: Type Description LocalRuntimeError If the model cannot fit, the binary is missing, or the server fails to become ready. set_hardware set_hardware(hardware: HardwareProfile) -> None Late-bind the detected hardware profile. Detection is deliberately lazy — probing at construction would tax clients that never run a local model, so the adapter hands the profile over once it has one. Admission control and backend fallback stay disabled until then. set_runtime_backend set_runtime_backend( backend: AcceleratorKind | None, ) -> None Select a named installed backend, or return to automatic selection. Existing child processes keep the executable they started with; this affects only later server starts. resolve_binary resolve_binary() -> Path Locate the llama-server executable without starting anything. Public so a health probe can answer "could a server start?" cheaply. Falls back to the best installed backend variant for this hardware when the configured name is not on PATH (a CUDA build in a known runtime directory beats a missing PATH entry). Raises: Type Description LocalRuntimeError When no usable binary exists anywhere. collect_idle async collect_idle() -> int Stop servers idle beyond the TTL. Returns how many were stopped. Call periodically. Servers with active streams are never collected, however long the generation has been running. aclose async aclose() -> None Stop every supervised server. anyinfer.local.ManagedServer ManagedServer(handle: ServerHandle) A context manager marking a server busy for the duration of a request. This is what makes the idle timer honest: the server is busy while a stream is open, not merely while a request is arriving. base_url property base_url: str The server's base URL. __enter__ __enter__() -> ManagedServer Mark the server busy. take_load_ms take_load_ms() -> float | None This request's share of a cold start: the load it caused, or None. __exit__ __exit__(*exc: object) -> None Release the server and restart its idle clock. anyinfer.local.ServerHandle dataclass ServerHandle( model_key: str, model_path: Path, plan: ServerPlan, host: str, port: int, process: Popen[bytes], started_at: float, load_ms: float | None = None, log_tail: deque[str] = ( lambda: deque(maxlen=_LOG_TAIL_LINES) )(), active_streams: int = 0, last_activity: float = time.monotonic(), persist: bool = False, stopping: bool = False, ) A running llama-server and everything known about it. model_key instance-attribute model_key: str The model this server serves; also its key in the supervisor's server table. model_path instance-attribute model_path: Path The GGUF file the server was started with. plan instance-attribute plan: ServerPlan The tuned launch plan; its memory estimate is what admission control committed. host instance-attribute host: str Interface the server is bound to — loopback unless exposure was explicitly allowed. port instance-attribute port: int TCP port the server listens on, allocated just before spawning. process instance-attribute process: Popen[bytes] The supervised child, polled for liveness and terminated on stop. started_at instance-attribute started_at: float Monotonic time the child was spawned. load_ms class-attribute instance-attribute load_ms: float | None = None How long this server took to become ready, in milliseconds, or None once that has been reported. The supervised runtime's equivalent of a hosted engine's load duration. It is a property of the request that caused the start, not of the server, so it is consumed exactly once — every later request on the same server is warm by definition, and re-reporting the original load would turn one cold start into a permanent one. log_tail class-attribute instance-attribute log_tail: deque[str] = field( default_factory=lambda: deque(maxlen=_LOG_TAIL_LINES) ) The child's most recent output lines, kept so failures can explain themselves. active_streams class-attribute instance-attribute active_streams: int = 0 Open response streams. Nonzero means busy: the idle clock and eviction ignore it. last_activity class-attribute instance-attribute last_activity: float = field(default_factory=time.monotonic) Monotonic time of the last request or stream release; the idle clock's baseline. persist class-attribute instance-attribute persist: bool = False Exempt this server from idle collection and capacity eviction. stopping class-attribute instance-attribute stopping: bool = False Set while the supervisor is tearing this server down. base_url property base_url: str The OpenAI-compatible base URL this server serves. is_running property is_running: bool Whether the child process is still alive. is_idle property is_idle: bool Whether no request is currently streaming from this server. idle_seconds idle_seconds() -> float How long this server has been idle. Zero while any stream is active. touch touch() -> None Mark activity, resetting the idle clock. take_load_ms take_load_ms() -> float | None Return this server's load duration once, then forget it. anyinfer.local.LifecycleCallback module-attribute LifecycleCallback = Callable[[ServerLifecycle], None] Receives every lifecycle transition of a supervised server. anyinfer.local.allocate_port allocate_port(host: str = LOOPBACK_HOST) -> int Reserve an ephemeral port by binding and immediately releasing it. Inherently racy, but the alternative — letting llama-server pick and then discovering which port it chose — requires parsing its log output, which is far more fragile. anyinfer.local.LOOPBACK_HOST module-attribute LOOPBACK_HOST = '127.0.0.1' Local servers bind loopback only, unless the caller explicitly opts out. anyinfer.local.is_loopback is_loopback(base_url: str | None) -> bool Whether a base URL points at this machine. Used by two callers with the same underlying question — "is the thing at the other end of this URL running on hardware I can probe?". A remote Ollama daemon answers no, and everything downstream (hardware detection, fit classification, zero-cost pricing) depends on not pretending otherwise. A URL that cannot be parsed is treated as not loopback, because the safe default is to assume someone else's machine. Recommendation anyinfer.local.recommend_alias recommend_alias( hardware: HardwareProfile, catalog: TierSource, *, prefer_accelerated: bool = True, ) -> Recommendation Recommend the largest catalog tier this machine can comfortably run. Parameters: Name Type Description Default hardware HardwareProfile The detected profile. required catalog TierSource The catalog whose aliases carry min_ram_bytes/min_vram_bytes. required prefer_accelerated bool Budget against VRAM when an accelerator is present. With unified memory, system RAM is the budget regardless. True Returns: Type Description Recommendation A recommendation. When memory is unknown, the smallest tier is proposed with Recommendation confident=False rather than guessing upward. anyinfer.local.Recommendation dataclass Recommendation( alias: str | None, reason: str, confident: bool = True ) A recommended tier and the reasoning behind it. Attributes: Name Type Description alias str | None The recommended alias, or None when nothing fits. reason str Why this tier was chosen, for display to a user. confident bool False when the machine's memory could not be determined, so the recommendation is a floor rather than a fit. anyinfer.local.Tier Bases: Protocol The subset of a catalog alias this module needs. Structural rather than nominal so the local subsystem does not import the catalog: artifacts (local data) are depended on by the catalog, and a reverse dependency here would make that a cycle. name property name: str The alias name. min_ram_bytes property min_ram_bytes: int | None System RAM this tier needs, when stated. min_vram_bytes property min_vram_bytes: int | None Accelerator memory this tier needs, when stated. anyinfer.local.TierSource Bases: Protocol The subset of a catalog needed to recommend a tier. alias_names alias_names() -> tuple[str, ...] Every alias name. alias alias(name: str) -> Tier Look up one alias. Discovery What this machine can already use: engines answering on loopback, and credential variables that are actually set. This is what anyinfer init composes into a configuration file. anyinfer.local.discover async discover( registry: ProviderRegistry, *, timeout_s: float = _PROBE_TIMEOUT_S, probe: bool = True, keyring: bool = False, environ: Mapping[str, str] | None = None, transports: Mapping[str, Any] | None = None, ) -> tuple[DiscoveredProvider, ...] Report every provider this machine can already use. Parameters: Name Type Description Default registry ProviderRegistry Which providers to consider. Endpoints and variables both come from the descriptors it holds, so a registry carrying third-party providers discovers them on equal terms. required timeout_s float Wall clock for each endpoint probe. Endpoints are probed concurrently, so this bounds the whole endpoint phase rather than summing across it. _PROBE_TIMEOUT_S probe bool Whether to contact endpoints at all. False restricts discovery to credential evidence, which touches no socket. True keyring bool Whether to consult the OS credential vault. Off by default: an environment variable is already in this process, while reading a vault can prompt the user to unlock it, so vault evidence is asked for rather than collected for free. False environ Mapping[str, str] | None Environment to inspect; defaults to this process's. None transports Mapping[str, Any] | None Test seam — httpx2 transports keyed by provider id, used when building the probe adapter so a test can prove the probe logic without opening a socket. None Returns: Type Description DiscoveredProvider The evidence, endpoint findings first and each in registry order. At most one ... entry per provider: an engine that is both running and holds a key in the tuple[DiscoveredProvider, ...] environment is reported as running, since that is the stronger observation. Raises: Type Description ConfigError If keyring=True and the [keyring] extra is not installed. Asked for a vault and unable to open one, reporting "nothing found" would be a lie by omission. anyinfer.local.DiscoveredProvider dataclass DiscoveredProvider( provider_id: str, base_url: str | None, evidence: DiscoveryEvidence, detail: str, models: tuple[str, ...] = (), embedding_models: tuple[str, ...] = (), credential_key: str = "", credential_ref: str = "", ) A provider found usable on this machine, and the evidence for it. Attributes: Name Type Description provider_id str Registered id of the provider this evidence is for. base_url str | None The endpoint that answered, or the provider's default when the evidence is a credential rather than a running service. None when the provider has no default endpoint. evidence DiscoveryEvidence What was observed; see DiscoveryEvidence. detail str One line naming the observation, for display — "4 models", "ANTHROPIC_API_KEY set". Never contains a credential value. models tuple[str, ...] Model ids the endpoint listed, when it listed any. Empty for credential evidence, which says nothing about what a provider serves. embedding_models tuple[str, ...] The subset of models the endpoint stamped with the embedding operation (LM Studio and Cohere discovery do this; most providers report generation only, so this is empty far more often than models is). A separate field rather than replacing models — the ids stay the flat list every existing caller expects, and this is additive evidence for a caller that specifically wants to route embedding traffic. credential_key str Setup-field key this credential satisfies ("api_key"), or empty for endpoint evidence. credential_ref str The reference a configuration file should carry for that field — "env://ANTHROPIC_API_KEY", "credential://system/openai-api-key". A reference, never a value: this is the field a config writer copies, and it is built here precisely so no caller is ever tempted to resolve one first. anyinfer.local.DiscoveryEvidence module-attribute DiscoveryEvidence = Literal[ "endpoint", "environment", "credential-store" ] How a provider was found to be usable. endpoint A loopback address the provider declares as its default answered a model listing. environment A variable the provider declares as its conventional credential source is set and non-blank. Its value was not read. credential-store A secret is stored in the OS vault under a conventional identifier. Only ever produced when a caller passed keyring=True. anyinfer.local.endpoint_candidates endpoint_candidates( registry: ProviderRegistry, ) -> tuple[tuple[str, ...], ...] The loopback endpoints discover would contact, grouped by shared address. Returns: Type Description tuple[str, ...] One entry per distinct endpoint, as (base_url, provider_id, provider_id, …) ... in registry order. Several engines share a port — llamafile, localai, tuple[tuple[str, ...], ...] ramalama and llama-swap all default to 8080, so an address that answers tuple[tuple[str, ...], ...] cannot be attributed to one of them by probing alone, and grouping is how that tuple[tuple[str, ...], ...] stays visible instead of becoming a coin flip. A caller that wants to tell a user exactly what was contacted reads this; the same grouping is what keeps discover to one request per address. anyinfer.local.KEYRING_IDENTIFIER_SUFFIX module-attribute KEYRING_IDENTIFIER_SUFFIX = '-api-key' Suffix of the vault identifier discovery looks under, after the provider id. There is no protocol here to follow — a vault entry is whatever someone chose to call it — so discovery looks under two conventional spellings per provider (openai and openai-api-key) and finds nothing otherwise. A caller who named their entry something else writes the credential:// reference themselves; that is one line of configuration, and it is better than a command that rummages through a credential store by prefix. Engine-Managed Models anyinfer.PullRequest dataclass PullRequest( model: str, base_url: str, timeout_s: float = PULL_TIMEOUT_S, transport: Any | None = None, progress: Callable[[DownloadProgress], None] | None = None, ) What a puller needs to make one model available on one engine. Attributes: Name Type Description model str The model name in the engine's own namespace ("qwen3:8b"). base_url str The engine's endpoint, after defaults and shorthand expansion. timeout_s float Wall clock for the whole transfer. transport Any | None Optional httpx2 transport override, for tests. progress Callable[[DownloadProgress], None] | None Sink for DownloadProgress events, or None for a silent pull. anyinfer.PullReport dataclass PullReport( model: str, already_present: bool = False, bytes_transferred: int = 0, detail: str = "", ) What a pull did. Attributes: Name Type Description model str The model that is now available. already_present bool Whether the engine reported it was already there, so nothing was transferred. Worth distinguishing: "took two seconds" is reassuring when it means already installed and alarming when it means downloaded 8 GB. bytes_transferred int Bytes the engine reported pulling, when it reported any. detail str The engine's final status line. Confidential Execution Attestation Tier 3 of the Confidentiality Tiers: whether this host can back an attested-local-execution guarantee, and does it, right now. Advisory detection only; enforcement is anyinfer.providers.confidential_execution.ConfidentialExecutionAdapter, which calls the same function this section documents. anyinfer.local.confidential_execution_status confidential_execution_status( *, backend: Backend, model: ResolvedModel | None = None, use_cache: bool = True, manifest: ModelManifest | None = None, vendor_public_key: bytes | None = None, ) -> ConfidentialExecutionStatus Detect what this box can guarantee for confidential local execution. Never raises: anything that cannot be determined becomes "not detected," never a guess. Callers — including ConfidentialExecutionAdapter's own fail-closed check — treat the result as advice about a hardware fact, not as a decision. Parameters: Name Type Description Default backend Backend The selected local backend (used only to know whether this run targets a GPU-accelerated build at all, alongside model). required model ResolvedModel | None The selected model, when known. Its launch_hints["n_gpu_layers"] determines gpu_offload_required. When None (a caller checking capability before choosing a model), gpu_offload_required is conservatively True unless backend itself is a CPU-only build — a capability check must never over-promise before a model is even chosen. None use_cache bool Read and write the disk cache for the hardware-detection portion of the result. Overridden by ATTESTATION_CACHE_BYPASS_ENV and ATTESTATION_CACHE_REFRESH_ENV. Tier 4 verification is never cached, regardless of this flag — a swapped model file must be caught on the very next call, not masked by a stale cache entry. True manifest ModelManifest | None Tier 4 — a vendor-signed provenance.ModelManifest to verify model's weights against. Requires model and vendor_public_key too; ignored otherwise. None vendor_public_key bytes | None The vendor's Ed25519 public key for manifest's signature. None Returns: Type Description ConfidentialExecutionStatus The detected status. Raises: Type Description ConfigError manifest was supplied but the attest extra (pip install anyinfer[attest]) is not installed. anyinfer.local.ConfidentialExecutionStatus dataclass ConfidentialExecutionStatus( cpu_tee: CpuTeeKind | None, gpu_cc_capable: bool, gpu_cc_enabled: bool, gpu_offload_required: bool, end_to_end: bool, detail: str, model_verified: bool | None = None, ) What this box can actually guarantee for confidential local execution right now. Attributes: Name Type Description cpu_tee CpuTeeKind | None Detected CPU TEE, or None. gpu_cc_capable bool The primary detected GPU supports CC mode at all (a hardware fact); False when no GPU was detected. gpu_cc_enabled bool CC mode is actually active in the current driver/runtime configuration, when gpu_cc_capable is True. gpu_offload_required bool Whether the selected model's launch plan offloads to GPU at all; when False, gpu_cc_capable/gpu_cc_enabled do not gate end_to_end — a CPU-only backend has no PCIe bridge to worry about. end_to_end bool The one field most callers branch on — see the module docstring for the exact definition. detail str Human-readable why, the same role Backend.detail already plays. model_verified bool | None Tier 4 — whether confidential_execution_status's optional manifest/vendor_public_key arguments were supplied and verified against the running model's weights on disk. None means "not evaluated" (no manifest was supplied), never "failed." This field alone is not a Tier 4 claim — a hash-and-signature check on an unattested host is a real but weaker guarantee; only model_verified is True and end_to_end is True together is the full Tier 4 claim (see provenance.py's module docstring). anyinfer.local.CpuTeeKind module-attribute CpuTeeKind = Literal['sev-snp', 'tdx', 'nitro', 'sgx'] CPU TEE families this module can detect. sgx and nitro are detected and reported for completeness — a caller asking "what did you find" deserves the whole answer — but neither is part of the v1 end_to_end claim: SGX's enclave-shaped programming model is not the lift-and-shift story SEV-SNP/TDX give, and Nitro Enclaves have no persistent storage or general networking, so serving a model inside one needs real integration work this module does not attempt to paper over (see the market findings in DESIGN.md §30.4). anyinfer.local.attestation_cache_path attestation_cache_path() -> Path Where the detection cache lives — the same directory hardware.py uses. anyinfer.local.ATTESTATION_CACHE_BYPASS_ENV module-attribute ATTESTATION_CACHE_BYPASS_ENV = ( "ANYINFER_ATTESTATION_CACHE_BYPASS" ) Set to skip the cache entirely (read and write). anyinfer.local.ATTESTATION_CACHE_REFRESH_ENV module-attribute ATTESTATION_CACHE_REFRESH_ENV = ( "ANYINFER_ATTESTATION_CACHE_REFRESH" ) Set to ignore a cached result and re-probe, then rewrite the cache. ConfidentialExecutionAdapter anyinfer.providers.confidential_execution.ConfidentialExecutionAdapter ConfidentialExecutionAdapter( inner: ProviderAdapter, *, backend: Backend, model: ResolvedModel | None = None, ) Wraps a local ProviderAdapter, refusing generate() unless attestation succeeds. Discovery and health pass straight through to the inner adapter unchanged — attestation is a property of execution, not of what models are discoverable or whether the process is reachable at all. Bind the wrapper to one inner adapter and the backend/model it will attest. Parameters: Name Type Description Default inner ProviderAdapter An already-configured local adapter instance to delegate to once attestation succeeds. required backend Backend The local backend inner runs — passed straight to confidential_execution_status on every generate() call. required model ResolvedModel | None The selected model, when known; also passed straight through. See confidential_execution_status's own docstring for what a missing model means for the check. None list_models async list_models() -> Sequence[DiscoveredModel] Delegate to the inner adapter unchanged. health async health() -> Health Delegate to the inner adapter unchanged. aclose async aclose() -> None Delegate to the inner adapter unchanged. generate async generate(req: WireRequest) -> AsyncIterator[AdapterEvent] Attest, then generate — or refuse, and never touch the inner adapter at all. Raises: Type Description ConfidentialExecutionError The attested guarantee is not available on this host right now. Carries ConfidentialExecutionStatus.detail so a caller can render why. Model Provenance Verification (Tier 4) Whether the model weights actually on disk are the exact artifact a vendor signed (verification only, never signing); see the module docstring for why that boundary is absolute. Only a Tier 4 claim in combination with ConfidentialExecutionStatus.end_to_end; see the Confidentiality Tiers guide. anyinfer.local.ModelManifest dataclass ModelManifest( model_id: str, weight_hash: str, vendor_key_id: str, signed_at: str, signature: bytes, ) A vendor-signed record of what one set of model weights should hash to. Attributes: Name Type Description model_id str The vendor's identifier for this model variant. weight_hash str SHA-256 of the weight file (or, for a multi-file snapshot, of the sorted relative_path:sha256 listing — see hash_model_weights), as a hex string. vendor_key_id str Which vendor key this manifest was signed with, for a caller managing more than one registered vendor public key. signed_at str ISO-8601 signing timestamp, for audit trails. signature bytes The vendor's signature over this manifest's canonical payload. to_dict to_dict() -> dict[str, Any] A JSON-safe mapping — the on-disk manifest format. from_dict classmethod from_dict(data: dict[str, Any]) -> ModelManifest Load a manifest previously written by to_dict. anyinfer.local.hash_model_weights hash_model_weights(path: Path) -> str Hash the weights at path, hex-encoded SHA-256. A single file (the GGUF case) is hashed directly. A directory (an hf_repo snapshot) is hashed as the SHA-256 of a sorted relative_path:sha256\n listing over every file it contains — deterministic regardless of filesystem enumeration order, and sensitive to every file's content, name, and presence. anyinfer.local.verify_model_manifest verify_model_manifest( manifest: ModelManifest, *, weights_path: Path, vendor_public_key: bytes, ) -> bool Verify a manifest's signature and that it matches the weights on disk. Parameters: Name Type Description Default manifest ModelManifest The vendor-signed manifest to check. required weights_path Path Where the model weights actually are — re-hashed and compared against manifest.weight_hash; a manifest is never trusted for the hash alone, since that would make the signature pointless. required vendor_public_key bytes The registered vendor's Ed25519 public key. required Returns: Type Description bool True only when the signature verifies against vendor_public_key and the bool recomputed hash of weights_path matches manifest.weight_hash exactly. Raises: Type Description ConfigError The attest extra is not installed. --- # Reference / Sidecar Frontend Source: https://anyinfer.dev/reference/api/serve/ Serve The anyinfer.serve frontend: an OpenAI-compatible loopback service over any configured provider, embeddable as an ASGI app. Guide: serve. Application anyinfer.serve.create_app create_app( client: Any, *, auth_token: str | None = None, expose_targets: Sequence[str] = (), ) -> Any Build the ASGI application. Parameters: Name Type Description Default client Any An AsyncClient to federate through. required auth_token str | None Bearer token clients must present. None disables authentication, which is only appropriate on loopback. None expose_targets Sequence[str] Concrete provider:model targets to advertise from /v1/models, in addition to catalog aliases. () Returns: Type Description Any A Starlette application. Raises: Type Description ConfigError If the [serve] extra is not installed. OpenAI Codec The translation layer between the OpenAI wire dialect and AnyInfer's native types (see the architecture overview). Useful directly when embedding the frontend or building a custom edge. anyinfer.serve.request_from_openai request_from_openai( body: Mapping[str, Any], ) -> tuple[str, GenerationRequest, bool] Decode an OpenAI chat-completions request body. Parameters: Name Type Description Default body Mapping[str, Any] The parsed request JSON. required Returns: Type Description str A (target, request, stream) triple. target is the model field taken GenerationRequest verbatim — an AnyInfer target is an OpenAI model string (invariant 3), which is bool what makes federation free. anyinfer.serve.request_to_openai request_to_openai( target: str, request: GenerationRequest, *, stream: bool = False, ) -> dict[str, Any] Encode a request back into OpenAI wire form. The inverse of request_from_openai(), and the basis of the round-trip test that enforces invariant 1: anything the OpenAI surface can express must survive the trip. anyinfer.serve.completion_from_generation completion_from_generation( result: Generation, *, model: str, completion_id: str = "chatcmpl-anyinfer", created: int | None = None, include_manifest: bool = False, ) -> dict[str, Any] Render a Generation as a chat.completion object. Parameters: Name Type Description Default result Generation The generation to render. required model str The model string to echo back. required completion_id str The completion id to stamp. 'chatcmpl-anyinfer' created int | None Unix timestamp; defaults to now. None include_manifest bool Attach the run manifest under MANIFEST_FIELD. Off by default, so a stock client's response is byte-identical to what it was before manifests existed. Serialization only — nothing here assembles a manifest. False anyinfer.serve.chunk_from_event chunk_from_event( event: StreamEvent, *, model: str, completion_id: str = "chatcmpl-anyinfer", created: int | None = None, ) -> dict[str, Any] | None Render one stream event as a chat.completion.chunk. Returns None for events with no OpenAI equivalent (timing marks, attempt records) — they are AnyInfer-native observability that the OpenAI wire format cannot carry. anyinfer.serve.final_chunk final_chunk( result: Generation, *, model: str, completion_id: str = "chatcmpl-anyinfer", created: int | None = None, include_usage: bool = True, ) -> Iterable[dict[str, Any]] Render the terminal chunks: the finish reason, then optionally usage. Usage rides in its own trailing chunk with an empty choices array, matching stream_options.include_usage. Clients that stop reading at finish_reason miss it, which is exactly the bug the core's own parser is written to avoid. anyinfer.serve.encode_messages encode_messages( messages: Sequence[Message], ) -> list[dict[str, Any]] Encode typed messages back into an OpenAI messages array. anyinfer.serve.decode_messages decode_messages( raw: Sequence[Mapping[str, Any]], ) -> tuple[Message, ...] Decode an OpenAI messages array into typed messages. --- # Reference / Testing Utilities Source: https://anyinfer.dev/reference/api/testing/ Testing Utilities The anyinfer.testing package is public on purpose, for two audiences. An application tests its own routing, repair, and reduction logic against a scripted provider (guide: test your application offline). A third-party adapter certifies itself by running the same conformance suite the built-in adapters run (guide: the conformance suite). Scripted Providers A provider whose behavior is declared per model, including the failures that are otherwise unreachable without a real outage. anyinfer.testing.assert_manifest_matches assert_manifest_matches( manifest: RunManifest | Mapping[str, Any], path: Path | str, *, update: bool = False, ) -> None Assert a manifest matches its golden file, normalizing volatility away first. A missing golden is written rather than failed: the first run of a new test records what the behaviour is, and the diff in review is where it gets agreed to. A golden that exists and disagrees is a failure, reported as a field-by-field difference rather than two walls of JSON. Parameters: Name Type Description Default manifest RunManifest | Mapping[str, Any] The manifest the run produced. required path Path | str Where the golden lives. Parent directories are created as needed. required update bool Rewrite the golden instead of comparing. Wired to the pytest plugin's --update-manifests flag. False Raises: Type Description AssertionError If the golden exists and the run no longer matches it. anyinfer.testing.ScriptedProvider ScriptedProvider( provider_id: str = "scripted", models: Sequence[ScriptedModel] | None = None, *, aliases: Sequence[str] = (), locality: Literal[ "hosted", "local", "remote" ] = "local", base_url: str = "http://scripted.invalid/v1", display_name: str | None = None, ) A registered provider whose behaviour is declared, not coded. Parameters: Name Type Description Default provider_id str Id to register under, and the left half of every target it serves. 'scripted' models Sequence[ScriptedModel] | None The models this provider serves. At least one is required. None aliases Sequence[str] Additional names resolving to this provider. () locality Literal['hosted', 'local', 'remote'] "local" (the default) keeps cost at a genuine zero and hardware detection honest; pass "hosted" when a test needs pricing to apply. 'local' base_url str Endpoint recorded in settings. Never contacted — the transport intercepts every request, so it uses an unroutable .invalid host. 'http://scripted.invalid/v1' display_name str | None Human-readable name for UIs and error messages. Defaults to naming the provider as scripted, which is the right answer in a test and the wrong one in an application that ships a scripted provider as a visible offline mode. None Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, oldest first, across all models. models property models: tuple[ScriptedModel, ...] The models this provider serves, in declaration order. target target(model_id: str | None = None) -> str The target string for one of this provider's models. Parameters: Name Type Description Default model_id str | None Which model; defaults to the first declared. None Returns: Type Description str A provider:model target. descriptor descriptor() -> ProviderDescriptor The declarative descriptor this provider registers. register register( registry: ProviderRegistry | None = None, ) -> ProviderRegistry Register this provider, replacing any earlier registration of the same id. Parameters: Name Type Description Default registry ProviderRegistry | None Where to register. Defaults to the process-wide registry, which is convenient in a one-off script and wrong in a test suite — the pytest fixtures hand each test its own registry so parallel tests cannot collide. None Returns: Type Description ProviderRegistry The registry that was written to. settings settings(**overrides: Any) -> ProviderSettings Provider settings wired to this provider's in-process transport. transport transport() -> httpx2.MockTransport An httpx2 transport serving this provider's scripted behaviour. reset reset() -> None Rewind every model's failure script and forget recorded requests. call_count call_count(model_id: str | None = None) -> int How many generation calls were served, in total or for one model. with_model with_model(model: ScriptedModel) -> ScriptedProvider Return this provider with one model's declaration replaced or added. Convenience for a test that needs a variant of an otherwise shared provider without rebuilding the whole table. anyinfer.testing.ScriptedModel dataclass ScriptedModel( id: str, text: str = "Scripted answer.", structured: Mapping[str, Any] | None = None, tool_calls: tuple[tuple[str, str, str], ...] = (), finish_reason: str = "stop", usage: Mapping[str, Any] | None = ( lambda: { "prompt_tokens": 11, "completion_tokens": 7, "total_tokens": 18, } )(), chunk_size: int = 4, answer_after_tools: str | None = None, failures: tuple[ScriptedFailure, ...] = (), capabilities: ModelCapabilities = DEFAULT_SCRIPTED_CAPABILITIES, ) One model a scripted provider serves, and how it behaves. Attributes: Name Type Description id str Model id, as it appears after the colon in a target. text str Assistant text to answer with. Ignored when structured is set. structured Mapping[str, Any] | None An object to answer with, serialized as JSON. The convenient way to script a request that carries a schema. tool_calls tuple[tuple[str, str, str], ...] Tool calls to emit, as (id, name, arguments_json) triples. finish_reason str Normalized finish reason to report. usage Mapping[str, Any] | None Usage block to report, or None to report none at all, which is how a test reaches the estimated-usage path. chunk_size int Characters per streamed delta. Smaller values produce more events. answer_after_tools str | None Text to answer with once the conversation already carries a tool result. Without it, a model scripted to call a tool calls it on every round and the loop never converges, which is a test that only ever proves the round budget works. Set it and the model behaves like a real one: ask, then answer. failures tuple[ScriptedFailure, ...] Failures consumed in order before any success. A model with two failures and a retry budget of one will exhaust the budget; that is the point. capabilities ModelCapabilities What this model claims to support. Defaults to DEFAULT_SCRIPTED_CAPABILITIES; declare narrower capabilities to force a weaker structured-output mechanism. answer_text property answer_text: str The body this model answers with, structured content winning over prose. anyinfer.testing.ScriptedFailure dataclass ScriptedFailure( kind: FailureKind = "status", status: int = 503, retry_after_s: float | None = None, message: str = "scripted failure", ) One scripted failure, consumed by the next call to its model. Attributes: Name Type Description kind FailureKind Which failure to produce; see FailureKind. status int HTTP status for status failures. retry_after_s float | None Retry-After seconds to advertise, when the failure carries one. Use 0.0 in tests that assert retry behaviour without waiting for it. message str Error text the provider reports. Bounded and inert — it is test data, not a template. anyinfer.testing.FailureKind module-attribute FailureKind = Literal[ "status", "truncate", "malformed-json", "timeout", "refusal", ] How a scripted call fails. Each kind reaches a different part of the core, and each is otherwise reachable only by provoking a real provider into misbehaving: status An HTTP error, optionally carrying Retry-After. Exercises status classification, backoff, and the retry event. truncate A stream cut mid-event. Exercises stream teardown and partial-result handling. malformed-json A body that will not validate against the requested schema. Exercises validation and the bounded repair loop. timeout A read timeout, raised rather than slept, so the case is deterministic. refusal A completed response reporting content_filter. Exercises the content-policy chain. anyinfer.testing.DEFAULT_SCRIPTED_CAPABILITIES module-attribute DEFAULT_SCRIPTED_CAPABILITIES = ModelCapabilities( context_window=Sourced(32768, "default"), features=Sourced( Feature.STREAMING | Feature.TOOLS | Feature.SYSTEM_PROMPT | Feature.JSON_MODE, "default", ), ) Capabilities a scripted model claims when it declares none of its own. Deliberately default provenance: a scripted provider is not a source of truth about what any real model supports, and a test that needs a trusted window should say so by declaring one. anyinfer.testing.VOLATILE_FIELDS module-attribute VOLATILE_FIELDS: Mapping[str, tuple[str, ...]] = { "": ("request_id", "anyinfer_version"), "attempts": ( "first_token_ms", "total_ms", "queued_ms", "retry_delay_s", "paced_s", ), "timing": ( "first_token_ms", "total_ms", "output_tokens_per_s", "phases", ), } Which fields normalize() drops, keyed by the facet they live on. Everything here is a wall-clock or a per-process identifier: true of the run, and false of the next identical one. Nothing about a decision is in this list, because a decision that changed is precisely what a golden manifest exists to catch. anyinfer.testing.normalize normalize( manifest: RunManifest | Mapping[str, Any], ) -> dict[str, Any] Strip the fields that differ between two identical runs. Parameters: Name Type Description Default manifest RunManifest | Mapping[str, Any] A RunManifest or its RunManifest.to_dict form. required Returns: Type Description dict[str, Any] JSON-safe data with every volatile field removed, ready to compare or write. Fake Embedding and Rerank Providers An in-process fake implementing EmbedsText/ReranksText directly; there is no wire dialect to reproduce for these operations, so this fake needs no mock transport. anyinfer.testing.FakeEmbeddingRerankProvider FakeEmbeddingRerankProvider( provider_id: str = "fake-embed", *, embedding_dimensions: Mapping[str, int] | None = None, rerank_models: Sequence[str] = (), embedding_failures: Mapping[ str, Sequence[ScriptedEmbeddingFailure] ] | None = None, normalized: bool = True, locality: Literal[ "hosted", "local", "remote" ] = "local", embedding_capabilities: Mapping[ str, EmbeddingCapabilities ] | None = None, declared_embedding: Mapping[str, EmbeddingCapabilities] | None = None, rerank_capabilities: Mapping[str, RerankCapabilities] | None = None, pricing: Mapping[str, Pricing] | None = None, ) An in-process fake supporting EmbedsText and ReranksText, or both. Parameters: Name Type Description Default provider_id str Id to register under, and the left half of every target it serves. 'fake-embed' embedding_dimensions Mapping[str, int] | None Vector length embed() produces, keyed by model id. A model id absent here does not embed. None rerank_models Sequence[str] Model ids that support reranking. A model id absent here does not rerank. () embedding_failures Mapping[str, Sequence[ScriptedEmbeddingFailure]] | None Failures consumed in order before any success, keyed by model id. None normalized bool Whether vectors this provider produces are reported as unit-normalized. True locality Literal['hosted', 'local', 'remote'] Recorded on the descriptor. 'local' embedding_capabilities Mapping[str, EmbeddingCapabilities] | None Static EmbeddingCapabilities recorded on the descriptor, keyed by model id — the way tests declare a verified batch limit (max_batch_inputs) for core-owned batching to resolve. None declared_embedding Mapping[str, EmbeddingCapabilities] | None EmbeddingCapabilities reported through list_models(), keyed by model id — the other way a provider states model-level facts, used by listings that tag their own models and by local engines reading a pinned catalog row. Distinct from embedding_capabilities on purpose: a descriptor table is keyed by ids known when the code was written, and a provider whose ids come from the machine it runs on cannot use one. None rerank_capabilities Mapping[str, RerankCapabilities] | None Static RerankCapabilities recorded on the descriptor, keyed by model id (max_documents drives rerank batching). None pricing Mapping[str, Pricing] | None Trusted per-model pricing recorded on the descriptor (provenance "catalog"), so cost computation and spend ceilings can be tested offline. None Attributes: Name Type Description embed_requests list[EmbeddingWireRequest] Every EmbeddingWireRequest received, oldest first. rerank_requests list[RerankWireRequest] Every RerankWireRequest received, oldest first. operations operations() -> frozenset[InferenceOperation] Which operations this provider declares, for building its descriptor. descriptor descriptor() -> ProviderDescriptor The declarative descriptor this provider registers. register register(registry: ProviderRegistry) -> ProviderRegistry Register this provider's descriptor, replacing any earlier registration. list_models async list_models() -> Sequence[DiscoveredModel] Enumerate the models this fake serves, with any facts they declare. health async health() -> Health Always healthy — this fake never simulates transport-level outages at health time. aclose async aclose() -> None No resources to release. embed async embed(req: EmbeddingWireRequest) -> EmbeddingWireResult Return deterministic pseudo-embeddings, or raise a scripted failure. rerank async rerank(req: RerankWireRequest) -> RerankWireResult Return documents ranked by deterministic lexical overlap with the query. anyinfer.testing.ScriptedEmbeddingFailure dataclass ScriptedEmbeddingFailure( kind: _EmbeddingFailureKind = "status", retry_after_s: float | None = None, message: str = "scripted failure", ) One scripted failure, consumed by the next call to the model it is attached to. Attributes: Name Type Description kind _EmbeddingFailureKind "status" raises a generic retryable ProviderError; "rate-limit" raises RateLimitError with retry_after_s; "transport" raises TransportError. retry_after_s float | None Advertised retry delay for a rate-limit failure. message str Error text reported. Test data only. Fake MCP Server An in-process Model Context Protocol server, for testing a tool loop fed by MCP tool sources. anyinfer.testing.FakeMCPServer FakeMCPServer( tools: Sequence[FakeMCPTool] | None = None, *, protocol_version: str = PROTOCOL_VERSION, page_size: int = 0, ) A scripted MCP server that answers over an in-process transport. Parameters: Name Type Description Default tools Sequence[FakeMCPTool] | None The tools to advertise. None protocol_version str Version to answer the handshake with. Override it to test a client's version negotiation. PROTOCOL_VERSION page_size int Advertise tools across several tools/list pages, to exercise pagination. 0 means one page. 0 Attributes: Name Type Description calls list[tuple[str, Mapping[str, Any]]] Every tools/call received, as (name, arguments) pairs. transport transport() -> _FakeMCPTransport An anyinfer.mcp.MCPTransport serving this fake. handle handle(message: Mapping[str, Any]) -> dict[str, Any] | None Answer one JSON-RPC message, or None for a notification. anyinfer.testing.FakeMCPTool dataclass FakeMCPTool( name: str, description: str = "A fake tool.", parameters: Mapping[str, Any] = dict(), result: str = "fake tool result", is_error: bool = False, annotations: Mapping[str, Any] = dict(), blocks: tuple[Mapping[str, Any], ...] = (), ) One tool a fake server advertises, and what calling it produces. Attributes: Name Type Description name str The tool's name, un-namespaced. description str What the model is told it does. parameters Mapping[str, Any] JSON Schema for its arguments; an empty object accepts anything. result str Text the tool returns. is_error bool Whether the call reports its own failure, which a loop feeds back to the model rather than raising. annotations Mapping[str, Any] Behavioural hints to advertise, in the protocol's own spelling (readOnlyHint and friends). blocks tuple[Mapping[str, Any], ...] Overrides result with raw content blocks, for testing what happens to non-text content. pytest Fixtures Registered automatically when anyinfer is installed; see the guide for the full table. anyinfer.testing.plugin.EventCollector EventCollector() Collects telemetry events for assertions. A collector, not an assertion library: it hands back what happened and leaves the judging to the test. Registered payload-free unless a test explicitly opts in, so a suite cannot start capturing prompt text by accident. on_event on_event(event: TelemetryEvent) -> None Record one event. of_type of_type(*types: type) -> list[Any] Every recorded event that is an instance of any of types. request_ids request_ids() -> list[str] Correlation ids seen, in first-seen order. clear clear() -> None Forget everything recorded so far. Fake Providers In-process fake servers that speak the real wire dialects, so examples and tests run without credentials or a network. anyinfer.testing.FakeOpenAIServer FakeOpenAIServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ( "fake-model-small", "fake-model-large", ), chunk_size: int = 4, reasoning_field: str | None = None, dimensions: int = 8, ) Bases: _FakeServerBase A configurable in-process OpenAI-compatible endpoint. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted, so a single-element list serves every request. None models Sequence[str] Model ids reported by GET /models. ('fake-model-small', 'fake-model-large') chunk_size int Characters per streamed text delta. 4 reasoning_field str | None Name of the dialect's reasoning key (DeepSeek and xAI both send reasoning_content). None — the plain OpenAI shape — omits reasoning entirely, so an adapter with no reasoning channel cannot accidentally pass the reasoning case. None dimensions int Width of the vectors POST /embeddings returns. 8 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. next_response next_response() -> FakeResponse The response for the next generation call. anyinfer.testing.FakeResponsesServer FakeResponsesServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ("gpt-5", "gpt-5-mini"), chunk_size: int = 4, dimensions: int = 8, ) Bases: _FakeServerBase A configurable in-process OpenAI Responses API endpoint. Not the same dialect as FakeOpenAIServer, which speaks /chat/completions. The Responses protocol is typed events — response.output_text.delta, response.output_item.added, response.completed — rather than choice deltas, and a finish reason is derived from the terminal response object's status and incomplete_details rather than sent as a field. Modelling that difference is the point: an adapter that quietly treated one as the other would pass a chat-shaped fake. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted, so a single-element list serves every request. None models Sequence[str] Model ids reported by GET /models. ('gpt-5', 'gpt-5-mini') chunk_size int Characters per streamed text delta. 4 dimensions int Width of the vectors POST /embeddings returns. 8 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. next_response next_response() -> FakeResponse The response for the next generation call. anyinfer.testing.FakeAnthropicServer FakeAnthropicServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ( "claude-sonnet-4-5", "claude-opus-4-1", ), chunk_size: int = 4, page_size: int = 1, ) Bases: _FakeServerBase A configurable in-process Messages API endpoint. Two properties of the real API shape this fake. It always streams — the adapter sends stream: true unconditionally, so there is no buffered branch to model — and it has no response-format field, so a schema is emulated as a single forced tool call and the structured answer arrives as a tool_use block rather than as text. That second point is why this fake reads the request before answering. A scenario says "return this JSON"; whether that JSON belongs in a text_delta or in the input of a tool_use block is a property of what was asked, not of the scenario. The tool name is echoed from the request's own tool_choice rather than hardcoded, so the fake cannot drift from whatever the core decides to call the schema. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted, so a single-element list serves every request. None models Sequence[str] Model ids reported by GET /v1/models. ('claude-sonnet-4-5', 'claude-opus-4-1') chunk_size int Characters per streamed text delta. 4 page_size int Model ids per listing page. The listing is cursor-paginated, and an adapter that ignores has_more silently reports only the first page -- which looks like a working discovery call. 1 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. next_response next_response() -> FakeResponse The response for the next generation call. anyinfer.testing.FakeBedrockServer FakeBedrockServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ( "anthropic.claude-sonnet-4-5-v1:0", "amazon.titan-embed-text-v2:0", ), chunk_size: int = 4, dimensions: int = 8, ) Bases: _FakeServerBase A configurable in-process Bedrock endpoint spanning its four actions. Bedrock is not one API. Generation is /model/{id}/converse buffered or /model/{id}/converse-stream in AWS's binary event framing; Titan embeddings are /model/{id}/invoke on the same runtime host; rerank is a different service (bedrock-agent-runtime's POST /rerank); and discovery is a different host again (the control plane's /foundation-models). Routing all four through one fake is what lets the shared suite treat Bedrock as one provider the way a caller does. Two Converse details the fake models deliberately. Usage arrives only in the terminal metadata event, so a stream that ends at messageStop reports no tokens — an adapter reading usage from the wrong event silently loses it. And a schema is emulated as a forced tool call, so the structured answer arrives as a toolUse block rather than as text. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted, so a single-element list serves every request. None models Sequence[str] Model ids reported by the control plane's /foundation-models. ('anthropic.claude-sonnet-4-5-v1:0', 'amazon.titan-embed-text-v2:0') chunk_size int Characters per streamed text delta. 4 dimensions int Width of the vectors /invoke returns. 8 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. next_response next_response() -> FakeResponse The response for the next call. anyinfer.testing.FakeOllamaServer FakeOllamaServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ("qwen3:8b", "qwen2.5:3b"), loaded: Mapping[str, int] | None = None, chunk_size: int = 4, embed_scenario: str | None = None, ) Bases: _FakeServerBase A configurable in-process Ollama server speaking the native NDJSON dialect. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted. None models Sequence[str] Models reported by /api/tags. ('qwen3:8b', 'qwen2.5:3b') loaded Mapping[str, int] | None model -> size_vram entries reported by /api/ps, for GPU-spill tests. None chunk_size int Characters per streamed text delta. 4 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. anyinfer.testing.FakeGeminiServer FakeGeminiServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, models: Sequence[str] = ( "gemini-2.5-flash", "gemini-2.5-pro", ), chunk_size: int = 4, ) Bases: _FakeServerBase A configurable in-process Gemini endpoint speaking the native protocol. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Responses to serve, one per request, in order. The last is reused once exhausted. None models Sequence[str] Model ids reported by GET /models. ('gemini-2.5-flash', 'gemini-2.5-pro') chunk_size int Characters per streamed text part. 4 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. anyinfer.testing.FakeRetrievalServer FakeRetrievalServer( responses: Sequence[FakeResponse] | FakeResponse | None = None, *, dimensions: int = 8, rerank_key: str = "data", top_n_key: str = "top_n", models_status: int = 404, ) Bases: _FakeServerBase An in-process endpoint for retrieval-only providers (Voyage, Jina). These providers speak a narrow dialect: POST /embeddings in the OpenAI shape, a POST /rerank that differs only in which key holds the ranking, and no listing route at all. Sharing one fake keeps their conformance rows honest about the same scenarios every other adapter answers. Parameters: Name Type Description Default responses Sequence[FakeResponse] | FakeResponse | None Scenario script, as for the other fakes. Only status, headers, and error_message apply — there is no text to generate. None dimensions int Width of the vectors POST /embeddings returns. 8 rerank_key str Key holding the ranking. Voyage uses data; Jina uses results. 'data' top_n_key str Request key carrying the truncation count. Voyage spells it top_k; Jina sends a plain top_n. A fake that accepted either would let an adapter send the wrong one and still pass. 'top_n' models_status int Status for GET /models. These providers document no listing route, so the honest default is a 404 that still proves reachability. 404 Attributes: Name Type Description requests list[dict[str, Any]] Every request body received, for assertions. next_response next_response() -> FakeResponse The response for the next retrieval call. anyinfer.testing.FakeResponse dataclass FakeResponse( text: str = "Hello from the fake provider.", reasoning: str = "", tool_calls: tuple[tuple[str, str, str], ...] = (), finish_reason: str = "stop", usage: Mapping[str, Any] | None = ( lambda: { "prompt_tokens": 11, "completion_tokens": 7, "total_tokens": 18, } )(), status: int = 200, error_message: str = "fake provider error", headers: Mapping[str, str] = dict(), malformed_sse: bool = False, ignore_stream: bool = False, omit_usage_chunk: bool = False, ) A scripted response the fake server should produce. Attributes: Name Type Description text str Assistant text to emit, chunked across deltas when streaming. reasoning str Thinking text to emit before the answer. Only dialects with a reasoning channel surface it (Ollama's thinking field, Gemini's thought-flagged parts). tool_calls tuple[tuple[str, str, str], ...] Tool calls to emit, as (id, name, arguments_json) triples. finish_reason str Finish reason to report. usage Mapping[str, Any] | None Usage block to report, or None to omit it entirely. status int HTTP status; >= 400 produces an error body instead of a completion. error_message str Message for error responses. headers Mapping[str, str] Extra response headers (e.g. retry-after). malformed_sse bool Emit an unparseable SSE data field, to exercise error handling. ignore_stream bool Answer a streaming request with a buffered JSON body. omit_usage_chunk bool Stream without a terminal usage chunk. anyinfer.testing.scenario_responses scenario_responses( scenario: str, *, text: str = "Hello from the fake provider.", reasoning: str = "Let me think.", probe_answer: str = _PROBE_ANSWER, ) -> list[FakeResponse] The canonical response programme for one conformance scenario. Every harness answers the same nine scenarios with the same shapes: a tool call for tools, an invalid-then-valid pair for repair, a 401 for auth_error, and so on. Only the wire encoding differs per dialect, and the fake server classes already own that. Programming the scenarios here means a new adapter's harness declares its dialect and its capabilities — never a ninth copy of the same if/elif chain, which is exactly where a subtly weaker probe slips into one provider's column unnoticed. Parameters: Name Type Description Default scenario str A key from CONFORMANCE_SCENARIOS. An unrecognized key is treated as default, so a harness never has to enumerate them. required text str Assistant text for the scenarios that just need a successful answer. 'Hello from the fake provider.' reasoning str Thinking text for the reasoning scenario. Dialects without a reasoning channel ignore it and declare reasoning=False instead. 'Let me think.' probe_answer str The JSON answer satisfying PROBE_SCHEMA. _PROBE_ANSWER Returns: Type Description list[FakeResponse] Responses to serve in order. The fake servers reuse the last one once exhausted, list[FakeResponse] so a single-element list answers every request in the scenario. anyinfer.testing.CONFORMANCE_SCENARIOS module-attribute CONFORMANCE_SCENARIOS: tuple[str, ...] = ( "default", "tools", "reasoning", "structured", "repair", "auth_error", "rate_limited", "oversized", "odd_finish", ) Every scenario key run_conformance hands to a harness's client factory. anyinfer.testing.chunk_text chunk_text(text: str, size: int = 4) -> list[str] Split text into fixed-size fragments, mimicking token-level streaming. anyinfer.testing.sse_lines sse_lines( payloads: Iterable[Any], *, done: bool = True ) -> bytes Encode payloads as an SSE body. anyinfer.testing.ndjson_lines ndjson_lines(payloads: Iterable[Any]) -> bytes Encode payloads as an NDJSON body (Ollama's framing). anyinfer.testing.eventstream_frame eventstream_frame(event_type: str, payload: Any) -> bytes Encode one application/vnd.amazon.eventstream frame. AWS frames its Converse stream in a binary envelope with two CRC32s rather than in SSE, so a fake that returned JSON lines would exercise a decoder the adapter does not have. The checksums are computed, not stubbed: an adapter that skips validating them should not be able to pass by being handed frames that were never valid. Cassettes Record/replay of real provider exchanges. anyinfer.testing.Cassette Cassette(path: Path) A file of recorded interactions. load load() -> None Read interactions from disk. save save() -> None Write interactions to disk, redacting secrets first. append append(interaction: Interaction) -> None Record an interaction. anyinfer.testing.CassetteTransport CassetteTransport( cassette: Cassette, *, record: bool = False, inner: AsyncBaseTransport | None = None, ) Bases: AsyncBaseTransport Replays a cassette, or records live traffic into one. Parameters: Name Type Description Default cassette Cassette The cassette to read or write. required record bool When True, forward requests to inner and record the results. False inner AsyncBaseTransport | None The transport used while recording. Required in record mode. None aclose async aclose() -> None Close the recording-mode inner transport, when one was opened. AsyncBaseTransport.aclose is a no-op by default; recording opens a real httpx2.AsyncHTTPTransport in __init__ and nothing else was closing its connection pool, which leaked a socket per test under this project's filterwarnings = ["error"] gate. Replay mode has no inner to close. handle_async_request async handle_async_request(request: Request) -> httpx2.Response Serve one request from the cassette, or record it live. anyinfer.testing.Interaction dataclass Interaction( method: str, url: str, request_body: str, status: int, headers: dict[str, str], body: str, ) One recorded request/response exchange. Attributes: Name Type Description method str HTTP method of the recorded request; replay matches on it. url str Full request URL; replay matches on its path, and redaction scrubs it before it reaches disk. request_body str The request body as text, redacted at save time. status int HTTP status code of the recorded response. headers dict[str, str] Response headers. Secret-bearing headers are replaced wholesale at save time; the rest pass through redaction. body str The response body as text, redacted at save time and replayed verbatim. to_json to_json() -> dict[str, Any] Serialize for storage. from_json classmethod from_json(data: dict[str, Any]) -> Interaction Deserialize from storage. Recording Turning a live run against the developer's own account into committable cassettes. The audit is a second, independent pass over the saved bytes: redaction removes secrets it was told about, and this looks for credential shapes it was not. anyinfer.testing.audit_cassette audit_cassette( source: Cassette | Path | Iterable[Interaction], ) -> list[AuditFinding] Findings across a whole cassette. A Path is read from disk, which is the form that matters: it audits the bytes that would actually be committed, after Cassette.save() has applied redaction, rather than the in-memory objects redaction was handed. Parameters: Name Type Description Default source Cassette | Path | Iterable[Interaction] A cassette, a path to a saved cassette file, or interactions directly. required Returns: Type Description list[AuditFinding] Every finding, in cassette order. anyinfer.testing.audit_interaction audit_interaction( interaction: Interaction, index: int = 0 ) -> list[AuditFinding] Findings for one recorded interaction. Parameters: Name Type Description Default interaction Interaction The interaction as it would be written to disk. required index int Its position in the cassette, for the report. 0 Returns: Type Description list[AuditFinding] Every credential-shaped survivor, in scan order. An empty list means nothing list[AuditFinding] matched — not that the interaction is provably clean. anyinfer.testing.AuditFinding dataclass AuditFinding( interaction: int, where: str, shape: str, excerpt: str ) One credential-shaped string that survived redaction. Attributes: Name Type Description interaction int Index of the interaction it appeared in. where str Which part carried it — url, request_body, body, or a header name. shape str Name of the pattern that matched, e.g. "bearer-token". excerpt str A short, masked excerpt for the report. Never the full value: a finding printed to a terminal or a CI log must not itself become the leak. __str__ __str__() -> str One line for a terminal report. anyinfer.testing.SECRET_SHAPES module-attribute SECRET_SHAPES: tuple[tuple[str, Pattern[str]], ...] = ( ( "vendor-api-key", re.compile( "\\b(?:sk-ant-|sk-|xai-|gsk_|AIza|co-|r8_)[A-Za-z0-9_\\-]{16,}" ), ), ( "bearer-token", re.compile("\\bBearer\\s+[A-Za-z0-9._\\-]{16,}"), ), ( "jwt", re.compile( "\\beyJ[A-Za-z0-9_\\-]{8,}\\.[A-Za-z0-9_\\-]{8,}\\.[A-Za-z0-9_\\-]{8,}" ), ), ( "aws-access-key-id", re.compile("\\b(?:AKIA|ASIA)[0-9A-Z]{16}\\b"), ), ( "private-key", re.compile( "-----BEGIN (?:[A-Z ]+ )?PRIVATE KEY-----" ), ), ( "credential-field", re.compile( '"(?:[a-z_]*(?:api[_-]?key|secret|token|password|credential)[a-z_]*)"\\s*:\\s*"(?!\\[redacted\\])[^"]{12,}"', re.IGNORECASE, ), ), ) Credential shapes an audit looks for, independently of what redaction knew about. Each is a public convention — a documented key prefix, a standard token encoding — which is precisely why it can be matched without knowing the secret. Anything genuinely opaque and unprefixed cannot be found this way, which is why audit_cassette reports findings for a human rather than claiming a cassette is clean. Conformance The parametrized suite behind the conformance matrix. anyinfer.testing.conformance.run_conformance async run_conformance( harness: ConformanceHarness, *, only: Sequence[str] | None = None, ) -> list[CaseResult] Run the suite against one adapter. Parameters: Name Type Description Default harness ConformanceHarness The adapter under test. required only Sequence[str] | None Restrict the run to these case names. None Returns: Type Description list[CaseResult] One CaseResult per case, in matrix order. anyinfer.testing.conformance.ConformanceHarness dataclass ConformanceHarness( provider_id: str, model: str, build_client: Callable[[str], Awaitable[AsyncClient]], supports: Capabilities = Capabilities(), embedding_model: str | None = None, rerank_model: str | None = None, retry: Retry = Retry( max_attempts=2, backoff_base_s=0.0 ), ) Everything the suite needs to exercise one adapter. Attributes: Name Type Description provider_id str The provider under test. model str Model id to send. build_client Callable[[str], Awaitable[AsyncClient]] Builds a client whose scripted responses match scenario. The suite passes a scenario name so the harness can program its fake or select its cassette. supports Capabilities Declared capabilities; unsupported cases are skipped. retry Retry The policy the rate-limiting cases route through. It defaults to no backoff because those cases assert the attempt trail -- that a 429 is recorded as a retried attempt and typed retryable -- and never assert how long the client waited. Sleeping the real backoff proves nothing and, at three cases per adapter across every preset, was the single largest block of wall time in the suite. A live-mode harness that talks to a real rate-limited endpoint should override this with a policy that honors Retry-After. target property target: str The target string for this harness. embedding_target property embedding_target: str The embedding target, falling back to the generation model. rerank_target property rerank_target: str The rerank target, falling back to the generation model. retry_route retry_route(target: str) -> Route A route to target carrying this harness's retry policy. The rate-limiting cases state their policy here rather than passing a bare target= and relying on the client to supply one. A client whose default route sets a policy would pass it down either way, but a harness is free to build a client without a default route at all, and these three cases are the ones where the difference is measured in seconds of real sleeping. anyinfer.testing.conformance.Capabilities dataclass Capabilities( list_models: bool = True, health: bool = True, non_streaming: bool = True, streaming: bool = True, ttft: bool = True, usage: bool = True, tools: bool = True, reasoning: bool = True, structured_output: bool = True, repair: bool = True, retry_after: bool = True, error_mapping: bool = True, byte_cap: bool = True, cancellation: bool = False, embedding: bool = False, rerank: bool = False, ) What a provider claims to support, so unsupported cases skip honestly. Each flag gates at least one case of the conformance matrix. Setting a flag to False is a documented ➖, not a pass. Attributes: Name Type Description list_models bool Model discovery returns the provider's models. health bool The provider answers a health probe. non_streaming bool Whole-response generation, including finish-reason normalization. streaming bool Incremental generation with the event-ordering guarantees. ttft bool Time to first token is measurable on streams. usage bool Token usage is reported, including usage that trails the finish reason. tools bool Tool calls surface completely, streaming and non-streaming. reasoning bool Reasoning streams on its own channel, excluded from the answer text. structured_output bool Schema-constrained generation yields a validated value. repair bool An invalid structured value can be repaired within the attempt budget. retry_after bool Rate limiting surfaces as a retryable, recorded attempt. error_mapping bool Provider failures map to typed errors with a correct retry flag. byte_cap bool An oversized response is rejected rather than silently truncated. cancellation bool Abandoning a stream mid-flight releases its upstream connection and leaves the client usable. Defaults to False: this is a case a harness opts into, because a fake that cannot be held open mid-stream would fail it for a reason that says nothing about the adapter. embedding bool embed() returns ordered, uniform, finite vectors. Defaults to False — most adapters generate only, and an operation nobody declared must skip rather than fail. rerank bool rerank() returns descending, identity-preserving rankings. Defaults to False for the same reason. anyinfer.testing.conformance.ConformanceCase dataclass ConformanceCase( name: str, scenario: str, requires: str, run: Callable[ [AsyncClient, ConformanceHarness], Awaitable[None] ], ) One named check. Attributes: Name Type Description name str Matrix row name. scenario str Scenario key handed to the harness's client factory. requires str Capability flag gating this case. run Callable[[AsyncClient, ConformanceHarness], Awaitable[None]] The check itself; raises AssertionError on failure. anyinfer.testing.conformance.CaseResult dataclass CaseResult( name: str, passed: bool, skipped: bool = False, detail: str = "", ) The outcome of one conformance case. Attributes: Name Type Description name str The case's row name in the conformance matrix. passed bool Whether the check succeeded. Also False for skipped cases; check skipped first. skipped bool The harness declared the capability unsupported, so the case did not run. detail str Why the case failed (truncated), or why it was skipped; empty on a pass. symbol property symbol: str Matrix symbol: ✅ pass, ➖ declared-unsupported, ❌ failure. anyinfer.testing.conformance.CONFORMANCE_CASES module-attribute CONFORMANCE_CASES: tuple[ConformanceCase, ...] = ( ConformanceCase( "list_models", "default", "list_models", _case_list_models, ), ConformanceCase( "health", "default", "health", _case_health ), ConformanceCase( "non_streaming", "default", "non_streaming", _case_non_streaming, ), ConformanceCase( "streaming", "default", "streaming", _case_streaming ), ConformanceCase( "event_ordering", "default", "streaming", _case_event_ordering, ), ConformanceCase("ttft", "default", "ttft", _case_ttft), ConformanceCase( "usage", "default", "usage", _case_usage ), ConformanceCase( "usage_survives_streaming", "default", "usage", _case_usage_survives_streaming, ), ConformanceCase( "tool_calls", "tools", "tools", _case_tool_calls ), ConformanceCase( "streaming_tool_calls", "tools", "tools", _case_streaming_tool_calls, ), ConformanceCase( "reasoning", "reasoning", "reasoning", _case_reasoning, ), ConformanceCase( "structured_output", "structured", "structured_output", _case_structured_output, ), ConformanceCase( "schema_repair", "repair", "repair", _case_repair ), ConformanceCase( "error_mapping", "auth_error", "error_mapping", _case_error_mapping, ), ConformanceCase( "retry_after", "rate_limited", "retry_after", _case_retry_after, ), ConformanceCase( "byte_cap", "oversized", "byte_cap", _case_byte_cap ), ConformanceCase( "cancellation", "streaming", "cancellation", _case_cancellation, ), ConformanceCase( "unknown_finish_reason", "odd_finish", "non_streaming", _case_unknown_finish_reason, ), ConformanceCase( "embedding", "default", "embedding", _case_embedding ), ConformanceCase( "embedding_duplicates", "default", "embedding", _case_embedding_duplicates, ), ConformanceCase( "rerank", "default", "rerank", _case_rerank ), ConformanceCase( "rerank_top_n", "default", "rerank", _case_rerank_top_n, ), ConformanceCase( "rerank_duplicate_text", "default", "rerank", _case_rerank_duplicate_text, ), ConformanceCase( "embedding_normalization_probe", "default", "embedding", _case_embedding_normalization_probe, ), ConformanceCase( "embedding_byte_cap", "oversized", "embedding", _case_embedding_byte_cap, ), ConformanceCase( "rerank_byte_cap", "oversized", "rerank", _case_rerank_byte_cap, ), ConformanceCase( "embedding_retry_after", "rate_limited", "embedding", _case_embedding_retry_after, ), ConformanceCase( "rerank_retry_after", "rate_limited", "rerank", _case_rerank_retry_after, ), ) Every conformance case, in matrix order. anyinfer.testing.conformance.PROBE_SCHEMA module-attribute PROBE_SCHEMA = { "type": "object", "properties": {"answer": {"type": "string"}}, "required": ["answer"], "additionalProperties": False, } The schema every structured-output probe requests. anyinfer.testing.conformance.PROBE_TOOL module-attribute PROBE_TOOL = ToolSpec( name="lookup", description="Look up a value by key.", parameters={ "type": "object", "properties": {"key": {"type": "string"}}, "required": ["key"], }, ) The tool every tool-calling probe advertises. anyinfer.testing.conformance.matrix_row matrix_row( provider_id: str, results: Sequence[CaseResult] ) -> str Render results as one Markdown conformance-matrix row. Provider documentation pages embed this so a page cannot overstate what the suite actually verified. anyinfer.testing.conformance.results_to_json results_to_json( provider_id: str, results: Sequence[CaseResult] ) -> str Serialize results for the docs build. --- # Reference / Errors Source: https://anyinfer.dev/reference/api/errors/ Errors A shallow hierarchy with structured fields (provider, phase, retryable, http_status, detail, hint); detail is bounded and redacted, hint is the actionable next step. The prose catalog with examples lives in the error reference. anyinfer.AnyInferError AnyInferError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: Exception Base class for everything this library raises. Attributes: Name Type Description detail Human-readable description, redacted and truncated to 512 characters. provider The provider id involved, when one is. phase Phase Lifecycle stage that failed. retryable Whether retrying the identical request could plausibly succeed. retry_after_s Server-advised delay before retrying, when supplied. http_status Status code, for errors that came from an HTTP response. hint The actionable next step to show a user, when one exists. __str__ __str__() -> str Render the detail, with the hint appended when present. snapshot snapshot() -> ErrorInfo Capture this error as a serializable ErrorInfo. Used to build attempt records, which travel in results and events long after the exception itself has been handled. anyinfer.Phase module-attribute Phase = Literal[ "configure", "discover", "generate", "stream", "validate", "cleanup", ] Which stage of the request lifecycle produced an error. anyinfer.ConfigError ConfigError( detail: str, *, provider: str | None = None, phase: Phase = "configure", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: _ConfigurePhaseError Invalid configuration, target string, catalog entry, or missing optional extra. anyinfer.CredentialError CredentialError( detail: str, *, provider: str | None = None, phase: Phase = "configure", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: _ConfigurePhaseError A credential reference could not be resolved. anyinfer.AuthError AuthError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: ProviderError Authentication or authorization failed. anyinfer.ProviderError ProviderError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: AnyInferError Base for anything a provider surfaced. Adapters raise only these; they never retry internally. The router decides what to do based on retryable and retry_after_s. anyinfer.ProviderUnavailableError ProviderUnavailableError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = True, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: _RetryableProviderError The provider is down, unreachable, or failed its health probe. anyinfer.RateLimitError RateLimitError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = True, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: _RetryableProviderError The provider rate-limited the request. anyinfer.ModelNotFoundError ModelNotFoundError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: ProviderError The requested model does not exist or is not available to this account. anyinfer.ContextLengthError ContextLengthError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: ProviderError The prompt exceeds the resolved model's context window. anyinfer.TransportError TransportError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = True, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: _RetryableProviderError Connect, timeout, or TLS failure — no usable response was received. anyinfer.StreamProtocolError StreamProtocolError( detail: str, *, provider: str | None = None, phase: Phase = "stream", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: ProviderError Malformed SSE/NDJSON framing, or a response exceeding its byte cap. anyinfer.SchemaViolationError SchemaViolationError( detail: str, *, raw_text: str = "", errors: tuple[str, ...] = (), partial: Mapping[str, Any] | None = None, missing_required: tuple[str, ...] = (), provider: str | None = None, phase: Phase = "validate", hint: str | None = None, ) Bases: AnyInferError The response did not satisfy the requested schema, and the repair budget is spent. Attributes: Name Type Description raw_text The model's raw output, so callers can inspect or salvage it. errors Human-readable validation error messages. partial Complete top-level members recovered without inference, or None. missing_required Required field names that were not completely received. anyinfer.UnsupportedInputError UnsupportedInputError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: ProviderError A trusted model capability proves it cannot accept an attached input modality. anyinfer.SpendLimitError SpendLimitError( message: str, *, limit_usd: Decimal | None = None, spent_usd: Decimal | None = None, estimated_usd: Decimal | None = None, hint: str | None = None, ) Bases: AnyInferError A request was refused because it would cross a caller-set spending ceiling. Raised before dispatch, so nothing was sent and nothing was billed. Deterministic by construction: the identical request refused once will be refused again, which is why the default retry predicate declines it alongside auth and context-length failures. A ceiling is the caller's own policy on their own client; not an organization quota, which this library deliberately does not implement. Attributes: Name Type Description limit_usd The ceiling that was crossed. spent_usd What this client had already spent when the request arrived. estimated_usd The high end of the refused request's estimated cost. anyinfer.ToolLoopError ToolLoopError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: AnyInferError A tool could not be dispatched, or the loop exceeded its round bound. anyinfer.AllTargetsFailedError AllTargetsFailedError( detail: str = "all routing targets failed", *, attempts: tuple[AttemptRecord, ...] = (), batch_failures: tuple[BatchFailure, ...] = (), hint: str | None = None, ) Bases: AnyInferError Every target in the route failed. Attributes: Name Type Description attempts The complete routing trail, in order, including skipped targets. batch_failures Per-internal-batch outcomes when the failed request had been split by core-owned batching — including the batches that succeeded, so a caller can see exactly what was spent before the failure. Empty for an unsplit request. anyinfer.LocalRuntimeError LocalRuntimeError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: AnyInferError llama-server lifecycle failure, or runtime/model integrity problem. anyinfer.ConfidentialExecutionError ConfidentialExecutionError( detail: str, *, provider: str | None = None, phase: Phase = "generate", retryable: bool = False, retry_after_s: float | None = None, http_status: int | None = None, hint: str | None = None, ) Bases: LocalRuntimeError ConfidentialExecutionAdapter refused to execute: the attestable guarantee a caller requested is not available on this host right now. Fails closed by design — this is never raised as a courtesy warning alongside a completed generation; the generation never happened. See anyinfer.local.attestation.confidential_execution_status for the detection this check is built on. --- # Reference / Error Catalog Source: https://anyinfer.dev/reference/errors/ Error Catalog Every exception AnyInfer raises, when it is raised, and what the user will see. The Shape of Every Error The hierarchy is shallow, about ten classes, with rich structured fields, because callers branch on fields far more often than on exception class: except ai.AnyInferError as error: error.detail # what happened; redacted, ≤512 chars error.hint # the actionable next step, when one exists error.provider # which provider, when applicable error.phase # configure | discover | generate | stream | validate | cleanup error.retryable # would repeating this identical request help? error.retry_after_s # server-advised delay, when supplied error.http_status # status code, for HTTP-sourced failures detail and hint always pass redaction, so no error can leak a credential no matter where it is logged. The Hierarchy AnyInferError ├── ConfigError bad config, target, catalog, or a missing extra ├── CredentialError a credential reference could not be resolved ├── ProviderError base for anything a provider surfaced │ ├── AuthError │ ├── RateLimitError │ ├── ModelNotFoundError │ ├── ContextLengthError │ ├── TransportError │ ├── StreamProtocolError │ ├── ProviderUnavailableError │ └── UnsupportedInputError ├── SpendLimitError a caller-set spending ceiling would be crossed ├── SchemaViolationError validation failed after the repair budget ├── ToolLoopError unknown tool, bad signature, or round bound exceeded ├── AllTargetsFailedError the router exhausted every target └── LocalRuntimeError llama-server lifecycle, or model integrity └── ConfidentialExecutionError the attested guarantee isn't available on this host ProviderError is a distinct branch on purpose: it is exactly what the router catches and may retry. ConfigError, SchemaViolationError, and AllTargetsFailedError are not provider errors and propagate straight to the caller. ConfigError When: an unknown target or provider, a malformed catalog, a missing required setting, or a provider whose optional extra is not installed. Retryable: no. ConfigError: unknown target 'gpt-5' (hint: use 'provider:model' (e.g. 'anthropic:claude-sonnet-4-5'), or one of these aliases: large, medium, small) ConfigError: the copilot provider requires the github-copilot-sdk extra (hint: pip install 'anyinfer[copilot]', then run 'copilot login') Embedding and Rerank Refusals embed() and rerank() raise no exception types of their own: an unsupported operation, a refused embedding fallback, or an oversized batch is a ConfigError whose message and hint name the rule that refused it. The rule behind fallback refusals is the embedding-space safety rule. CredentialError When: an env:// variable is unset, a keyring entry is missing, or the OS vault is locked or unavailable. Retryable: no. CredentialError: environment variable OPENAI_API_KEY is not set (hint: export OPENAI_API_KEY= and retry) How references resolve, and what redaction guarantees, is covered in credentials and redaction. AuthError When: 401 or 403 from a provider: a key that is invalid, expired, or without access to the requested model. Retryable: no; the same key will fail the same way, and retrying only spends budget a transient failure might have needed. AuthError: invalid api key (hint: check the configured API key or credential reference) RateLimitError When: 429. Retryable: yes, honoring Retry-After when longer than the computed backoff. RateLimitError: provider returned HTTP 429 What to check next If this recurs often, lower request concurrency or add a slower fallback target with Route.targets. ModelNotFoundError When: 404, or a provider reporting the model does not exist. Retryable: no. ModelNotFoundError: model "qwen3:70b" not found (hint: pull it first: ollama pull qwen3:70b) ContextLengthError When: the prompt exceeds the model's context window. Retryable: no; the same prompt is the same size. Use Route.context_window_targets to fall back to a larger model instead, or trim the prompt with token estimation and context budgets. TransportError When: a timeout, connection failure, or TLS error. No usable response arrived. Retryable: yes. Since nothing was delivered, a retry cannot duplicate output the consumer already saw (the boundary that makes StreamProtocolError different). See the event stream. TransportError: request to ollama timed out (hint: raise timeout_s, or choose a faster model) StreamProtocolError When: malformed SSE/NDJSON framing, or a response exceeding max_response_bytes. Retryable: no by default. If content had already been emitted, this is raised rather than retried: the consumer has seen text, and silently restarting would duplicate or contradict it. The framing and ordering guarantees are in the event stream. ProviderUnavailableError When: 5xx, or a failed health probe. Retryable: yes. Also marks the target unhealthy, so the health gate skips it briefly. ProviderUnavailableError: cannot connect to ollama: [Errno 111] Connection refused (hint: check the base URL and that the server is running) UnsupportedInputError When: a trusted model capability proves the target cannot accept an attached input modality (image, document, or audio). Raised before dispatch. Retryable: no; the same attachment against the same target fails the same way. What each provider accepts is covered in multimodal inputs. UnsupportedInputError: ollama cannot project audio input (model reports no audio support) (hint: choose a target that supports this input form or supply supported inline bytes) SpendLimitError When: a request would cross a caller-set max_request_usd or max_total_usd spending ceiling, or its cost cannot be estimated and the policy says not to spend blind. Raised before dispatch, so nothing was sent and nothing was billed. Retryable: no; deterministic by construction: the identical request refused once will be refused again. SpendLimitError: a request to anthropic:claude-sonnet-4-5 could cost 0.42, above the per-request ceiling of 0.25 (hint: shorten the prompt, cap max_output_tokens, or raise max_request_usd) How to fix Read error.hint, and inspect error.limit_usd, error.spent_usd, and error.estimated_usd for the exact numbers behind the refusal. See cost and spending. SchemaViolationError When: the response failed validation and the repair budget is spent. Retryable: no, and not a routing failure: the request reached the model and the model answered; it answered the wrong shape. except ai.SchemaViolationError as error: error.raw_text # what the model actually said error.errors # ("age: 'age' is a required property",) How to fix Increase Repair.max_attempts, simplify the schema, inspect error.partial and error.missing_fields, or debug the bounded error.raw_text. Partial members are not schema-validated and no truncated value is guessed. See structured output. ToolLoopError When: the model called an unregistered tool, a tool has an unsupported parameter type (raised at declaration time), or max_rounds was exhausted. Retryable: no. ToolLoopError: tool 'search' parameter 'options' has unsupported type 'MyClass' (hint: v1 tools support str, int, float, bool, list, and dict parameters) AllTargetsFailedError When: every target in the route failed or was skipped. except ai.AllTargetsFailedError as error: for attempt in error.attempts: print(attempt.target, attempt.outcome, attempt.error and attempt.error.detail) The attempts trail is the complete routing history: every target tried, in what order, and why each failed, including retries and health-gated skips. LocalRuntimeError When: llama-server failed to start, crashed, timed out becoming ready, could not be reaped, or a model artifact failed hash verification. LocalRuntimeError: llama-server exited with code 3 while loading qwen2.5-7b: fatal: unable to load model (hint: the model may be incompatible with this runtime build, or the machine may have run out of memory) The server's own log tail is included, because polling a health endpoint alone reveals nothing about why it failed. How to fix Read the included log tail first; it usually names the real cause (OOM, an incompatible GGUF, a port conflict). See the local subsystem. ConfidentialExecutionError When: ConfidentialExecutionAdapter.generate() was called and anyinfer.local.confidential_execution_status() reported end_to_end=False for this host. The inner local adapter is never called; this fails closed, not degraded. ConfidentialExecutionError: confidential execution was requested but is not available: no attestable CPU TEE detected (SEV-SNP/TDX guest device not present) How to fix Call confidential_execution_status() before committing to a request, so the application can degrade with a message the caller sees instead of hitting this error mid-call. See the Confidentiality Tiers guide. Things That Look Like Bugs but Are Not Four behaviors are reported as bugs often enough to state as intended: A cost of None when pricing is unknown. Coercing it to zero would turn a reporting gap into a silent accounting error; see cost and spending. SchemaViolationError does not trigger fallback. The request reached the model and the model answered; a different provider does not fix a shape problem. A mid-stream protocol error after content was emitted raises rather than retries. The consumer has already seen text. An unrecognized finish reason does not crash. FinishReason is an open enum, and unknown values normalize to "other". Handling Errors What the router retries, in what order, and when it falls back to the next target is covered in routing and rate limits. Application-side handling patterns are in the integration instructions. --- # Reference / Conformance Matrix Source: https://anyinfer.dev/reference/conformance-matrix/ Conformance Matrix Generated from a real conformance run — do not edit by hand. Regenerate with python workspace.py matrix. Legend: ✅ verified · ➖ declared unsupported · ❌ failing Each cell is one parametrized test case executed against that adapter in fake-server mode, at whatever boundary that adapter has: an in-process HTTP transport for the twenty that speak HTTP, and a fake SDK module for copilot, whose boundary is the github-copilot-sdk session API rather than a wire protocol. llama-cpp speaks HTTP to a server it supervises, so its row substitutes a stub supervisor and starts no process, downloads nothing, and binds no port. A ➖ is a declared limitation, not a pass. How a case is defined and how to record a cassette are covered in the conformance suite. Last generated: 2026-08-24. Provider list_models health non_streaming streaming event_ordering ttft usage usage_survives_streaming tool_calls streaming_tool_calls reasoning structured_output schema_repair error_mapping retry_after byte_cap cancellation unknown_finish_reason embedding embedding_duplicates rerank rerank_top_n rerank_duplicate_text embedding_normalization_probe embedding_byte_cap rerank_byte_cap embedding_retry_after rerank_retry_after anthropic ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ azure-foundry ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ➖ ✅ ➖ bedrock ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ cohere ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ copilot ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ✅ ✅ ➖ ➖ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ deepseek ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ gemini ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ groq ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ jina ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ llama-cpp ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ lm-studio ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ m365-copilot ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ moonshot ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ nebius ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ollama ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ➖ ✅ ➖ openai ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ➖ ✅ ➖ openai-compat ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ openrouter ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ reka ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ tei ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ venice ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ vertex ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ✅ ✅ ➖ ✅ ➖ voyage ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ xai ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ ➖ Every dedicated adapter now has a shared-harness row. The groq, moonshot, reka and venice rows exercise the shared adapter's quirk axes: bearer auth, the renamed output-token field, x-api-key auth, and the max_completion_tokens dialect. Every entry in the preset registry is separately instantiated and checked for registry invariants; these rows do not claim a live upstream verification. What the Cases Check Case Verifies list_models Discovery returns models with non-empty ids. health The readiness probe answers with a boolean. non_streaming A buffered generation produces text and a valid finish reason. streaming Deltas arrive and concatenate to the final text (ordering guarantee 4). event_ordering All four ordering guarantees hold. ttft First-token timing is measured and consistent with total duration. usage Token counts are reported and internally consistent. usage_survives_streaming A trailing usage chunk reaches the result and the event stream. tool_calls Tool calls carry an id, a name, and parsed arguments. streaming_tool_calls Argument fragments reassemble by index. reasoning Reasoning streams as its own channel, excluded from the answer text. structured_output A schema request yields a validated value and records its mechanism. schema_repair The repair loop recovers an initially-invalid response. error_mapping Failures are typed, carry an attempt trail, and mark retryability. retry_after A rate-limited attempt is retried and recorded. byte_cap An oversized response is rejected rather than silently truncated. cancellation Abandoning a stream releases its connection and leaves the client usable. unknown_finish_reason An unrecognized finish reason normalizes instead of crashing. embedding One vector per input, uniform non-zero dimensions, a space identity. embedding_duplicates Duplicate inputs come back positionally, never deduplicated. rerank Rankings descend, and caller document identity survives the round trip. rerank_top_n top_n truncates the ranking to the requested size. rerank_duplicate_text Identical document text keeps its distinct caller-owned ids. embedding_normalization_probe A probe measures normalization instead of assuming it. embedding_byte_cap An oversized embedding response is refused, not parsed. rerank_byte_cap An oversized rerank response is refused, not parsed. embedding_retry_after A rate-limited embedding call is retried and recorded. rerank_retry_after A rate-limited rerank call is retried and recorded. Modes fake-server: in-process transports asserting the library handles each protocol shape. Runs on every commit. cassette: recorded real traffic, asserting the library handles what providers actually send. live: opt-in, requires credentials. m365-copilot is exempt: its authentication is interactive-only and cannot run headless. See Also Provider pages for the human-readable version. Contract snapshots for the wire details each adapter depends on. --- # Reference / Glossary Source: https://anyinfer.dev/reference/glossary/ Glossary Terms this project uses precisely. Where a word has a loose industry meaning and a specific meaning here, the specific one is what the code implements. Adapter The per-provider module that translates a WireRequest into a provider's wire format and its responses back into events. Adapters only translate; the boundary is explained in writing a provider adapter. Alias A tier name (small, medium, large) that resolves to a concrete model per provider through the catalog; see targets and aliases. Attempt One try against one resolved target. A request may involve several, across retries and fallback; the full attempt trail is on every result. Capability Something a model can do, paired with the provenance of how that is known; see capabilities and provenance. Cassette Recorded HTTP traffic replayed in tests, so conformance can run without credentials; recording one takes a single command. Catalog The data file mapping aliases to per-provider targets, and artifact ids to pinned, hash-verified downloads; see the model catalog. Conformance Suite The shared test suite every adapter must pass before its matrix row is published; see the conformance suite. Descriptor Frozen, declarative data about a provider: how to build its adapter, what configuration it needs, how it spells reasoning effort, what it supports. See start from a descriptor. Drift Check The semi-automated audit comparing contract snapshots against providers' current public documentation, following contracts/DRIFT-CHECK.md. Event Stream The generation primitive. A generation is an ordered stream of typed events, described in the event stream; the non-streaming API drains it. Fallback Moving to the next target in a route after the current one fails. Health Gate Skipping a target that recently failed, for a short TTL, so one dead endpoint does not cost every request its full timeout; see health gating. Mechanism How structured output was requested: grammar, json_schema, json_mode, or prompt. Recorded on every result. Posture How much of a machine local inference may commit: conservative, balanced, or aggressive. Projection Rewriting a schema for a provider's wire format (stripping keywords a grammar compiler cannot handle efficiently). Never changes what validation checks client-side. Provenance Where a capability value came from: default, catalog, discovered, probed, or override, weakest to strongest; an application's override outranks everything the library collected. Repair Re-prompting the same model with validation errors after a schema violation, within a bounded budget; see repair. Route An ordered list of targets plus policy: retries, health gating, and failure-class-specific chains. See routing and rate limits. Sentinel Model A model id meaning "the provider chooses" (Copilot's auto). Capabilities for one are the conjunction across every candidate. Sidecar The OpenAI-compatible HTTP service, anyinfer serve. A wire codec around a normal client, never a second core. Target Where a request goes: an alias, or provider:model. TTFT Time to first token, measured by the core at the first content event, identically for every provider. Wire Request A fully-resolved request handed to an adapter: concrete model, chosen mechanism, projected schema, translated reasoning effort, merged options. See the adapter contract. --- # Contributing / Contributing Source: https://anyinfer.dev/contributing/ Contributing Security vulnerabilities do not belong in public issues. Follow the repository's security policy for private reporting. Setup git clone && cd AnyInfer python -m venv .venv && . .venv/bin/activate # or .venv\Scripts\activate python workspace.py setup # installs the project and dev extras python workspace.py check # every gate CI runs Python 3.11+. Windows, macOS, and Linux are all first-class and all tested in CI. The Task Runner workspace.py is the one entry point for routine commands: python workspace.py in a fresh clone, workspace once installed, or the checked-in ./workspace (sh) and workspace.cmd (Windows) wrappers from the repo root. Run it with no arguments to list every verb. Verb Does python workspace.py setup Install the project and all dev extras, editable python workspace.py check Run the quality gates as ordered phases; --skip=/--only= select phases, --fix lets ruff rewrite python workspace.py demo Launch the pack-in demo app python workspace.py serve Run the OpenAI-compatible sidecar python workspace.py build [wheel\|demo\|serve\|docs\|all] [platform] Build packages, native bundles, and/or the docs site python workspace.py clean Remove build artifacts and caches python workspace.py docs Serve the docs site locally with live reload python workspace.py web Build and serve the exact artifact Pages will publish python workspace.py doctor / providers Hardware report; registered providers Every third-party gate shells out to the same command CI runs and echoes it first, so the runner is a convenience rather than a second source of truth; you can always copy the printed line. First-party maintenance code (the docstring-coverage gate, the doc-link check, the conformance-matrix generator, the demo-bundle build) lives in workspace.py itself rather than a scripts directory, so every dev and devops task is in one file with one --help. The Quality Gates python workspace.py check runs the gates as named, ordered phases (fastest feedback first), and every phase runs in CI and must pass. It keeps going after a failure so one invocation tells you everything that is broken: Phase Runs lint ruff check src tests workspace.py (--fix applies fixes) types mypy (strict) contracts lint-imports: the architecture contracts test pytest -q, the full suite, headless conformance The provider conformance suite and the serve invariants docs-check Docstring coverage, doc links, and the runnable doc examples docs-build mkdocs build --strict: the exact artifact the Pages deploy publishes That table is the whole pipeline: every step of every CI job is one of these phases, invoked as python workspace.py check --only=. A green check therefore means what a green CI run means, with one exception: CI also runs the test phase across Python 3.11–3.14 on Linux, Windows, and macOS, which only the runners can cover. --skip=a,b leaves phases out; --only=a,b runs just those; the two are mutually exclusive. python workspace.py build docs runs the same strict site build as the docs-build phase, for when you want the artifact rather than the verdict. The formatter is not a default gate, since it reflows argv-style flag/value pairs into a less readable shape; python workspace.py check --only=format --fix formats anyway. lint-imports is the unusual one: it enforces the architecture boundaries mechanically rather than relying only on review. The contracts and what each one forbids are listed in architecture; when one fails, the fix is almost always to move code, not to loosen the contract. Where to Read First DESIGN.md: architecture and decision rationale. Start with §3 and §23. Architecture: the condensed version of the rules. Choose the Owning Workstream Core engine, shared configuration, CLI, sidecar, and demo code have separate ownership boundaries. The coding-agent instructions page maps each workstream to its paths and explains the canonical-instructions model the tool-specific files defer to. The same boundaries apply whether the contributor is using an agent or editing by hand. The One Rule Adapters only translate. The core orchestrates. If you are adding control flow to an adapter — a retry, a validation step, a fallback — stop. It belongs in the core, where it is implemented once and behaves identically for every provider. That property is the product. Conventions Frozen dataclasses with slots=True for domain types; typing.Protocol for interfaces. No pydantic dependency. Caller-supplied pydantic models are accepted via duck-typed model_json_schema() only. Errors carry structured fields and an actionable hint. A hint that does not tell the user what to do is not a hint. Anything credential-shaped goes through anyinfer.credentials and is registered for redaction. Local servers bind 127.0.0.1 unless a caller explicitly opts out. New mandatory dependencies need justification. The slim core is a security property as much as an aesthetic one. Comments Comment the why, not the what. The valuable comments in this codebase explain non-obvious constraints: # Closing a buffered pipe while another thread is blocked reading it deadlocks, and on # Windows a grandchild process can keep the write end open after its parent exits, so the # stream is closed here, on the way out, and nowhere else. Pull Requests A change touching an adapter's wire behavior includes: the adapter change; an updated contract snapshot in the same change set; conformance results; its provider page, if behavior changed. Run the drift check before starting adapter work, so you are coding against what the provider does now. Where Next Writing a provider adapter: the adapter contract, the descriptor, and the OpenAI-dialect shortcut. The conformance suite: certifying an adapter and contributing cassettes. Testing guide: the fast track, the gate, and where a test belongs. Branching and releases: the branch model and how a version bump reaches PyPI. Branding and visual assets: the canonical marks and the rules for using them. --- # Contributing / Architecture Source: https://anyinfer.dev/contributing/architecture/ Architecture The condensed version. DESIGN.md §23 has the complete rationale; the glossary defines the vocabulary these rules use. The Shape flowchart TD A[Application] --> B["Client / AsyncClient — orchestration"] B --> C["catalog · schema · routing · events · capabilities"] C -->|WireRequest / AdapterEvent| D["registry — descriptors, entry points"] D --> E["adapters — openai, anthropic, ollama, copilot, ... (translation only)"] E --> F[local subsystem] The Load-Bearing Rules 1. The primitive is GenerationRequest → typed event stream. Never make the OpenAI wire format the internal representation. It is one dialect at the edges. This is what makes the sidecar a thin projection rather than a second core. 2. Adapters only translate. Four methods: list_models, health, generate, aclose. Retry, fallback, validation, repair, timing, usage normalization, cost, telemetry, and redaction live in the core. Thin adapters are coverable by one shared conformance suite; thick ones are not. Writing a provider adapter covers the contract. 3. Async core, sync facade. One implementation. Client wraps AsyncClient with a background event-loop thread. 4. llama.cpp is a supervised subprocess. No llama-cpp-python. 5. Capability data is provenance-tagged. catalog | discovered | probed | default. Never present an estimate as authoritative, and never coerce unknown to zero. 6. Telemetry is typed in-process events. OTel is a lazy optional bridge. Nothing is written anywhere by default; events are payload-free by default. 7. Slim core. Mandatory dependencies are httpx2 and jsonschema. Everything else is an extra. 8. Providers register via frozen descriptors. Declarative setup specs mean no per-engine if/elif in core, config, or UI code. 9. The sidecar is a wire codec. Four invariants, enforced from M0 and round-trip tested: request-surface superset, event-stream sufficiency, target-in-model-string, concurrent streams. See the sidecar. Enforcement Three of these are checked mechanically by lint-imports, not left to review: [[tool.importlinter.contracts]] name = "Adapters never orchestrate" source_modules = ["anyinfer.providers"] forbidden_modules = ["anyinfer.routing", "anyinfer.schema.validate", "anyinfer.schema.repair", "anyinfer._client", "anyinfer.capabilities"] When one fails, move the code. Loosening a contract requires a documented reason; anyinfer.local is absent from the adapter contract on purpose, because composing the local subsystem is translation, not orchestration. Request Lifecycle Resolve the target: alias or provider:model → ResolvedTarget. Gate on health, if the target recently failed. Assemble capabilities from the layered sources. Build the wire request: choose the mechanism, project the schema, translate reasoning effort, inject a schema prompt when needed. Stream adapter events, marking first token and accumulating buffers. Validate structured output against the original schema; repair within budget. Assemble the Generation: timings, usage, cost, warnings, attempt trail. Emit StreamEnded. On failure: record the attempt, emit AttemptFailed, then retry or advance to the next target. Exhaustion raises AllTargetsFailedError with the whole trail. Behaviors that look like bugs but are intended (unknown cost reported as None, no fallback on a schema violation, a mid-stream error raising after content) are listed in the error catalog. --- # Contributing / Coding Agents and Workstreams Source: https://anyinfer.dev/contributing/automation/ Coding-Agent Instructions AnyInfer supports Codex, Claude Code, and GitHub Copilot with one authoritative instruction set. AGENTS.md is the source of truth for repository rules, architecture constraints, tests, and workstream ownership. The tool-specific files are discovery shims only: Tool Discovery file Workflow shims Codex AGENTS.md .agents/skills//SKILL.md Claude Code CLAUDE.md .claude/skills//SKILL.md GitHub Copilot .github/copilot-instructions.md .github/prompts/.prompt.md The shims point back to the canonical source and must not restate repository rules. This prevents one tool from operating under a stale or subtly different architecture. Each workflow follows the same shape (one tool-neutral procedure file, three thin entry points that only make it discoverable): Workflow Canonical procedure What it owns add-provider contracts/NEW-PROVIDER.md Adding a preset, a dedicated adapter, or a new embedding/rerank binding: research first, then adapter, registration, docs, tests, verification check-provider-drift contracts/DRIFT-CHECK.md Auditing existing contract snapshots against current upstream docs, and the report format anyinfer-integration docs/agents/INTEGRATION.md Using AnyInfer from an application, read in somebody else's repository The first two are two halves of one lifecycle: NEW-PROVIDER.md produces a contract snapshot, DRIFT-CHECK.md keeps it true afterwards. Scheduled Repository Checks Three weekly workflows watch for the world changing underneath the repository. The pricing-drift check is a detector, not an updater. It compares the bundled pricing table's exact provider/model keys against Chutes and Avian as direct sources, with OpenRouter as a secondary tripwire for ten mapped entries. A clean run means the sources were reachable and every compared value matched; it does not re-verify prices those sources no longer list, and a contributor changing a rate still confirms it against the provider's own pricing documentation. Run it locally with python scripts/check_pricing_drift.py (--format json for machine output, --report PATH to write the versioned JSON report, --live-source NAME to probe one real source). Exit codes: 0 clean, 1 drift found, 2 a source failed or the report is invalid. Source fixtures for its tests live under tests/fixtures/pricing/, and where the bundled prices come from is covered in cost and spending. The weekly catalog re-check confirms the model catalog's pinned entries still match upstream. The contract drift check audits provider contract snapshots against current public API documentation, following contracts/DRIFT-CHECK.md. Workstream Boundaries Start in the narrowest workstream that owns the behavior: Workstream Primary paths Responsibility Core engine and Python SDK src/anyinfer/ except cli.py and serve/ Requests, typed events, routing, adapters, capabilities, local inference, client lifecycle Shared configuration src/anyinfer/config/ The versioned JSON contract used by every integration path Command-line tool src/anyinfer/cli.py Human and shell interface for run, doctor, providers, and sidecar startup OpenAI-compatible sidecar src/anyinfer/serve/ OpenAI wire codec and ASGI application; never a second routing core Demo application src/demo_app/ Offline reference UI and integration example; not part of core behavior Tests mirror those boundaries under tests/. Shared behavior belongs in the engine or configuration package, not copied into the CLI, sidecar, or demo. A change that crosses a boundary should say why in its pull request and update every affected guide. Keeping the Shims in Sync The test suite checks that each tool-specific shim points to its canonical file and stays small. When a rule changes: Edit AGENTS.md or the canonical workflow document. Change a shim only if its pointer or invocation syntax changed. Run python workspace.py check. Do not add a second copy of the rules for convenience; link to the owning document. --- # Contributing / Branding and Visual Assets Source: https://anyinfer.dev/contributing/branding/ Branding and Visual Assets The canonical logo files and palette live in docs/assets/: anyinfer-icon-512.svg and anyinfer-icon-512.png for square marks and favicons; anyinfer-horizontal-light.svg and anyinfer-horizontal-dark.svg for wordmarks; anyinfer-palette.css for the deep-teal, amber, gold, and slate color tokens; anyinfer-social-card.svg and anyinfer-social-card.png for link previews. Generated, not hand-drawn; see below. Use the supplied assets: do not redraw the mark, substitute a provider logo, recolor a wordmark, or add generated approximations. The icon art is background-independent; only the wordmark text changes between light and dark variants. The demo needs package-local copies under src/demo_app/assets/ because those files ship in the wheel and standalone application. They are content-identical mirrors, not a second source of truth; SVG line endings may follow the checkout platform. tests/test_branding.py fails if they drift from docs/assets/. src/demo_app/theme.py translates the canonical palette into Qt tokens, and the same tests pin its brand constants. How the Published Site Uses the Marks mkdocs.yml points the header logo and favicon at the icon, and overrides/main.html adds the SVG favicon, the Apple touch icon, and the Open Graph and Twitter card tags that decide how an AnyInfer link renders when pasted elsewhere. Every page unfurls with its own title and description over the shared 1200×630 card, a size Slack, Discord, X, LinkedIn, and iMessage all render without re-cropping. The card is not a hand-drawn file. scripts/render_social_card.py composes it from the canonical dark wordmark on the deep-teal surface and rasterizes it with Qt's SVG renderer, writing both anyinfer-social-card.svg and anyinfer-social-card.png. A wordmark edit that skips this step fails tests/test_branding.py. The card is a website asset only; it is not mirrored into the demo package. GitHub's own repository preview image is a repository setting, not a file: upload the same anyinfer-social-card.png under Settings → General → Social preview. When changing the brand kit: Edit the canonical files in docs/assets/. Copy the four runtime assets exactly into src/demo_app/assets/. Update the Qt tokens only if anyinfer-palette.css changed. Run python scripts/render_social_card.py if a wordmark or the surface color changed. Run python workspace.py check and python workspace.py build docs. Keep screenshots and promotional media out of the canonical asset set. They become stale independently of the logo and should be linked or generated as release collateral instead of being treated as product identity. --- # Contributing / Writing an Adapter Source: https://anyinfer.dev/contributing/writing-an-adapter/ Writing a Provider Adapter An adapter translates. That is the whole job, and keeping it that way is what lets one conformance suite cover every provider. This page explains the shape and the reasoning behind it. For the step-by-step procedure (what to research before writing code, which registration gates a new provider trips, and what "done" means), follow contracts/NEW-PROVIDER.md. It is the canonical checklist that this repository's coding-agent skills run, and it starts where every provider should: fetching the current API reference and recording what it says, before any code exists to be biased by. The Contract Four methods: class MyAdapter: provider_id: ClassVar[str] = "my-provider" def __init__(self, config: ProviderConfig) -> None: ... async def list_models(self) -> Sequence[DiscoveredModel]: ... async def health(self) -> Health: ... async def generate(self, req: WireRequest) -> AsyncIterator[AdapterEvent]: ... async def aclose(self) -> None: ... WireRequest arrives fully resolved: concrete model, chosen mechanism, projected schema, translated reasoning effort, merged options. You never see aliases, routing policy, or repair state. You may emit only: TextDelta, ReasoningDelta, ToolCallDelta, UsageUpdate, and exactly one terminal AdapterFinal. What You Must Not Do Retry. The router does that, and doing both multiplies the attempts. Validate schemas or repair responses. The core does. Measure TTFT or duration. The core does, identically for everyone. Consult routing policy or the catalog. lint-imports enforces this. If it fails, move the code. Start from a Descriptor descriptor = ProviderDescriptor( id="my-provider", display_name="My Provider", aliases=("mine",), factory=MyAdapter, locality="hosted", default_base_url="https://api.example.com/v1", requires_base_url=False, setup=ProviderSetupSpec( fields=( SetupField( key="api_key", label="API key", kind="secret", required=True, help_text="Accepts env:// and credential:// references.", ), SetupField( key="base_url", label="Base URL", kind="endpoint", advanced=True, default_value="https://api.example.com/v1", help_text="Defaults to https://api.example.com/v1.", ), ), ), reasoning_translator=lambda effort: {} if effort is None else {"effort": effort}, default_capabilities=ModelCapabilities( features=Sourced(Feature.STREAMING | Feature.TOOLS, "default") ), ) setup is what lets config UIs stay generic; never add a per-engine branch to UI code when you can add a declarative field here. Full signatures for ProviderDescriptor, ProviderSetupSpec, and SetupField are in the registry API, and how advanced and default_value drive a setup form is covered in shared configuration. Two fields worth understanding: grammar_needs_prompt_injection: set it when your engine compiles a schema to a decoding grammar without conditioning the model on it. A grammar guarantees well-formed JSON, not meaningful JSON. ignored_parameters: declare anything your provider accepts and silently discards. The core emits ParameterDropped so users find out. If It Speaks OpenAI Subclass the OpenAI-compatible adapter and override only what differs: class MyAdapter(OpenAICompatAdapter): provider_id: ClassVar[str] = "my-provider" output_tokens_field: ClassVar[str] = "max_completion_tokens" def _build_headers(self, config: ProviderConfig) -> dict[str, str]: headers = super()._build_headers(config) headers["x-my-header"] = "value" return headers azure_foundry.py and openrouter.py are both small for exactly this reason; read them before writing a new dialect from scratch. Errors Raise only ProviderError subclasses, with retryable and retry_after_s set. Use the shared helpers so classification is consistent: from .http import classify_status, map_transport_error, read_error_detail raise classify_status( response.status_code, provider=self.provider_id, detail=read_error_detail(body), headers=response.headers, ) Every error deserves an actionable hint. "Invalid request" is not a hint; "verify the model id, or list available models with client.models()" is. Register It Built-ins go in providers/__init__.py. Third-party adapters advertise themselves: [project.entry-points."anyinfer.providers"] my_provider = "my_package.adapter:descriptor" Discovery is lazy and collision-safe. A plugin that fails to import is skipped rather than breaking every other provider. Certify It Run the conformance suite against your adapter through a ConformanceHarness, declaring anything the provider cannot do in Capabilities so the matrix records a ➖ instead of overstating support. The harness, the run modes, and cassette recording are all covered there. --- # Contributing / The Conformance Suite Source: https://anyinfer.dev/contributing/conformance/ The Conformance Suite One suite, run against every adapter. It is what makes "the behavior does not change when you change providers" a checked claim rather than an aspiration. The suite is public API, under anyinfer.testing, so a third-party adapter certifies itself the same way the built-in ones do. Division of Labor The conformance suite proves the code matches its claims. The drift check proves those claims still match upstream. Both are needed: passing tests against a protocol that changed last month proves nothing. Running It pytest tests/test_conformance.py tests/test_ollama.py -q python workspace.py matrix # regenerate the published matrix Each case is its own parametrized test, so a failure names the broken behavior rather than just "conformance failed". The full case list, what each case verifies, and the three run modes (fake-server, cassette, live) are documented on the conformance matrix. The fakes behind fake-server mode are in-process transports, covered in the testing guide. Declaring What You Cannot Do supports = Capabilities(reasoning=False, tools=False) An unsupported case is reported skipped and renders as ➖, a declared limitation rather than a pass, so the matrix cannot overstate a provider. Contributing a Cassette Cassette coverage is the one thing here that does not scale with maintainer effort: it needs an account on the provider, and no maintainer holds accounts on every supported service. If you already have one, recording is a single command. anyinfer conform groq --model llama-3.3-70b --config anyinfer.json --record tests/cassettes The run makes real calls against your account and writes one cassette per scenario. From then on the suite replays them offline, in CI, with no credentials: you spend a few cents once, and every future run of that adapter's suite is free for everyone. Two complementary passes stand between your traffic and the committed file. Saving a cassette strips the known auth headers wholesale and runs every body through the redaction registry, which removes the secrets AnyInfer was told about: anything resolved through env://, credential://, or anyinfer.credentials. Then audit_cassette re-reads the saved bytes looking for credential shapes it was never told about: vendor-prefixed keys, bearer tokens written into a body, JWTs, AWS key ids, private key material. A finding withholds that cassette rather than warning about it, because a file left on disk after a warning is a file someone commits after skimming past it. The audit is heuristic and says so. It cannot find an opaque, unprefixed secret, so read the cassettes before committing them — they are small, and they are your own traffic. If the audit withholds one, the usual cause is a credential passed as a literal rather than through a reference; route it through anyinfer.credentials so redaction knows about it, and re-record. Recording preserves any transport your config already sets, so a deployment that routes through a proxy records what it sends through that proxy rather than bypassing it. Adding a Case Add it to CONFORMANCE_CASES in anyinfer/testing/conformance.py: ConformanceCase("my_behavior", "default", "streaming", _case_my_behavior) # name scenario capability check The check raises AssertionError with a message explaining what the provider got wrong. Add a matching scenario to each harness's fake, then regenerate the matrix. A new case usually means a new guarantee, so document it in the relevant concept page too. The Published Matrix docs/reference/conformance-matrix.md is generated from a real run. Never hand-edit it: a hand-maintained matrix drifts from reality and then misleads. --- # Contributing / Testing Source: https://anyinfer.dev/contributing/testing/ Testing Guide How to run and write tests for AnyInfer itself. If you are testing an application that uses AnyInfer, the guide is test your application offline; the fakes it teaches are the same public anyinfer.testing package this suite is built on. Running Two tracks, because the gate and the inner loop want different things. workspace test # fast track — seconds workspace test --provider cohere # one provider's modules + shared invariants workspace check # the gate: everything, plus lint/types/docs workspace test cannot run the whole suite, by design. check runs the quality gates and is the only thing that tells you the suite passes, so there is exactly one answer to "is it green"; test tells you the code you are editing still works, which is a different question. The fast track skips two markers: Marker What it covers Run it when exhaustive The full preset matrix: eighty-six presets through every conformance case. Half the suite's wall time, and it re-proves the shared OpenAI dialect. You changed openai_compat.py, a preset entry, or the conformance suite itself. slow Packaging and subprocess builds. Before committing; check runs it. Adding or editing one adapter changes nothing either marker covers, which is the point: that work needs its own module and the shared invariants, not the other twenty adapters. Everything runs in parallel by default (pytest-xdist, -n auto), which is worth a ~7x speedup: every test builds its own in-process fakes, so there is nothing to share and nothing to serialize on. Pass -j0 to debug in a single process. Raw pytest still works, and is what you want for a single test: pytest -q -n auto # everything, parallel pytest tests/test_routing.py -q # one module pytest -k "fallback" -q # by name pytest -q --durations=15 # find slow tests filterwarnings = ["error"] is set: a ResourceWarning from a leaked socket or an unclosed event loop fails the suite. That strictness caught three real concurrency bugs in the llama-server supervisor. Where a Test Belongs Testing Put it in A behavior every provider must have testing/conformance.py (the conformance suite) One provider's dialect quirks tests/test_.py Core logic (routing, schema, events) The matching tests/test_*.py A serve-frontend invariant tests/test_openai_roundtrip.py If a behavior should hold for all providers, it belongs in the conformance suite so it is checked for all of them, not just the one you were looking at. Fakes, Not Sockets from anyinfer.testing.fakes import FakeOpenAIServer, FakeResponse server = FakeOpenAIServer(FakeResponse(text="hello", finish_reason="stop")) client = ai.AsyncClient( [ ai.ProviderSettings.of( "openai-compat", base_url="https://fake.invalid/v1", transport=server.transport() ), ] ) Fakes are httpx2 transports: no ports, no cleanup races, identical on every platform. They can be scripted for errors, malformed SSE, servers that ignore stream, omitted usage chunks, and multi-response sequences for retry and repair tests. The full fake-server surface is in the testing API. FakeOpenAIServer( [ FakeResponse(status=503), # first attempt fails FakeResponse(text="recovered"), # retry succeeds ] ) Assert on what was actually sent: assert server.requests[0]["temperature"] == 0.3 assert server.call_count == 2 Cassettes Recorded real traffic, replayed deterministically; bodies pass through redaction before touching disk, so a committed cassette cannot carry a key. The record, replay, and audit API is under recording, and contributing a cassette covers the workflow. Writing Good Tests Here Name the behavior, not the function. def test_unknown_memory_is_not_confident() -> None: ... # yes def test_recommend_alias_2() -> None: ... # no Explain non-obvious assertions. A one-line docstring on a test that encodes a subtle rule saves the next reader a lot of guessing: def test_an_active_stream_is_never_collected() -> None: """A long generation with no *new* requests is not idle — the classic false-idle kill.""" Assert the message, not just the type, when the message is the feature: assert excinfo.value.hint is not None assert "anyinfer[keyring]" in excinfo.value.hint Test the failure mode you are defending against. Most of the sharpest tests in this suite exist because a comparable tool shipped that exact bug. Async asyncio_mode = "auto" is set; write async def test_... with no decorator. Subprocess Tests tests/test_local_server.py spawns a fake llama-server (a Python script behind a platform-appropriate shim) rather than requiring a real llama.cpp build. It exercises the supervisor's real contract: spawn, poll, distinguish loading from failed, reap. Since it drives actual subprocesses, it is the slowest module in the suite. --- # Contributing / Branching and Releases Source: https://anyinfer.dev/contributing/releasing/ Branching and Releases How changes travel from a feature branch to a published release, and what is mechanical versus what a maintainer decides. The Branch Model flowchart LR A["feature/<topic>"] -->|PR| B[develop] B -->|PR| C[main] C --> D[release packages] feature/*: all work happens here, one topic per branch. Branched from develop. develop: the integration branch. Feature branches merge in by pull request; CI must be green before the merge button works. main: always releasable. Receives only pull requests from develop, again gated on CI. Every merge to main rebuilds the release packages; a merge that bumps the version also publishes them. Both protected branches require exactly one status check: the aggregate ci-ok. Requiring one stable check name means adding a CI job (or a matrix row) can never silently escape the protection rules. ci-ok passes when every upstream job either succeeded or was skipped by one of two deliberate conditions; a job that runs and fails always fails the gate. The two deliberate skips exist to keep billed minutes proportional to risk. First, a change confined to docs/, mkdocs.yml, and overrides/ skips the whole test matrix (the changes job classifies it); the docs gates, the executable doc examples, and the strict site build still run on every change. Second, the expensive lanes — the two middle interpreters and the macOS row, which bills at 10x Linux — run only on the pull request into main, where a merge should prove the release is buildable, and not on feature PRs into develop. CI itself is triggered by pull requests into either protected branch, and by pushes to main (which run the bracketed matrix only when code changed, never the main-PR-only lanes; their verdict was already produced on the promotion PR). Pushes to develop do not trigger it: develop only takes merges through a pull request that already had to be green. Promotions from develop to main use a merge commit, never a squash or rebase merge: those mint new commit ids on main and permanently diverge the two branches. After the promotion merges, fast-forward develop onto main so both branches point at the same commit. Versioning The single source of truth is project.version in pyproject.toml; anyinfer.__version__ mirrors it and a test keeps them in agreement. Pre-1.0, versions follow 0.MINOR.PATCH: breaking changes bump MINOR, everything else bumps PATCH. From 1.0.0 on, plain SemVer. Bumping the version is an ordinary change: edit both files on a feature branch and let it ride to main through develop. What a Release Is The release workflow runs on every merge to main: It reads project.version and checks whether tag v already exists. It rebuilds the release artifacts, so main is continuously proven releasable: the library: sdist + wheel, twine check-ed and smoke-installed, on every merge; the demo bundles: standalone PyInstaller builds of the pack-in demo app on native runners for Windows (x64), macOS (arm64 and x64), and Linux (x64 and arm64), named without a version (anyinfer-demo--.zip) so the site's downloads page can link releases/latest/download/ URLs that never go stale; the sidecar bundles: native builds on the same runners, named anyinfer-serve--.zip, with a build-time --help smoke test. Since freezing a PySide6 application on five runners (two of them macOS, at 10x Linux billing) is the most expensive thing this repository asks CI to do, the bundle matrix depends on the version: a version bump builds all five, and an unchanged version builds only the Linux x64 canary that catches a change breaking the frozen build at all. Platform-specific freeze breakage therefore surfaces at the version bump rather than at the merge before it; no release can be cut without all five going green. 3. Only when the version is new does it tag v and create the GitHub Release, with notes generated from the merged pull requests, every package attached, and a SHA256SUMS file covering every artifact. An unchanged version (docs-only merges, CI tweaks) leaves what it built as workflow artifacts and cuts nothing; releases stay 1:1 with versions. Publishing a release therefore takes exactly one deliberate act: merging a version bump to main. There is no separate tagging step to forget or get wrong, and a release's tag always points at the exact commit it was built from. The docs site redeploys on every merge to main and again when a release publishes, so the site and the newest release never disagree for long. PyPI The release workflow's publish-pypi job uploads the library distribution to PyPI on the same condition that cuts a GitHub Release: a new version on main. It downloads the library-dist artifact rather than rebuilding, so what lands on the index is byte-for-byte what twine check passed, what the smoke test installed, and what is attached to the release — one build, three destinations. Uploads authenticate by Trusted Publishing (OIDC): PyPI mints a short-lived credential for a workflow run whose repository, workflow file, and environment match the project's publisher configuration. No API token exists in this repo's secrets, so there is none to leak or rotate. Because publishing is irreversible (a version number on PyPI can be yanked but never reused), the job runs in the pypi environment, which is where a required-reviewer gate belongs if you want a human to approve each upload. The version bump is still the single deliberate act; the environment just adds a pause before the copy leaves the building. Checklist for Cutting a Release develop is green and contains everything the release should. On a feature branch: bump project.version and anyinfer.__version__, note anything user-facing in the PR description (it becomes the generated release notes). PR into develop; merge when green. PR develop into main; merge when green. Watch the release workflow attach v and publish the wheel to PyPI. Verify the downloads page, checksum file, and PyPI project page. If a step fails, when a release goes wrong lists the recovery for each failure mode. Native beta bundles are not code-signed. macOS Gatekeeper and Windows SmartScreen may therefore require an explicit local approval. Signing and notarization require external certificates and are a release-infrastructure follow-up; the wheel, source distribution, checksums, and reproducible workflow remain the authoritative 0.1 release path. When a Release Goes Wrong Publishing is the only irreversible step in this repository: a version number on PyPI can be yanked but never reused, not even after deleting the file. Recovery by failure mode: Symptom What happened What to do Release cut, publish-pypi failed The publisher fields or the pypi environment do not match the run Fix the registration on PyPI, then re-run the failed job from the Actions run page. Upload rejected: file already exists That version was uploaded before Nothing to recover. Bump to the next patch version and let it ride to main again. A published version is broken It is on the index and installable Yank it (Manage → Releases → Yank): resolvers stop selecting it while existing pins keep working. Then release a fix. Deleting instead burns the number permanently. Release cut from the wrong commit The tag points somewhere unintended Delete the GitHub Release and its tag, fix main, and bump the version; reusing the tag would disagree with whatever PyPI already accepted. Run stuck before uploading The pypi environment is waiting on a required reviewer Approve the deployment on the run page. A run left pending is failed automatically after 30 days. Re-running publish-pypi never rebuilds: it downloads the same library-dist artifact the build job produced, so a retry cannot ship different bytes than the ones already attached to the GitHub Release. That artifact is subject to the repository's normal artifact retention (90 days by default); after it expires, re-run the whole workflow rather than the single job.