# AnyInfer > AnyInfer provides an application-owned hybrid inference runtime for Python, spanning hosted APIs, routing hubs, existing local services, and supervised llama.cpp, as well as the typed contract that makes them behave alike. Version 0.1.2. Full text of every page below: https://anyinfer.dev/llms-full.txt (all sections; larger than one context window). Each section heading links its own full-text bundle, sized to fit one. ## Home Full text of this section: https://anyinfer.dev/llms/home.txt - [Home](https://anyinfer.dev/): AnyInfer provides a provider-independent inference runtime for Python applications that span hosted providers and local models, as well as an OpenAI-compatible sidecar for everything that is not Py... ## Integrate Full text of this section: https://anyinfer.dev/llms/integrate.txt - [Integrate AnyInfer](https://anyinfer.dev/guides/): Three supported production paths share the same core behavior and configuration file; only the process boundary changes. Not every application needs this layer at all; if a provider-switching clien... - [Quickstart](https://anyinfer.dev/guides/quickstart/): From pip install to a working result. Every example on this page is executed in CI against the fake providers, so none of it can quietly rot. - [Installation and Extras](https://anyinfer.dev/guides/installation/): The core depends on httpx2 and jsonschema and nothing else. That constraint is a security argument as much as an aesthetic one: a small mandatory dependency surface is a small supply-chain attack s... - [Why and When to Use AnyInfer](https://anyinfer.dev/why-anyinfer/): Most libraries in this space solve provider switching: one function, many APIs, one response shape. That is a real problem and several tools solve it well. AnyInfer is built for the problem that st... - [Shared Configuration](https://anyinfer.dev/reference/configuration/): AnyInfer has one JSON configuration format for every integration method: load it from Python with load_config, pass it to anyinfer run, or start the OpenAI-compatible sidecar with it; each path is... - [Python SDK](https://anyinfer.dev/guides/python-sdk/): Use the SDK when AnyInfer runs inside a Python application. Quickstart is the fastest path to a first result; this page is the reference for embedding the SDK properly: the client lifecycle and the... - [Command-Line Tool](https://anyinfer.dev/guides/cli/): anyinfer run sends one prompt through the same path the library uses (routing, fallback, structured output, telemetry) and then exits. It is the shell-shaped way to reach everything AnyInfer abstra... - [OpenAI-Compatible Sidecar](https://anyinfer.dev/serve/): An OpenAI-compatible projection of the same configured hybrid runtime used by the Python SDK and command-line tool. It allows existing clients to use hosted, hub, and local routes without creating... - [Running the Sidecar as a Service](https://anyinfer.dev/serve/running-as-a-service/): An application pointing at http://127.0.0.1:8080/v1 needs that endpoint to exist at boot, not just while somebody keeps a terminal window open. anyinfer serve install writes the systemd unit, launc... - [Stream Typed Events](https://anyinfer.dev/guides/streaming/): A stream yields typed events while the request runs: text deltas, reasoning, timing marks, and attempt failures. This page shows the patterns a terminal frontend needs; the full event vocabulary is... - [Enforce a JSON Schema](https://anyinfer.dev/guides/structured-output/): Pass a schema and result.structured comes back already validated against it, whichever mechanism the target supports: - [Add a Fallback Chain](https://anyinfer.dev/guides/fallback/): A Route names the targets to try in order and the retry policy for each; the full semantics live in routing. - [Run the Tool Loop](https://anyinfer.dev/guides/tool-loop/): run_tools runs the generate, call tools, feed results back loop for you, with bounded rounds and normalized errors: - [Fit a Corpus to a Budget](https://anyinfer.dev/guides/fitting-context/): You have a pile of documents and a model with a finite window. This is the four-step pattern: build documents, ask what fits, reduce, and place the result. - [Test Your Application Offline](https://anyinfer.dev/guides/testing-your-app/): Your application's inference code has behavior worth testing: it falls back when a provider is down, it repairs a malformed structured answer, it reduces a corpus to fit a budget. Testing that norm... - [Compare Targets Without Spending](https://anyinfer.dev/guides/comparing-targets/): compare() resolves a concrete request (the exact messages, schema, tools, sampling controls, and cache policy) against every target you name and reports what it would become on each: whether it fit... - [Add Your Own Provider](https://anyinfer.dev/guides/custom-providers/): A provider AnyInfer does not ship is not a fork: a provider is a small installable package, and once it is installed, yourprovider:model targets resolve everywhere: the Python API, the command line... - [Embed, Store, and Query a Small Corpus](https://anyinfer.dev/guides/vector-store/): anyinfer-store is an embedded vector store: one SQLite file that persists the vectors client.embed() returns and answers similarity queries in-process, with no server. It serves the same audience l... - [Run a Model Locally](https://anyinfer.dev/guides/local-inference/): From a bare machine to a generated answer without installing or operating a separate model daemon. AnyInfer shows what this machine can run, downloads and hash-verifies the weights, fetches a pinne... - [Observability](https://anyinfer.dev/guides/observability/): Every request emits typed telemetry events as it runs. This page shows how to consume them in-process and how to export them to OpenTelemetry; the event stream page covers the per-request events a... - [Confidentiality Tiers](https://anyinfer.dev/guides/confidentiality-tiers/): BYOK (bring-your-own-key) inference already answers one confidentiality question: your application's calls go straight from your process to the provider you configured, AnyInfer is never a proxy, a... - [Coding Agents](https://anyinfer.dev/guides/coding-agents/): The most common way a library gets integrated in 2026 is that a coding agent writes the integration. That path has a specific failure mode here, and this page is about closing it. - [Integration Procedure](https://anyinfer.dev/agents/INTEGRATION/): The canonical, tool-neutral procedure for a coding agent adding or changing AnyInfer code in an application. The Codex skill, the Claude Code skill, and the Copilot prompt are thin entry points tha... - [Reference Application](https://anyinfer.dev/guides/demo-app/): AnyInfer ships a PySide6 reference application in src/demo_app/. It is a worked example of integration, not part of the library's public API; nothing in anyinfer imports it, and nothing in it is im... ## Concepts Full text of this section: https://anyinfer.dev/llms/concepts.txt - [Concepts](https://anyinfer.dev/concepts/): Eighteen ideas. Read them once and the rest of the API follows from them. They build on each other roughly in this order, but each page stands alone. - [Targets and Aliases](https://anyinfer.dev/concepts/targets/): A target says where a request goes. There are two spellings and one resolution path. - [The Event Stream](https://anyinfer.dev/concepts/events/): A generation is an ordered stream of typed events. Everything else (the non-streaming generate(), the OpenAI chunk format, a progress bar) is a projection of that one primitive. This page covers th... - [Routing and Rate Limits](https://anyinfer.dev/concepts/routing/): A route decides where a request goes and what happens when an attempt fails: retries, fallback chains, and health gating, all deterministic and fully traceable afterward. Rate limiting is the other... - [Structured Output](https://anyinfer.dev/concepts/structured-output/): A schema is a contract, not a hint. Passing one returns a value that satisfies it, or an error explaining why not; never a "mostly right" string to re-parse. - [Sessions](https://anyinfer.dev/concepts/sessions/): Every request is independent by default, which keeps results reproducible and fallback safe. A conversation is the case where that default wastes work: the provider often already has everything the... - [Embeddings and Reranking](https://anyinfer.dev/concepts/embeddings/): Embedding and reranking are stateless inference operations, typed and routed the same way generation is, but they are not generation. EmbeddingRequest and RerankRequest are their own types; nothing... - [Multimodal Inputs](https://anyinfer.dev/concepts/multimodal-inputs/): AnyInfer generation requests may contain images, documents, and audio alongside text. The output is still text and tool calls; this does not add image generation, speech output, transcription, or a... - [Capabilities and Provenance](https://anyinfer.dev/concepts/capabilities/): Every capability value records where it came from: its provenance. Providers omit, misreport, and change these numbers, so before routing, budgeting, or billing against one, a consumer needs to kno... - [Token Estimation and Context Budgets](https://anyinfer.dev/concepts/budgeting/): How many input tokens will this request spend, and does it fit the model it is going to? Every app that assembles large prompts ends up hand-rolling this arithmetic per provider. AnyInfer answers i... - [Cost and Spending](https://anyinfer.dev/concepts/cost/): AnyInfer computes what each call cost, keeps a per-client spend ledger, and can refuse a request before it crosses a ceiling. One rule underlies all three: an unknown cost is reported as unknown, n... - [Prompt Caching](https://anyinfer.dev/concepts/caching/): Most providers can hold on to a prefix of the prompt and charge less the next time they see it. What "hold on to it" means differs: some want to be told exactly where the reusable part ends, others... - [Context Reduction](https://anyinfer.dev/concepts/context-reduction/): You have more material than the model's window holds. anyinfer.context decides what to send and tells you exactly what it dropped. Together with client.budget(), this makes context preparation part... - [The Local Subsystem](https://anyinfer.dev/concepts/local/): Running a model on the local machine should be one target string, with the same guarantees as a hosted call. This page covers the machinery behind that string: hardware detection, server tuning, su... - [The Model Catalog](https://anyinfer.dev/concepts/catalog/): The catalog answers three questions in sequence: what local models exist, whether this machine can run them, and how a pick becomes verified bytes an engine can load. It serves two shapes of caller... - [Credentials and Redaction](https://anyinfer.dev/concepts/credentials/): Two guarantees: - [Telemetry](https://anyinfer.dev/concepts/telemetry/): The telemetry contract is typed in-process events delivered to registered observers. OpenTelemetry is an optional bridge over that contract, not the contract itself. - [Run Manifests](https://anyinfer.dev/concepts/run-manifests/): A run manifest is the portable explanation of one generation. It records the route that won, attempts and fallback, structured-output and cache mechanisms, context reductions, capability provenance... - [Arena Runs](https://anyinfer.dev/concepts/arena/): An arena sends the same request to a fixed set of targets and selects one answer, keeping every candidate as evidence. A three-target arena costs up to three ordinary generations before selection,... ## Examples Full text of this section: https://anyinfer.dev/llms/examples.txt - [Examples](https://anyinfer.dev/examples/): Small, complete programs, not fragments. Each one is a pattern the library was designed around. The shape of every example is exercised in CI against the in-process fake providers (tests/test_docs_... - [Structured Summaries with Fallback](https://anyinfer.dev/examples/summarize-with-fallback/): A command-line tool that turns arbitrary text into a schema-validated summary, staying up when a provider is not: it tries Anthropic first, falls back to OpenAI, and finally to a local Ollama model... - [A Local Tool-Calling Assistant](https://anyinfer.dev/examples/local-tool-agent/): An assistant that answers questions about your project by calling Python functions you hand it. It does not run offline as written: it needs a running Ollama with qwen3:8b pulled. Since that is the... - [Distill a Corpus](https://anyinfer.dev/examples/distill-a-corpus/): distill reads material that will never fit at any fidelity and writes something shorter: each chunk is summarized against your question, then the notes are synthesized into one answer. Since that s... - [Regression-Test Fallback and Repair](https://anyinfer.dev/examples/golden-manifest/): This test asserts the part of inference that should be deterministic (which route ran and which mechanism enforced the schema), and it runs offline: a scripted provider and stored fixtures, no netw... - [Semantic Search over a Small Corpus](https://anyinfer.dev/examples/semantic-search/): For a corpus small enough to hold in memory, semantic search needs no vector store: this example embeds a handful of documents, embeds a query with the matching intent, and ranks by cosine similari... ## Providers Full text of this section: https://anyinfer.dev/llms/providers.txt - [Providers](https://anyinfer.dev/providers/): This section is the compatibility inventory: dedicated adapters for protocols that require real translation, plus declarative presets for OpenAI-compatible services and engines. Breadth is useful,... - [Every Provider](https://anyinfer.dev/providers/all/): This is AnyInfer's compatibility inventory, not its primary value proposition: 106 providers comprising 20 dedicated adapters with provider-specific behavior and 86 presets over the shared OpenAI-c... - [Hosted & Local Presets](https://anyinfer.dev/providers/presets/): One implementation, many brandings. Every provider on this page speaks the chat-completions dialect closely enough that AnyInfer's shared OpenAI-compatible adapter covers it, so each ships as a pre... - [OpenAI](https://anyinfer.dev/providers/openai/): Uses the Responses API, OpenAI's current surface, which exposes reasoning effort and reasoning-token accounting the older chat-completions shape does not. For the chat-completions dialect, point op... - [Anthropic](https://anyinfer.dev/providers/anthropic/): The Messages API over raw httpx2. Registered as anthropic, with the alias claude. - [Google Gemini](https://anyinfer.dev/providers/gemini/): The native generateContent protocol. Google's OpenAI-compatibility layer is documented as beta and ignores parameters it does not implement, while thinking levels, response schemas, safety settings... - [DeepSeek](https://anyinfer.dev/providers/deepseek/): An OpenAI-compatible dialect with three deltas that would otherwise cost a developer silently: reasoning arrives on its own channel, thinking is on by default, and cache accounting is automatic and... - [xAI (Grok)](https://anyinfer.dev/providers/xai/): An OpenAI-compatible dialect whose distinctive value is reported cost: xAI reports the exact amount billed on every response, and its model listing carries real prices and context windows. - [Google Vertex AI](https://anyinfer.dev/providers/vertex/): The same Gemini models as the AI Studio API, over the same protocol, with enterprise addressing and Google Cloud authentication. AnyInfer reuses the Gemini adapter's translation wholesale: Vertex c... - [AWS Bedrock](https://anyinfer.dev/providers/bedrock/): The Converse API: Bedrock's unified interface, where one request shape serves Claude, Nova, Llama, Mistral, and DeepSeek alike. Generation never uses InvokeModel and its per-model request bodies; o... - [Cohere](https://anyinfer.dev/providers/cohere/): The native v2 Chat API, chosen over the OpenAI compatibility layer because v2 is where the things worth choosing Cohere for live: grounded generation with document citations, a separate thinking ch... - [Azure AI Foundry](https://anyinfer.dev/providers/azure-foundry/): An openai-compat subclass carrying Azure's parameter renames and its two authentication modes. - [GitHub Copilot](https://anyinfer.dev/providers/copilot/): The only adapter that is not raw HTTP. Copilot is reached by driving the Copilot CLI as a subprocess runtime through github-copilot-sdk, so there is no wire protocol for AnyInfer to speak (the slim... - [Microsoft 365 Copilot](https://anyinfer.dev/providers/m365-copilot/): The most constrained provider in the set. - [OpenRouter](https://anyinfer.dev/providers/openrouter/): An openai-compat subclass. Its distinctive value is the model listing: OpenRouter reports per-model context length and per-token pricing, so its costs carry discovered provenance rather than catalo... - [Nebius Token Factory](https://anyinfer.dev/providers/nebius/): Nebius uses the OpenAI chat-completions dialect but exposes a richer model listing. AnyInfer uses that listing to discover current context windows, quantization, feature support, and prices instead... - [Ollama](https://anyinfer.dev/providers/ollama/): Uses Ollama's native /api/chat API, not its /v1 OpenAI-compatibility layer. The native API carries grammar-enforced structured output, per-phase nanosecond timings, keep_alive session retention, an... - [LM Studio](https://anyinfer.dev/providers/lm-studio/): LM Studio's local server. Generation uses its OpenAI-compatible endpoint, a shared and well-understood dialect; discovery uses the native API, which reports what a local engine's model list actuall... - [llama.cpp](https://anyinfer.dev/providers/llama-cpp/): A supervised llama-server subprocess speaking the OpenAI-compatible dialect over loopback. In-process llama-cpp-python is not supported: one wire protocol for every engine, crash isolation, and no... - [Text Embeddings Inference](https://anyinfer.dev/providers/tei/): Hugging Face's TEI server, spoken in its native dialect; the one local provider with a real reranking endpoint. TEI serves exactly one model per container (an embedding model or a reranker, chosen... - [Voyage AI and Jina AI](https://anyinfer.dev/providers/retrieval/): The two hosted specialist retrieval providers: embeddings and reranking, no generation. Both serve native query/document input intents and a real reranker; they differ in task vocabulary, batch cei... - [OpenAI-Compatible](https://anyinfer.dev/providers/openai-compat/): The base dialect for any endpoint speaking POST /chat/completions: vLLM, LM Studio, an externally-run llama-server, a corporate gateway, or OpenAI itself. ## Reference Full text of this section: https://anyinfer.dev/llms/reference.txt - [Reference](https://anyinfer.dev/reference/): Look things up. - [SDK Reference](https://anyinfer.dev/reference/api/): Generated from the docstrings of every public symbol in anyinfer; coverage is a CI gate, so nothing here is an empty page. The core surface is importable from the top-level package; the local, serv... - [Clients and Streams](https://anyinfer.dev/reference/api/client/): The two entry points, Client (sync) and AsyncClient (async), expose the same surface; the sync client is a facade over the async core (see the architecture overview). - [Requests and Messages](https://anyinfer.dev/reference/api/requests/): The request side of the one primitive: everything a GenerationRequest can carry. See the event stream for how a request becomes output. - [Results and Stream Events](https://anyinfer.dev/reference/api/results/): The output side: the final Generation, its usage and timing, and the typed events a stream yields on the way there. Ordering guarantees are documented in the event stream. - [Embeddings and Reranking](https://anyinfer.dev/reference/api/embeddings/): Two stateless inference operations alongside generation: turning text into vectors, and ranking documents against a query. Both are typed and routed the same way generation is (target resolution, r... - [Portability Diff Tool](https://anyinfer.dev/reference/api/compare-diff/): anyinfer.compare_diff: snapshot compare() output for a fixture set, and diff two snapshots structurally. No ranking, scoring, or live provider calls; every function here either calls compare() (its... - [Vector Store Add-On](https://anyinfer.dev/reference/api/vector-store/): anyinfer_store: a small-scale, single-process, embedded vector store; a separate installable distribution, never imported by anyinfer core and never a dependency of it. See the vector store guide f... - [Routing](https://anyinfer.dev/reference/api/routing/): Retry, fallback, and target resolution: the policy layer that adapters are forbidden from containing. Behavior is described in routing. - [Capabilities](https://anyinfer.dev/reference/api/capabilities/): Provenance-tagged model metadata: every value knows whether it was cataloged, discovered, probed, or defaulted. The reasoning is in capabilities and provenance; token estimation and the budget calc... - [Context Reduction](https://anyinfer.dev/reference/api/context/): Fit a document corpus to a token budget. The application collects; this subpackage reduces. The reasoning and the strategy tradeoffs are in context reduction; the task-oriented walkthrough is fitti... - [Telemetry and Redaction](https://anyinfer.dev/reference/api/telemetry/): Typed in-process events, payload-free by default, plus the redaction registry that keeps secrets out of everything. Concepts: telemetry · credentials and redaction. - [Registry, Catalog, Credentials](https://anyinfer.dev/reference/api/registry/): How providers describe themselves (frozen descriptors, declarative setup specs), how targets and aliases resolve, and how credential references become secrets. Concepts: targets and aliases · crede... - [Configuration](https://anyinfer.dev/reference/api/configuration/): The versioned JSON loader and what it produces: load_config and loads_config parse and validate a file into an AnyInferConfig, and dump_config and dumps_config write the same format back. The prose... - [Local Inference](https://anyinfer.dev/reference/api/local/): The anyinfer.local subsystem: hardware detection, backend selection, runtime acquisition, tuning, fit classification, model acquisition and storage, server supervision, and hardware→tier recommenda... - [Sidecar Frontend](https://anyinfer.dev/reference/api/serve/): The anyinfer.serve frontend: an OpenAI-compatible loopback service over any configured provider, embeddable as an ASGI app. Guide: serve. - [Testing Utilities](https://anyinfer.dev/reference/api/testing/): The anyinfer.testing package is public on purpose, for two audiences. An application tests its own routing, repair, and reduction logic against a scripted provider (guide: test your application off... - [Errors](https://anyinfer.dev/reference/api/errors/): A shallow hierarchy with structured fields (provider, phase, retryable, http_status, detail, hint); detail is bounded and redacted, hint is the actionable next step. The prose catalog with examples... - [Error Catalog](https://anyinfer.dev/reference/errors/): Every exception AnyInfer raises, when it is raised, and what the user will see. - [Conformance Matrix](https://anyinfer.dev/reference/conformance-matrix/): Generated from a real conformance run — do not edit by hand. Regenerate with python workspace.py matrix. - [Glossary](https://anyinfer.dev/reference/glossary/): Terms this project uses precisely. Where a word has a loose industry meaning and a specific meaning here, the specific one is what the code implements. ## Contributing Full text of this section: https://anyinfer.dev/llms/contributing.txt - [Contributing](https://anyinfer.dev/contributing/): Security vulnerabilities do not belong in public issues. Follow the repository's security policy for private reporting. - [Architecture](https://anyinfer.dev/contributing/architecture/): The condensed version. DESIGN.md §23 has the complete rationale; the glossary defines the vocabulary these rules use. - [Coding Agents and Workstreams](https://anyinfer.dev/contributing/automation/): AnyInfer supports Codex, Claude Code, and GitHub Copilot with one authoritative instruction set. AGENTS.md is the source of truth for repository rules, architecture constraints, tests, and workstre... - [Branding and Visual Assets](https://anyinfer.dev/contributing/branding/): The canonical logo files and palette live in docs/assets/: - [Writing an Adapter](https://anyinfer.dev/contributing/writing-an-adapter/): An adapter translates. That is the whole job, and keeping it that way is what lets one conformance suite cover every provider. This page explains the shape and the reasoning behind it. - [The Conformance Suite](https://anyinfer.dev/contributing/conformance/): One suite, run against every adapter. It is what makes "the behavior does not change when you change providers" a checked claim rather than an aspiration. The suite is public API, under anyinfer.te... - [Testing](https://anyinfer.dev/contributing/testing/): How to run and write tests for AnyInfer itself. If you are testing an application that *uses* AnyInfer, the guide is test your application offline; the fakes it teaches are the same public anyinfer... - [Branching and Releases](https://anyinfer.dev/contributing/releasing/): How changes travel from a feature branch to a published release, and what is mechanical versus what a maintainer decides.