Skip to content

Quickstart

From pip install to a working result. Every example on this page is executed in CI against the fake providers, so none of it can quietly rot.

Install

pip install anyinfer

The core depends on httpx2 and jsonschema and nothing else. Providers that need more come as extras; see installation.

The fastest start is anyinfer init: it inspects the machine, reports which providers are already usable (a running Ollama, a set credential variable), and writes a valid anyinfer.json plus a runnable starter.py, without ever storing a secret or installing anything. The CLI guide covers what it detects and its flags. The file it writes is the shared configuration the SDK, CLI, and sidecar all read.

Your First Call

import anyinfer as ai

text = "The 0.1 release adds embeddings, reranking, and an OpenAI-compatible sidecar."

client = ai.Client(
    [
        ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"),
    ]
)

result = client.generate(
    "Summarize this in one sentence:\n" + text,
    target="anthropic:claude-sonnet-4-5",
)

print(result.text)
client.close()

import anyinfer as ai

text = "The 0.1 release adds embeddings, reranking, and an OpenAI-compatible sidecar."

async with ai.AsyncClient(
    [
        ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"),
    ]
) as client:
    result = await client.generate(
        "Summarize this in one sentence:\n" + text,
        target="anthropic:claude-sonnet-4-5",
    )

    print(result.text)

The highlighted line is the only one that changes when you point the same call at a different provider or a local model.

Note the credential: "env://ANTHROPIC_API_KEY" is a reference, safe to keep in a config file. It is resolved once and registered for redaction, so the key can never appear in a log line or an error message. See credentials.

Client owns a background event loop, so use it as a context manager or call close():

with ai.Client([ai.ProviderSettings.of("ollama")]) as client:
    result = client.generate("Why is the sky blue?", target="ollama:qwen3:8b")

Streaming

with client.stream(messages, target="ollama:qwen3:8b") as stream:
    for event in stream:
        if isinstance(event, ai.TextDelta):
            print(event.text, end="", flush=True)

    final = stream.result
    print(f"\n\n{final.usage.output_tokens} tokens in {final.timing.total_ms:.0f} ms")

Using the stream as a context manager matters: leaving the block early cancels the in-flight request instead of letting it run on. See stream typed events.

Aliases: Don't Hardcode Model Names

small, medium, and large resolve to a concrete model for whichever provider you have configured:

client = ai.Client(
    [
        ai.ProviderSettings.of("ollama"),
        ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"),
    ]
)

result = client.generate(prompt, target="medium")  # -> ollama, since it is listed first

The order you configure providers is the preference order. See targets and aliases.

Structured Output

Pass a JSON schema and get back a validated Python value:

SUMMARY = {
    "type": "object",
    "properties": {
        "headline": {"type": "string"},
        "topics": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["headline", "topics"],
}

result = client.generate(
    article,
    target="medium",
    schema=SUMMARY,
    repair=ai.Repair(max_attempts=1),
)

print(result.structured["headline"])
print(result.structured_mechanism)  # "json_schema", "grammar", "json_mode", or "prompt"

AnyInfer uses the strongest mechanism the provider supports, then validates the result against your schema regardless. repair allows the model to correct itself once if it gets the shape wrong. See structured output.

Fallback Chains

route = ai.Route(
    targets=("anthropic:claude-sonnet-4-5", "openai:gpt-5", "ollama:qwen3:8b"),
    retry=ai.Retry(max_attempts=3),
)

result = client.generate(prompt, route=route)

print(f"served by {result.target}")
for attempt in result.attempts:
    print(f"  {attempt.target} -> {attempt.outcome}")

Every result carries its full routing trail, so "why was this slow?" is answerable after the fact. See routing and rate limits.

Beyond Generation

The same client embeds and reranks (client.embed(), client.rerank()), typed and routed like generation, with a safety rule that keeps fallback from mixing incompatible vector spaces. And a local model is just another target: with llama-cpp configured, one generate() call downloads a pinned, hash-verified model, tunes a server for your hardware, and answers on loopback; see run a model locally.

Key Takeaways

  • One call shape covers every provider; only the target= string changes.
  • Credentials are references (env://…), resolved once and redacted everywhere.
  • A schema is validated client-side no matter which mechanism the provider offers.
  • Every result carries its attempt trail, so routing decisions are inspectable after the fact.

See Also