Skip to content

Structured summaries with a fallback chain

A command-line tool that turns arbitrary text into a schema-validated summary, staying up when a provider is not: it tries Anthropic first, falls back to OpenAI, and finally to a local Ollama model. Whatever path the request took, the caller gets the same validated shape back — or one typed error naming every attempt.

"""summarize.py — `python summarize.py < release-notes.txt`"""

import json
import sys

import anyinfer as ai

SUMMARY_SCHEMA = {
    "type": "object",
    "properties": {
        "headline": {"type": "string"},
        "topics": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["headline", "topics"],
}

client = ai.Client(
    [
        ai.ProviderSettings.of("anthropic", api_key="env://ANTHROPIC_API_KEY"),
        ai.ProviderSettings.of("openai", api_key="env://OPENAI_API_KEY"),
        ai.ProviderSettings.of("ollama"),
    ]
)

result = client.generate(
    "Summarize this text:\n" + sys.stdin.read(),
    route=ai.Route(
        targets=(
            "anthropic:claude-sonnet-4-5",
            "openai:gpt-5.2",
            "ollama:qwen3:8b",
        ),
        retry=ai.Retry(max_attempts=2),
    ),
    schema=SUMMARY_SCHEMA,
    repair=ai.Repair(max_attempts=1),
)

print(json.dumps(result.structured, indent=2))
print(f"via {result.target} using {result.structured_mechanism}", file=sys.stderr)
for attempt in result.attempts:
    print(f"  {attempt.target}: {attempt.outcome}", file=sys.stderr)

What to notice

  • result.structured is always valid against SUMMARY_SCHEMA — validation happens client-side regardless of which provider answered, and result.structured_mechanism tells you honestly how it was enforced (grammar, json_schema, json_mode, or prompt). See structured output.
  • The fallback chain is data, not code. Route.targets is an ordered tuple; retry policy applies per target. No try/except pyramid, and the attempt trail (result.attempts) records every hop for your logs.
  • Credentials never appear in source. env://ANTHROPIC_API_KEY is a credential reference; the resolved secret is registered for redaction, so it cannot leak through errors or telemetry.
  • The local fallback needs no key at all. If both hosted providers are down, the same call lands on Ollama — and if everything fails, you get one AllTargetsFailedError carrying the per-target causes, not the last exception to happen to escape.

Related guides: Add a fallback chain · Enforce a JSON schema