Results and stream events

The output side: the final Generation, its usage and timing, and the typed events a stream yields on the way there. Ordering guarantees are documented in the event stream.

The final result

anyinfer.Generation dataclass

Generation(
    text: str,
    structured: Any | None,
    tool_calls: tuple[ToolCall, ...],
    target: ResolvedTarget,
    finish_reason: FinishReason,
    usage: Usage,
    timing: Timing,
    structured_mechanism: Mechanism | None = None,
    repair_attempts: int = 0,
    attempts: tuple[AttemptRecord, ...] = (),
    warnings: tuple[str, ...] = (),
    raw: Any | None = None,
)

The final result of a generation request.

Attributes:

Name	Type	Description
text	str	The assistant's full text output. Empty when the model answered only with tool calls or structured output.
structured	Any | None	The parsed, schema-validated object when structured output was requested; None otherwise.
tool_calls	tuple[ToolCall, ...]	Tool invocations the model requested, in order. Empty when none.
target	ResolvedTarget	The provider and model that actually produced this result — after routing, so it may differ from the first target asked for.
finish_reason	FinishReason	Normalized reason the generation stopped.
usage	Usage	Token accounting, normalized across providers.
timing	Timing	Centrally-measured latency for the winning attempt.
structured_mechanism	Mechanism | None	How structured output was enforced for this result (grammar, json_schema, json_mode, or prompt); None when no schema was requested.
repair_attempts	int	How many schema-repair round-trips were needed before structured validated. 0 means the first response validated.
attempts	tuple[AttemptRecord, ...]	The full routing trail, including failed and retried attempts.
warnings	tuple[str, ...]	Non-fatal notices accumulated along the way (capability downgrades, estimated values, and the like).
raw	Any | None	The provider-native response payload, as an escape hatch for fields the normalized types do not carry. None unless the request asked to keep it.

anyinfer.Usage dataclass

Usage(
    input_tokens: int | None = None,
    output_tokens: int | None = None,
    total_tokens: int | None = None,
    cache_read_tokens: int | None = None,
    cache_write_tokens: int | None = None,
    reasoning_tokens: int | None = None,
    cost_usd: Decimal | None = None,
)

Token accounting for one generation.

Every field is optional: a provider that does not report a number leaves it None rather than reporting a guess.

Attributes:

Name	Type	Description
input_tokens	int | None	Tokens in the prompt, as counted by the provider.
output_tokens	int | None	Tokens the model generated, including tool-call payloads.
total_tokens	int | None	Prompt plus completion tokens; normalized fills it from the other two when the provider does not report it.
cache_read_tokens	int | None	Prompt tokens served from the provider's prompt cache.
cache_write_tokens	int | None	Prompt tokens written into the provider's prompt cache.
reasoning_tokens	int | None	Tokens spent on hidden reasoning/thinking, where reported.
cost_usd	Decimal | None	Cost of the call in US dollars, computed from per-token pricing when pricing is known.

normalized

normalized() -> Usage

Fill total_tokens from input + output when both are known.

merge

merge(other: Usage) -> Usage

Overlay other's known fields onto this one.

Later usage reports win; None never overwrites a known value. Streaming providers report usage incrementally, so the router merges rather than replaces.

anyinfer.Timing dataclass

Timing(
    started_at: float,
    first_token_ms: float | None = None,
    total_ms: float = 0.0,
    output_tokens_per_s: float | None = None,
    phases: Mapping[str, float] = dict(),
)

Centrally-measured timings for one attempt.

All values are measured by the core against time.monotonic() so that definitions are identical across providers. phases carries provider-reported sub-timings (e.g. Ollama's model load) in milliseconds.

Attributes:

Name	Type	Description
started_at	float	Monotonic-clock reading when the attempt began; meaningful only for computing intervals, not as wall-clock time.
first_token_ms	float | None	Time to the first content event (text, reasoning, or tool-call delta) after attempt start; None when no content ever arrived.
total_ms	float	Full duration of the attempt, start to completion.
output_tokens_per_s	float | None	Decode throughput, measured from first token to completion; None when output tokens or first-token time are unknown.
phases	Mapping[str, float]	Provider-reported sub-timings, keyed by phase name, in milliseconds.

anyinfer.AttemptRecord dataclass

AttemptRecord(
    target: ResolvedTarget,
    outcome: Outcome,
    error: ErrorInfo | None = None,
    timing: Timing | None = None,
)

One entry in a request's routing trail.

Attributes:

Name	Type	Description
target	ResolvedTarget	The provider and model this attempt was sent to.
outcome	Outcome	How the attempt ended.
error	ErrorInfo | None	Snapshot of the failure, for attempts that did not succeed.
timing	Timing | None	Measured timings, when the attempt progressed far enough to have any.

anyinfer.Outcome module-attribute

Outcome = Literal[
    "ok",
    "retried",
    "failed",
    "skipped_unhealthy",
    "redirected",
]

How a single routing attempt ended.

redirected marks a completed attempt whose content-filter refusal sent the route to Route.content_policy_targets instead of surfacing the refusal.

anyinfer.ErrorInfo dataclass

ErrorInfo(
    type_name: str,
    provider: str | None,
    phase: str,
    retryable: bool,
    http_status: int | None,
    detail: str,
)

A serializable, already-redacted snapshot of any AnyInfer error.

Captured from the exception at failure time, so attempt records and telemetry can carry the failure long after the exception itself has been handled.

Attributes:

Name	Type	Description
type_name	str	Class name of the exception this snapshot was captured from.
provider	str | None	Id of the provider involved, or None for provider-independent failures.
phase	str	Request-lifecycle stage that failed (configure, discover, generate, stream, validate, or cleanup).
retryable	bool	Whether retrying the identical request could plausibly succeed.
http_status	int | None	Status code, for failures that came from an HTTP response.
detail	str	Human-readable description, already redacted and capped at DETAIL_MAX_CHARS characters.

anyinfer.FinishReason module-attribute

FinishReason = Literal[
    "stop",
    "length",
    "tool_calls",
    "content_filter",
    "other",
]

Normalized reason a generation stopped.

Stream events

anyinfer.StreamEvent module-attribute

StreamEvent = (
    TextDelta
    | ReasoningDelta
    | ToolCallDelta
    | UsageUpdate
    | TimingMark
    | AttemptFailed
    | StreamEnded
)

Any event a consumer may observe.

anyinfer.TextDelta dataclass

TextDelta(text: str)

A fragment of visible answer text.

anyinfer.ReasoningDelta dataclass

ReasoningDelta(text: str)

A fragment of reasoning/thinking text, excluded from the answer text.

anyinfer.ToolCall dataclass

ToolCall(id: str, name: str, arguments: Mapping[str, Any])

A model's request to invoke a tool.

Attributes:

Name	Type	Description
id	str	Provider-assigned call id. Adapters synthesize "call_0", "call_1"… when the provider omits one, so downstream correlation always has a key.
name	str	The tool being called.
arguments	Mapping[str, Any]	Parsed JSON arguments. An unparseable argument payload yields {} and a warning on the Generation.

anyinfer.ToolCallDelta dataclass

ToolCallDelta(
    index: int,
    call_id: str | None,
    name: str | None,
    arguments_fragment: str,
)

A fragment of a tool call.

Fragments are correlated by index — the tool-call slot within the response. Concatenate arguments_fragment per index, then JSON-parse the result.

Attributes:

Name	Type	Description
index	int	The tool-call slot within the response this fragment belongs to.
call_id	str | None	Provider-assigned call id, on fragments that carry it.
name	str | None	Name of the tool being called, on fragments that carry it.
arguments_fragment	str	The next piece of this slot's JSON argument text; may be empty.

anyinfer.UsageUpdate dataclass

UsageUpdate(usage: Usage)

A usage report; may arrive mid-stream and more than once.

anyinfer.TimingMark dataclass

TimingMark(name: TimingMarkName, at_ms: float)

A centrally-measured timing point, in milliseconds since attempt start.

Attributes:

Name	Type	Description
name	TimingMarkName	Which point on the attempt clock this marks.
at_ms	float	Milliseconds elapsed since the attempt started.

anyinfer.TimingMarkName module-attribute

TimingMarkName = Literal['attempt_start', 'first_token']

Named points on the attempt clock.

anyinfer.AttemptFailed dataclass

AttemptFailed(record: AttemptRecord)

A target attempt failed; a retry or fallback may follow.

Attributes:

Name	Type	Description
record	AttemptRecord	The failed attempt's routing-trail entry: target, outcome, error snapshot, and any timing.

anyinfer.StreamEnded dataclass

StreamEnded(result: Generation)

Terminal event carrying the assembled result.