Requests and messages

The request side of the one primitive: everything a GenerationRequest can carry. See the event stream for how a request becomes output.

anyinfer.GenerationRequest dataclass

GenerationRequest(
    messages: tuple[Message, ...],
    schema: SchemaSpec | None = None,
    tools: tuple[ToolSpec, ...] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling = Sampling(),
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    max_response_bytes: int = DEFAULT_MAX_RESPONSE_BYTES,
    repair: Repair | None = None,
    provider_options: Mapping[
        str, Mapping[str, Any]
    ] = dict(),
    metadata: Mapping[str, str] = dict(),
)

A fully-specified generation request, independent of any provider.

This type is a deliberate superset of the OpenAI chat-completions request surface so the serve frontend stays a lossless codec.

provider_options is the escape hatch for provider-specific parameters: keys are provider ids, values are mappings passed to that provider's adapter verbatim. The special key "*" applies to whichever provider ends up serving the request, with a provider-specific namespace winning field-by-field over the wildcard.

Attributes:

Name	Type	Description
messages	tuple[Message, ...]	The conversation so far, oldest turn first.
schema	SchemaSpec | None	Structured-output contract to enforce, when one was requested.
tools	tuple[ToolSpec, ...]	Tools the model is allowed to call.
tool_choice	ToolChoice	Whether tool use is automatic, forbidden, required, or pinned to one named tool.
sampling	Sampling	Sampling controls; unset fields fall through to provider defaults.
reasoning	ReasoningEffort | None	Requested reasoning effort, for models that support it.
timeout_s	float | None	Per-attempt wall-clock budget; None means DEFAULT_TIMEOUT_S.
max_response_bytes	int	Cap on total streamed response bytes per attempt.
repair	Repair | None	Budget for schema-repair round-trips; None disables repair.
provider_options	Mapping[str, Mapping[str, Any]]	Provider-specific parameters keyed by provider id, passed to the serving adapter verbatim (see above for the "*" wildcard rule).
metadata	Mapping[str, str]	Opaque caller-supplied labels, carried through unchanged and echoed in telemetry.

effective_timeout_s property

effective_timeout_s: float

The per-attempt timeout, resolving None to the default.

with_messages

with_messages(
    messages: Sequence[Message],
) -> GenerationRequest

Return a copy of this request with different messages.

Used by the repair loop and the tool loop, which extend the conversation without touching any other request field.

anyinfer.Message dataclass

Message(role: Role, content: tuple[ContentPart, ...])

One turn in a conversation.

Attributes:

Name	Type	Description
role	Role	Who authored the turn.
content	tuple[ContentPart, ...]	Ordered parts making up the turn — text runs, tool calls, and tool results.

text property

text: str

Concatenation of this message's Text parts.

anyinfer.Role module-attribute

Role = Literal['system', 'user', 'assistant', 'tool']

Who authored a message.

anyinfer.ContentPart module-attribute

ContentPart = Text | ToolCall | ToolResult

A single piece of message content.

anyinfer.Text dataclass

Text(text: str)

A run of plain text within a message.

anyinfer.system

system(text: str) -> Message

Build a system message from plain text.

anyinfer.user

user(text: str) -> Message

Build a user message from plain text.

anyinfer.assistant

assistant(text: str) -> Message

Build an assistant message from plain text.

anyinfer.Sampling dataclass

Sampling(
    temperature: float | None = None,
    top_p: float | None = None,
    max_output_tokens: int | None = None,
    stop: tuple[str, ...] = (),
)

Sampling controls.

Every field defaults to None/empty meaning provider default. AnyInfer never invents a temperature: an unset value is omitted from the wire request entirely.

Attributes:

Name	Type	Description
temperature	float | None	Randomness of token selection; higher values sample more freely.
top_p	float | None	Nucleus-sampling cutoff — the probability mass considered for each token.
max_output_tokens	int | None	Upper bound on how many tokens the model may generate.
stop	tuple[str, ...]	Sequences that end generation as soon as one is produced.

anyinfer.ReasoningEffort module-attribute

ReasoningEffort = Literal[
    "minimal", "low", "medium", "high"
]

Normalized reasoning effort; each descriptor translates it to its provider's wire form.

anyinfer.SchemaSpec dataclass

SchemaSpec(
    json_schema: Mapping[str, Any], name: str = "response"
)

A structured-output contract.

The json_schema here is the canonical schema. Providers may receive a projected variant on the wire, but responses are always validated against this one.

Attributes:

Name	Type	Description
json_schema	Mapping[str, Any]	The canonical JSON Schema every response is validated against.
name	str	Label for the schema, passed to providers whose wire format names the expected response shape.

coerce classmethod

coerce(
    obj: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any],
) -> SchemaSpec

Accept a SchemaSpec, a JSON-schema mapping, or a pydantic-style model.

Parameters:

Name	Type	Description	Default
obj	SchemaSpec | SupportsJSONSchema | Mapping[str, Any]	The schema in any accepted form.	required

Returns:

Type	Description
SchemaSpec	An equivalent SchemaSpec.

Raises:

Type	Description
TypeError	If obj is none of the accepted forms.

anyinfer.SupportsJSONSchema

Bases: Protocol

Duck type for pydantic-style models supplied as schemas.

model_json_schema

model_json_schema() -> Mapping[str, Any]

Return the JSON Schema describing this model.

anyinfer.Repair dataclass

Repair(max_attempts: int = 1)

Opt-in bounded repair budget for schema violations.

Attributes:

Name	Type	Description
max_attempts	int	Maximum repair round-trips after a schema violation before the violation is surfaced as an error.

anyinfer.ToolSpec dataclass

ToolSpec(
    name: str,
    description: str,
    parameters: Mapping[str, Any],
)

A tool the model may call.

Attributes:

Name	Type	Description
name	str	Identifier the model uses to invoke the tool.
description	str	What the tool does, shown to the model to guide when to call it.
parameters	Mapping[str, Any]	JSON Schema describing the tool's arguments.

anyinfer.ToolChoice module-attribute

ToolChoice = Literal['auto', 'none', 'required'] | str

"auto" | "none" | "required", or a specific tool name.

anyinfer.ToolResult dataclass

ToolResult(
    call_id: str, content: str, is_error: bool = False
)

The outcome of executing a tool call, fed back to the model.

Attributes:

Name	Type	Description
call_id	str	Id of the ToolCall this result answers.
content	str	The tool's output, rendered as text for the model to read.
is_error	bool	Whether the tool failed; the content then describes the failure.

anyinfer.Target module-attribute

Target = str

Where a request should go.

Grammar::

Target  = ALIAS | PROVIDER ":" MODEL
PROVIDER = [a-z0-9-]+      (after normalization: lowercased, stripped, "_" -> "-")
MODEL    = the remainder verbatim

Split on the first colon only — model ids may themselves contain colons ("ollama:qwen3:8b"). A target without a colon must match a catalog alias.