Requests and Messages

The request side of the one primitive: everything a GenerationRequest can carry. See the event stream for how a request becomes output.

anyinfer.GenerationRequest dataclass

GenerationRequest(
    messages: tuple[Message, ...],
    schema: SchemaSpec | None = None,
    tools: tuple[ToolSpec, ...] = (),
    tool_choice: ToolChoice = "auto",
    sampling: Sampling = Sampling(),
    reasoning: ReasoningEffort | None = None,
    timeout_s: float | None = None,
    max_response_bytes: int = DEFAULT_MAX_RESPONSE_BYTES,
    max_input_part_bytes: int = DEFAULT_MAX_INPUT_PART_BYTES,
    max_input_bytes: int = DEFAULT_MAX_INPUT_BYTES,
    repair: Repair | None = None,
    provider_options: Mapping[
        str, Mapping[str, Any]
    ] = dict(),
    metadata: Mapping[str, str] = dict(),
    history: HistoryPolicy | None = None,
    cache: CachePolicy | None = None,
    arena: ArenaPolicy | None = None,
    context: ContextRequest | None = None,
    logprobs: int | None = None,
    cite_documents: bool = False,
    server_tools: tuple[ServerToolSpec, ...] = (),
)

A fully-specified generation request, independent of any provider.

This type is a deliberate superset of the OpenAI chat-completions request surface so the serve frontend stays a lossless codec.

provider_options is the escape hatch for provider-specific parameters: keys are provider ids, values are mappings passed to that provider's adapter verbatim. The special key "*" applies to whichever provider ends up serving the request, with a provider-specific namespace winning field-by-field over the wildcard.

Attributes:

Name	Type	Description
messages	tuple[Message, ...]	The conversation so far, oldest turn first.
schema	SchemaSpec | None	Structured-output contract to enforce, when one was requested.
tools	tuple[ToolSpec, ...]	Tools the model is allowed to call.
tool_choice	ToolChoice	Whether tool use is automatic, forbidden, required, or pinned to one named tool.
sampling	Sampling	Sampling controls; unset fields fall through to provider defaults.
reasoning	ReasoningEffort | None	Requested reasoning effort, for models that support it.
timeout_s	float | None	Per-attempt wall-clock budget; None means DEFAULT_TIMEOUT_S.
max_response_bytes	int	Cap on total streamed response bytes per attempt.
repair	Repair | None	Budget for schema-repair round-trips; None disables repair.
provider_options	Mapping[str, Mapping[str, Any]]	Provider-specific parameters keyed by provider id, passed to the serving adapter verbatim (see above for the "*" wildcard rule).
metadata	Mapping[str, str]	Opaque caller-supplied labels, carried through unchanged and echoed in telemetry.
history	HistoryPolicy | None	Per-request conversation-compaction policy, overriding the client's default. None means "use the client's". Carried on the request rather than passed alongside it so the serve frontend stays a lossless codec: an anyinfer_history object on the wire decodes into exactly this field.
cache	CachePolicy | None	Per-request prompt-cache placement, overriding the client's default. None — the default — means no placement at all, since caching changes what a provider bills. Carried on the request for the same codec reason as history: an anyinfer_cache object on the wire decodes into this field.
arena	ArenaPolicy | None	Per-request fixed fan-out policy, overriding the client's default. Adapters never see it; the client runs and selects every candidate before projection.
context	ContextRequest | None	Explicit caller-approved documents to reduce for the resolved target. None preserves ordinary generation byte-for-byte.
logprobs	int | None	How many alternative tokens to report a log-probability for at each generated position. None — the default — asks for none, and is the only value that costs nothing: every provider that returns logprobs inflates the response with them. 0 asks for the chosen token's own probability and no alternatives, which is what a confidence-scoring caller needs; a positive value additionally asks for that many runners-up. A target known not to report them emits a dropped-parameter event rather than returning an empty logprobs a caller would have to notice for themselves.
server_tools	tuple[ServerToolSpec, ...]	Capabilities the provider should run itself during this generation — web search, code execution. Empty by default and never inferred: every one is billed per invocation, and a request that quietly started searching would surprise a caller on their bill rather than in their output. A target whose capabilities trustedly lack one refuses before dispatch, since an answer produced without the search the caller asked for is a different answer, not a degraded one.
cite_documents	bool	Ask the target to attribute its answer to the documents this request supplied. Every dialect that can do this treats it as a request-side opt-in — a model does not volunteer citations — and several bill differently for a cited answer, so it is off by default and never inferred from the mere presence of a DocumentPart. A target that cannot cite reports a dropped parameter; the answer still arrives, without attributions.

effective_timeout_s property

effective_timeout_s: float

The per-attempt timeout, resolving None to the default.

__post_init__

__post_init__() -> None

Enforce multimodal request byte ceilings before any adapter can run.

with_messages

with_messages(
    messages: Sequence[Message],
) -> GenerationRequest

Return a copy of this request with different messages.

Used by the repair loop and the tool loop, which extend the conversation without touching any other request field.

anyinfer.Message dataclass

Message(role: Role, content: tuple[ContentPart, ...])

One turn in a conversation.

Attributes:

Name	Type	Description
role	Role	Who authored the turn.
content	tuple[ContentPart, ...]	Ordered parts making up the turn — text runs, tool calls, and tool results.

text property

text: str

Concatenation of this message's Text parts.

anyinfer.Role module-attribute

Role = Literal['system', 'user', 'assistant', 'tool']

Who authored a message.

anyinfer.ContentPart module-attribute

ContentPart = (
    Text
    | ToolCall
    | ToolResult
    | ImagePart
    | DocumentPart
    | AudioPart
    | VideoPart
)

A single piece of message content.

anyinfer.Text dataclass

Text(text: str)

A run of plain text within a message.

anyinfer.ImagePart dataclass

ImagePart(
    data: bytes | None = None,
    url: str | None = None,
    media_type: str = "image/png",
    detail: Literal["auto", "low", "high"] | None = None,
)

An image supplied inline or by remote URL for a multimodal model.

__post_init__

__post_init__() -> None

Require exactly one source and an image media type.

anyinfer.DocumentPart dataclass

DocumentPart(
    data: bytes | None = None,
    url: str | None = None,
    media_type: str = "application/pdf",
    filename: str | None = None,
)

A document supplied inline or by remote URL for a capable model.

__post_init__

__post_init__() -> None

Require exactly one source and a non-empty media type.

anyinfer.AudioPart dataclass

AudioPart(data: bytes, media_type: str = 'audio/wav')

Inline audio input for a multimodal model.

__post_init__

__post_init__() -> None

Require bytes and an audio media type.

anyinfer.VideoPart dataclass

VideoPart(
    data: bytes | None = None,
    url: str | None = None,
    media_type: str = "video/mp4",
    start_offset_s: float | None = None,
    end_offset_s: float | None = None,
    fps: float | None = None,
)

A video supplied inline or by provider-hosted URI for a capable model.

Unlike AudioPart, a URL form is not a convenience here — it is the normal path. Video is the one modality whose realistic payloads dwarf a request body: the providers that accept it publish an upload endpoint and expect a URI back, and one of them also accepts a public video URL directly. Inline bytes are supported for short clips and are bounded by the same per-part and per-request ceilings as every other inline payload (GenerationRequest.max_input_part_bytes), which is deliberately not raised for video: a ceiling quietly loosened for one modality is a ceiling that no longer means what its name says. A caller sending a large clip inline raises the request's own ceiling and thereby says so.

Attributes:

Name	Type	Description
data	bytes | None	Inline video bytes, or None when the video is referenced by URI.
url	str | None	Provider-hosted URI or public video URL, or None for inline bytes.
media_type	str	IANA media type; must be a video/* subtype.
start_offset_s	float | None	Where to begin reading the video, in seconds. None means the beginning. Clipping is a request parameter rather than something the caller does with a decoder, because the providers bill for the frames they sample and the clip bounds are what decide how many there are.
end_offset_s	float | None	Where to stop reading, in seconds; None means the end.
fps	float | None	Frames per second to sample at. None leaves the provider's own default, which is what almost every caller wants — the field exists because raising it for a fast-moving clip, or lowering it for a screen recording, changes both the answer and the bill.

__post_init__

__post_init__() -> None

Require exactly one source, a video media type, and a sane clip window.

Raises:

Type	Description
ValueError	If both or neither source is given, the media type is not video/*, an offset is negative, the window ends before it starts, or the frame rate is not positive.

anyinfer.system

system(text: str) -> Message

Build a system message from plain text.

anyinfer.user

user(text: str) -> Message

Build a user message from plain text.

anyinfer.assistant

assistant(text: str) -> Message

Build an assistant message from plain text.

anyinfer.Sampling dataclass

Sampling(
    temperature: float | None = None,
    top_p: float | None = None,
    max_output_tokens: int | None = None,
    stop: tuple[str, ...] = (),
    seed: int | None = None,
    presence_penalty: float | None = None,
    frequency_penalty: float | None = None,
)

Sampling controls.

Every field defaults to None/empty meaning provider default. AnyInfer never invents a temperature: an unset value is omitted from the wire request entirely.

Attributes:

Name	Type	Description
temperature	float | None	Randomness of token selection; higher values sample more freely.
top_p	float | None	Nucleus-sampling cutoff — the probability mass considered for each token.
max_output_tokens	int | None	Upper bound on how many tokens the model may generate.
stop	tuple[str, ...]	Sequences that end generation as soon as one is produced.
seed	int | None	Requested sampling seed. A provider that honors it makes repeated identical requests more likely to produce identical output — more likely, never guaranteed: every provider that ships this field documents it as best-effort, and none of them promise reproducibility across model or backend revisions. Each descriptor spells it in its own dialect (seed, random_seed); targets that have no such field report a dropped parameter rather than silently sampling freely.
presence_penalty	float | None	Penalty applied to tokens that have already appeared at all, discouraging repeated topics. Provider scales differ; the value is passed through unchanged rather than rescaled, because a rescaled penalty is a number the caller cannot reason about.
frequency_penalty	float | None	Penalty scaled by how often a token has already appeared, discouraging verbatim repetition.

__post_init__

__post_init__() -> None

Reject sampling values no provider could act on.

Raises:

Type	Description
ValueError	If the seed is negative, or a penalty is not finite.

anyinfer.ReasoningEffort module-attribute

ReasoningEffort = Literal[
    "none", "minimal", "low", "medium", "high"
]

Normalized reasoning effort; each descriptor translates it to its provider's wire form.

none asks for reasoning to be disabled, and is distinct from both minimal and from leaving the field unset. minimal means "think as little as you can", none means "do not think", and None means "whatever this model does by default" — three different requests that produce three different wire forms. The level exists because OpenAI's own vocabulary accepts it on current models, and a sidecar caller sending it against an OpenAI backend must not be refused for using the dialect the gateway claims.

Not every provider can express it. Where a provider publishes a reasoning enum with no off value, the descriptor omits the field rather than substituting a level the caller did not ask for; each ReasoningTranslator documents its own choice.

anyinfer.SchemaSpec dataclass

SchemaSpec(
    json_schema: Mapping[str, Any], name: str = "response"
)

A structured-output contract.

The json_schema here is the canonical schema. Providers may receive a projected variant on the wire, but responses are always validated against this one.

Attributes:

Name	Type	Description
json_schema	Mapping[str, Any]	The canonical JSON Schema every response is validated against.
name	str	Label for the schema, passed to providers whose wire format names the expected response shape.

coerce classmethod

coerce(
    obj: SchemaSpec
    | SupportsJSONSchema
    | Mapping[str, Any],
) -> SchemaSpec

Accept a SchemaSpec, a JSON-schema mapping, or a pydantic-style model.

Parameters:

Name	Type	Description	Default
obj	SchemaSpec | SupportsJSONSchema | Mapping[str, Any]	The schema in any accepted form.	required

Returns:

Type	Description
SchemaSpec	An equivalent SchemaSpec.

Raises:

Type	Description
TypeError	If obj is none of the accepted forms.

anyinfer.SupportsJSONSchema

Bases: Protocol

Duck type for pydantic-style models supplied as schemas.

model_json_schema

model_json_schema() -> Mapping[str, Any]

Return the JSON Schema describing this model.

anyinfer.Repair dataclass

Repair(max_attempts: int = 1)

Opt-in bounded repair budget for schema violations.

Attributes:

Name	Type	Description
max_attempts	int	Maximum repair round-trips after a schema violation before the violation is surfaced as an error.

anyinfer.ArenaPolicy dataclass

ArenaPolicy(
    targets: tuple[str, ...],
    strategy: Literal[
        "first_valid",
        "consensus",
        "cheapest",
        "fastest",
        "judge",
        "synthesize",
    ] = "first_valid",
    judge_target: str | None = None,
    instructions: str | None = None,
    concurrency: int = 4,
    min_candidates: int = 1,
    reveal_targets: bool = False,
    memoize_tools: Literal[
        "read_only", "all", "opt_in", "off"
    ] = "read_only",
)

Fan one request out to fixed targets and select after every branch finishes.

__post_init__

__post_init__() -> None

Reject an arena whose fixed bounds or strategy are not executable.

anyinfer.ContextRequest dataclass

ContextRequest(
    documents: tuple[ContextDocument, ...],
    query: str | None = None,
    strategy: str = "auto",
    max_tokens: int | None = None,
    placement: Literal["system", "prepend_user"] = "system",
    tuning: ContextTuning = DEFAULT_TUNING,
    max_request_documents: int = DEFAULT_REQUEST_DOCUMENTS,
    max_request_bytes: int = DEFAULT_REQUEST_BYTES,
)

Documents explicitly approved by the caller for stateless reduction.

__post_init__

__post_init__() -> None

Reject inference-spending strategies and oversized request payloads.

anyinfer.ToolSpec dataclass

ToolSpec(
    name: str,
    description: str,
    parameters: Mapping[str, Any],
    annotations: ToolAnnotations = ToolAnnotations(),
)

A tool the model may call.

Attributes:

Name	Type	Description
name	str	Identifier the model uses to invoke the tool.
description	str	What the tool does, shown to the model to guide when to call it.
parameters	Mapping[str, Any]	JSON Schema describing the tool's arguments.
annotations	ToolAnnotations	Untrusted behavioural hints from whatever declared the tool. Empty for tools declared in Python, where the code is its own description.

anyinfer.ServerToolSpec dataclass

ServerToolSpec(
    kind: ServerToolKind, max_uses: int | None = None
)

A capability the provider executes itself during one generation.

Distinct from ToolSpec in the one way that matters: nothing comes back to the caller to run. The provider searches, or executes code, inside the same request and folds the result into its own answer — so there is no tool-call loop, no arguments to parse, and no result to feed back.

Kept to a normalized kind plus a use ceiling rather than a per-provider option bag. Each provider spells these very differently (a dated tool type, a bare marker object, a container spec), and a spec that carried every provider's knobs would be the per-engine branch this library exists to remove. Provider-specific tuning stays in provider_options, where it is visibly provider-specific.

Attributes:

Name	Type	Description
kind	ServerToolKind	Which capability to enable.
max_uses	int | None	Most invocations the provider may make within one generation, or None for the provider's own default. Worth having typed because these are billed per invocation: a search tool with no ceiling is an unbounded line item on a request the caller thought was fixed-price.

__post_init__

__post_init__() -> None

Reject a ceiling no provider could act on.

Raises:

Type	Description
ValueError	The kind is unknown, or the use ceiling is not positive.

anyinfer.ServerToolKind module-attribute

ServerToolKind = Literal['web_search', 'code_execution']

A tool the provider runs, rather than one the caller runs.

Only two, and deliberately so: these are the capabilities several providers execute server-side inside a single request/response, which is squarely translate-only territory. Anything requiring the caller to run code, hold state between turns, or plan is a client-executed ToolSpec — or an agent framework, which this library is explicitly not.

anyinfer.ToolChoice module-attribute

ToolChoice = Literal['auto', 'none', 'required'] | str

"auto" | "none" | "required", or a specific tool name.

anyinfer.ToolResult dataclass

ToolResult(
    call_id: str, content: str, is_error: bool = False
)

The outcome of executing a tool call, fed back to the model.

Attributes:

Name	Type	Description
call_id	str	Id of the ToolCall this result answers.
content	str	The tool's output, rendered as text for the model to read.
is_error	bool	Whether the tool failed; the content then describes the failure.

anyinfer.Target module-attribute

Target = str

Where a request should go.

Grammar::

Target  = ALIAS | PROVIDER ":" MODEL
PROVIDER = [a-z0-9-]+      (after normalization: lowercased, stripped, "_" -> "-")
MODEL    = the remainder verbatim

Split on the first colon only — model ids may themselves contain colons ("ollama:qwen3:8b"). A target without a colon must match a catalog alias.

Prompt Caching

Opt-in placement of a provider's prompt cache. Off unless asked for: caching changes what a provider bills and how long it keeps a copy of the prompt. What it caches is the prefix the caller sends, on the provider's side; it never skips a call or reuses an answer.

anyinfer.CachePolicy dataclass

CachePolicy(
    mode: CacheMode = "auto",
    min_segment_tokens: int = 1024,
    max_marks: int = 4,
    include_tools: bool = True,
    include_system: bool = True,
)

Opt-in prompt-cache placement for one request.

Off unless asked for. Caching changes what a provider bills and how long it retains a copy of the prompt, and neither is a decision this library makes on a caller's behalf — a request that carries no policy is cached exactly as much as it was before this existed, which is not at all.

What it caches is the prefix you send, on the provider's side, for the provider's retention window. It never skips a call and never reuses an answer.

Attributes:

Name	Type	Description
mode	CacheMode	off disables placement; auto uses the strongest available mechanism; explicit requires per-segment marks and reports a dropped parameter when the target has none.
min_segment_tokens	int	Segments estimated smaller than this are not worth marking — below a provider's own floor a mark is billed as a cache write that no later read ever amortizes.
max_marks	int	Most marks to place, clamped down to whatever the provider accepts.
include_tools	bool	Whether tool declarations may be marked. They are stable across a conversation and often large, so they are usually the best single mark.
include_system	bool	Whether the system block may be marked.

active property

active: bool

Whether this policy will attempt any placement.

__post_init__

__post_init__() -> None

Reject a policy that cannot be applied.

Raises:

Type	Description
ValueError	On an unknown mode, a negative floor, or a non-positive mark budget.

anyinfer.CacheMode module-attribute

CacheMode = Literal['off', 'auto', 'explicit']

What a request asks for.

auto uses the strongest mechanism the target offers. explicit asks for marks and reports a dropped parameter if the target has none, which is the setting for a caller who would rather know than silently get nothing.

anyinfer.CacheMechanism module-attribute

CacheMechanism = Literal['explicit', 'implicit']

How a provider's prompt cache is engaged.

explicit The provider accepts per-segment cache marks on the wire, so the core decides where the cacheable prefix ends and the adapter spells that mark. implicit The provider caches stable prefixes on its own. There is nothing to send; the core's only duty is to leave the prefix undisturbed and to notice when the caller's own request defeats it.