Embeddings and Reranking¶

Two stateless inference operations alongside generation: turning text into vectors, and ranking documents against a query. Both are typed and routed the same way generation is (target resolution, retries, fallback, usage, and telemetry), but neither is a field on GenerationRequest. See the embeddings concept page for the embedding-space safety rule that governs fallback.

Embedding¶

anyinfer.EmbeddingRequest `dataclass` ¶

EmbeddingRequest(
    inputs: tuple[str, ...],
    input_type: EmbeddingInputIntent | None = None,
    dimensions: int | None = None,
    expected_space: EmbeddingSpace | None = None,
    timeout_s: float | None = None,
    max_response_bytes: int = DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES,
    metadata: Mapping[str, str] = dict(),
    provider_options: Mapping[
        str, Mapping[str, Any]
    ] = dict(),
    batch: BatchPolicy = BatchPolicy(),
    retain_raw: bool = False,
    allow_incompatible_fallback: bool = False,
)

A request to embed one or more texts into vectors.

Attributes:

Name	Type	Description
`inputs`	`tuple[str, ...]`	Texts to embed, in order. Duplicates are preserved exactly — deduplication is never implicit, because it would change reported usage and may change provider-side billing or behavior.
`input_type`	`EmbeddingInputIntent \| None`	What the embedded text will be used for, when the target model distinguishes it. `None` means no intent is asserted.
`dimensions`	`int \| None`	Requested output dimensionality, for models supporting native dimensionality reduction. `None` means the model's default.
`expected_space`	`EmbeddingSpace \| None`	An `EmbeddingSpace` the caller expects the result to match. When set, a successful but incompatible provider response is rejected rather than returned, per the cross-space safety rule.
`timeout_s`	`float \| None`	Per-attempt wall-clock budget; `None` means `DEFAULT_TIMEOUT_S`.
`max_response_bytes`	`int`	Hard cap on one provider response body. Defaults to `DEFAULT_MAX_EMBEDDING_RESPONSE_BYTES` — vector batches are far larger than chat responses.
`metadata`	`Mapping[str, str]`	Caller-supplied, opaque request metadata carried through telemetry.
`provider_options`	`Mapping[str, Mapping[str, Any]]`	Escape hatch, namespaced by provider id, passed through verbatim to the matching adapter and consulted by no core logic.
`batch`	`BatchPolicy`	Core-owned batching policy for this request.
`retain_raw`	`bool`	Whether to keep the provider's raw response payload on the result.
`allow_incompatible_fallback`	`bool`	Explicit opt-in permitting fallback to a route target that cannot be proven to share the primary target's embedding space. Off by default because wrong-space vectors fail silently when compared; a result served through this opt-in always carries a warning naming both targets.

effective_timeout_s `property` ¶

effective_timeout_s: float

The timeout that applies, honoring the module default when unset.

__post_init__ ¶

__post_init__() -> None

Reject a request with no inputs.

Empty input is a local validation error performing no provider call — it is never sent, never billed, and never retried.

Raises:

Type	Description
`ValueError`	If `inputs` is empty.

anyinfer.EmbeddingResult `dataclass` ¶

EmbeddingResult(
    vectors: tuple[EmbeddingVector, ...],
    target: ResolvedTarget,
    space: EmbeddingSpace,
    usage: Usage,
    timing: Timing,
    attempts: tuple[AttemptRecord, ...] = (),
    warnings: tuple[str, ...] = (),
    raw: Any | None = None,
    manifest: Any | None = None,
)

The result of one embedding request.

Attributes:

Name	Type	Description
`vectors`	`tuple[EmbeddingVector, ...]`	Embedding vectors in the exact order of `EmbeddingRequest.inputs`.
`target`	`ResolvedTarget`	The provider and model that actually produced these vectors.
`space`	`EmbeddingSpace`	Identity of the vector space these vectors live in.
`usage`	`Usage`	Token/billing accounting, normalized across providers.
`timing`	`Timing`	Centrally-measured latency for the winning attempt.
`attempts`	`tuple[AttemptRecord, ...]`	The full routing trail, including failed and retried attempts.
`warnings`	`tuple[str, ...]`	Non-fatal notices accumulated along the way.
`raw`	`Any \| None`	The provider-native response payload, when the request asked to keep it.
`manifest`	`Any \| None`	The run manifest for this call, or `None` when manifests are off.

anyinfer.EmbeddingVector `dataclass` ¶

EmbeddingVector(values: tuple[float, ...])

One immutable embedding vector, validated on construction.

Attributes:

Name	Type	Description
`values`	`tuple[float, ...]`	The vector's components, in order.

__post_init__ ¶

__post_init__() -> None

Reject ragged, non-numeric, boolean, or non-finite vector data.

Raises:

Type	Description
`ValueError`	If `values` is empty, contains a `bool` (which is a `int` subclass in Python and would otherwise pass a numeric check silently), or contains a non-finite float (`NaN` or infinity).

len ¶

__len__() -> int

Number of dimensions.

anyinfer.EmbeddingSpace `dataclass` ¶

EmbeddingSpace(
    provider_id: str,
    model: str,
    model_revision: str | None = None,
    dimensions: int | None = None,
    input_intent_aware: bool = False,
    normalized: bool | None = None,
    compatibility_id: str | None = None,
)

Identity of the vector space one embedding call's output lives in.

Two embedding results are only safely comparable — for storage, search, or a fallback retry — when they came from the same space. This is the strongest identity AnyInfer can construct from what a provider tells it; it is deliberately not a guess when a provider tells us too little.

Attributes:

Name	Type	Description
`provider_id`	`str`	The provider that produced the vectors.
`model`	`str`	The concrete model id, verbatim as resolved (not an alias).
`model_revision`	`str \| None`	A pinned revision/snapshot identifier, when the provider or catalog exposes one; `None` when the model string is the only version signal available.
`dimensions`	`int \| None`	The vector length actually returned.
`input_intent_aware`	`bool`	Whether this model's output depends on the requested `EmbeddingInputIntent` — if `True`, a query embedded without the same intent handling as the documents it will be compared against is not safely comparable even within the same model.
`normalized`	`bool \| None`	Whether the provider states its vectors are unit-normalized; `None` when undocumented.
`compatibility_id`	`str \| None`	An application-supplied identifier asserting that this space is interchangeable with another sharing the same id. Never inferred by AnyInfer — a guessed equivalence is exactly what this type exists to refuse to produce. `None` means no compatibility claim has been made.

compatible_with ¶

compatible_with(other: EmbeddingSpace) -> bool

Whether a vector from other may be safely compared with one from this space.

True only when both spaces carry the same caller-asserted compatibility_id, or when provider, model, and revision are all identical. Matching dimensions alone is not sufficient — two different models can share a dimension count while encoding semantically incompatible spaces.

anyinfer.EmbeddingCapabilities `dataclass` ¶

EmbeddingCapabilities(
    dimensions: int | None = None,
    dimension_choices: tuple[int, ...] = (),
    max_batch_inputs: int | None = None,
    max_input_tokens: int | None = None,
    max_input_bytes: int | None = None,
    input_intents: tuple[EmbeddingInputIntent, ...] = (),
    normalized: bool | None = None,
)

What an embedding-capable model can do, sourced only from facts a provider states.

Every field mirrors the provenance discipline of ModelCapabilities: a field this provider does not document stays None rather than becoming a guess.

Attributes:

Name	Type	Description
`dimensions`	`int \| None`	The vector length this model produces, when fixed.
`dimension_choices`	`tuple[int, ...]`	Alternative dimensions this model supports via provider-native dimensionality reduction (e.g. Matryoshka-trained models), when documented.
`max_batch_inputs`	`int \| None`	Most inputs one request may carry, when the provider states a limit.
`max_input_tokens`	`int \| None`	Largest single input this model accepts, in tokens.
`max_input_bytes`	`int \| None`	Largest single input this model accepts, in bytes, when the provider bounds by bytes rather than tokens.
`input_intents`	`tuple[EmbeddingInputIntent, ...]`	Which `EmbeddingInputIntent` values this model distinguishes; empty when the model does not support the concept at all.
`normalized`	`bool \| None`	Whether output vectors are unit-normalized; `None` when undocumented.

overlay ¶

overlay(
    other: EmbeddingCapabilities,
) -> EmbeddingCapabilities

Layer other's known fields over this one; unknown never displaces known.

The assembly rule for these records mirrors ModelCapabilities.overlay without per-field provenance: they carry only provider-stated facts, so the later layer (config or catalog over static) wins wherever it actually says something.

anyinfer.EmbeddingInputIntent `module-attribute` ¶

EmbeddingInputIntent = Literal[
    "query", "document", "classification", "clustering"
]

What an embedded text will be used for, when the provider or model distinguishes it.

Several embedding models produce measurably better retrieval when a query and the documents it will be compared against are embedded with different instructions even though both pass through the same model. A provider that has no such distinction ignores this field entirely; one that requires it but received None degrades per its own documented default, recorded as a warning rather than silently substituted.

Reranking¶

anyinfer.RerankRequest `dataclass` ¶

RerankRequest(
    query: str,
    documents: tuple[RerankDocument, ...],
    top_n: int | None = None,
    timeout_s: float | None = None,
    max_response_bytes: int = DEFAULT_MAX_RERANK_RESPONSE_BYTES,
    metadata: Mapping[str, str] = dict(),
    provider_options: Mapping[
        str, Mapping[str, Any]
    ] = dict(),
    batch: BatchPolicy = BatchPolicy(),
    return_documents: bool = False,
    retain_raw: bool = False,
)

A request to rank documents by relevance to a query.

Attributes:

Name	Type	Description
`query`	`str`	The query text every document is scored against.
`documents`	`tuple[RerankDocument, ...]`	Documents to rank, in the caller's original order. Document ids must be unique within the request; duplicate text under distinct ids is permitted and preserved.
`top_n`	`int \| None`	Return only the top N ranked items. `None` returns every document ranked.
`timeout_s`	`float \| None`	Per-attempt wall-clock budget; `None` means `DEFAULT_TIMEOUT_S`.
`max_response_bytes`	`int`	Hard cap on one provider response body.
`metadata`	`Mapping[str, str]`	Caller-supplied, opaque request metadata carried through telemetry.
`provider_options`	`Mapping[str, Mapping[str, Any]]`	Escape hatch, namespaced by provider id.
`batch`	`BatchPolicy`	Core-owned batching policy for this request.
`return_documents`	`bool`	Whether the result should echo document text back on each `RankedItem`. Off by default — the caller already has the text it sent.
`retain_raw`	`bool`	Whether to keep the provider's raw response payload on the result.

effective_timeout_s `property` ¶

effective_timeout_s: float

The timeout that applies, honoring the module default when unset.

__post_init__ ¶

__post_init__() -> None

Reject a request with an empty query, no documents, or duplicate document ids.

Raises:

Type	Description
`ValueError`	If `query` is blank, `documents` is empty, `top_n` is not positive when set, or two documents share an id.

anyinfer.RerankResult `dataclass` ¶

RerankResult(
    items: tuple[RankedItem, ...],
    target: ResolvedTarget,
    usage: Usage,
    timing: Timing,
    attempts: tuple[AttemptRecord, ...] = (),
    warnings: tuple[str, ...] = (),
    raw: Any | None = None,
    manifest: Any | None = None,
)

The result of one rerank request.

Attributes:

Name	Type	Description
`items`	`tuple[RankedItem, ...]`	Ranked documents, ordered by descending relevance (or by whatever order the provider certifies as its ranking — always the intended reading order).
`target`	`ResolvedTarget`	The provider and model that actually produced this ranking.
`usage`	`Usage`	Token/billing accounting, normalized across providers.
`timing`	`Timing`	Centrally-measured latency for the winning attempt.
`attempts`	`tuple[AttemptRecord, ...]`	The full routing trail, including failed and retried attempts.
`warnings`	`tuple[str, ...]`	Non-fatal notices accumulated along the way.
`raw`	`Any \| None`	The provider-native response payload, when the request asked to keep it.
`manifest`	`Any \| None`	The run manifest for this call, or `None` when manifests are off.

anyinfer.RerankDocument `dataclass` ¶

RerankDocument(
    id: str, text: str, metadata: Mapping[str, str] = dict()
)

One document offered to a rerank request.

Attributes:

Name	Type	Description
`id`	`str`	Caller-owned opaque identifier, unique within one request. Never interpreted or generated by AnyInfer.
`text`	`str`	The document text sent to the provider.
`metadata`	`Mapping[str, str]`	Caller-owned metadata retained locally; never sent to a provider unless a provider option explicitly requests it.

anyinfer.RankedItem `dataclass` ¶

RankedItem(
    index: int,
    document_id: str,
    score: float,
    text: str | None = None,
)

One document's position and score in a rerank result.

Attributes:

Name	Type	Description
`index`	`int`	The document's position in `RerankRequest.documents`, preserved so callers can recover order or metadata without a lookup.
`document_id`	`str`	The caller-supplied id of the ranked document.
`score`	`float`	The provider's relevance score. Finite by construction; meaningful only within the result produced by the same target and not comparable across different providers or models.
`text`	`str \| None`	The document's text, present only when the request asked for it.

__post_init__ ¶

__post_init__() -> None

Reject a non-finite score, a negative index, or a blank document id.

Raises:

Type	Description
`ValueError`	If `score` is `NaN`/infinite, `index` is negative, or `document_id` is empty.

anyinfer.RerankCapabilities `dataclass` ¶

RerankCapabilities(
    max_documents: int | None = None,
    max_tokens_per_document: int | None = None,
    max_bytes_per_document: int | None = None,
    native_top_n: bool = False,
)

What a reranking-capable model can do, sourced only from facts a provider states.

Attributes:

Name	Type	Description
`max_documents`	`int \| None`	Most documents one request may carry, when the provider states a limit.
`max_tokens_per_document`	`int \| None`	Largest single document this model accepts, in tokens.
`max_bytes_per_document`	`int \| None`	Largest single document this model accepts, in bytes.
`native_top_n`	`bool`	Whether the provider accepts `RerankRequest.top_n` natively rather than the core truncating a full ranking locally.

overlay ¶

overlay(other: RerankCapabilities) -> RerankCapabilities

Layer other's known fields over this one; unknown never displaces known.

Shared¶

anyinfer.InferenceOperation `module-attribute` ¶

InferenceOperation = Literal[
    "generation", "embedding", "rerank"
]

The inference operations AnyInfer models as first-class primitives.

anyinfer.BatchPolicy `dataclass` ¶

BatchPolicy(
    max_concurrency: int = 4,
    allow_split: bool = True,
    rerank_cross_batch: bool = False,
    max_items_override: int | None = None,
)

Core-owned policy for splitting a request that exceeds a provider's verified limit.

Batching is centralized policy, not adapter behavior — providers disagree on maximum inputs, documents, tokens, and bytes, and an adapter never decides how to split a request; it only ever sees one already-sized wire call.

Attributes:

Name	Type	Description
`max_concurrency`	`int`	Most internal batches dispatched concurrently for one request.
`allow_split`	`bool`	Whether the core may split this request across multiple provider calls at all. `False` means a request exceeding the resolved limit fails locally rather than being silently divided.
`rerank_cross_batch`	`bool`	Whether reranking may be split across documents when the provider offers no documented globally-comparable batch contract. Off by default because concatenating scores from separate rerank calls is not a valid global ordering unless the provider says otherwise.
`max_items_override`	`int \| None`	Caller-supplied ceiling on items per provider call — inputs for embedding, documents for reranking. Beats any provider-declared limit; useful when a provider misbehaves below its documented maximum, or to enable splitting against a target with no verified limit. `None` defers to the resolved target's verified capability.

__post_init__ ¶

__post_init__() -> None

Reject a non-positive concurrency bound or item ceiling.

Raises:

Type	Description
`ValueError`	If `max_concurrency` or `max_items_override` is less than 1.

Embeddings and Reranking¶

Embedding¶

anyinfer.EmbeddingRequest dataclass ¶

effective_timeout_s property ¶

__post_init__ ¶

anyinfer.EmbeddingResult dataclass ¶

anyinfer.EmbeddingVector dataclass ¶

__post_init__ ¶

__len__ ¶

anyinfer.EmbeddingSpace dataclass ¶

compatible_with ¶

anyinfer.EmbeddingCapabilities dataclass ¶

overlay ¶

anyinfer.EmbeddingInputIntent module-attribute ¶

Reranking¶

anyinfer.RerankRequest dataclass ¶

effective_timeout_s property ¶

__post_init__ ¶

anyinfer.RerankResult dataclass ¶

anyinfer.RerankDocument dataclass ¶

anyinfer.RankedItem dataclass ¶

__post_init__ ¶

anyinfer.RerankCapabilities dataclass ¶

overlay ¶

Shared¶

anyinfer.InferenceOperation module-attribute ¶

anyinfer.BatchPolicy dataclass ¶

__post_init__ ¶

anyinfer.EmbeddingRequest `dataclass` ¶

effective_timeout_s `property` ¶

anyinfer.EmbeddingResult `dataclass` ¶

anyinfer.EmbeddingVector `dataclass` ¶

len ¶

anyinfer.EmbeddingSpace `dataclass` ¶

anyinfer.EmbeddingCapabilities `dataclass` ¶

anyinfer.EmbeddingInputIntent `module-attribute` ¶

anyinfer.RerankRequest `dataclass` ¶

effective_timeout_s `property` ¶

anyinfer.RerankResult `dataclass` ¶

anyinfer.RerankDocument `dataclass` ¶

anyinfer.RankedItem `dataclass` ¶

anyinfer.RerankCapabilities `dataclass` ¶

anyinfer.InferenceOperation `module-attribute` ¶

anyinfer.BatchPolicy `dataclass` ¶