Local Inference

The anyinfer.local subsystem: hardware detection, backend selection, runtime acquisition, tuning, fit classification, model acquisition and storage, server supervision, and hardware→tier recommendation. Concepts: the local subsystem · the model catalog · the model catalog · guides: run a model locally · run a model locally.

from anyinfer import local

Hardware

anyinfer.local.detect

detect(*, use_cache: bool = True) -> HardwareProfile

Detect this machine's hardware.

Never raises: anything that could not be determined becomes a warning and a None field. Callers treat the result as advice, not as fact.

Parameters:

Name	Type	Description	Default
use_cache	bool	Read and write the disk cache. Overridden by CACHE_BYPASS_ENV and CACHE_REFRESH_ENV.	True

Returns:

Type	Description
HardwareProfile	The detected profile.

anyinfer.local.HardwareProfile dataclass

HardwareProfile(
    os_name: str,
    arch: str,
    total_ram_bytes: int | None = None,
    available_ram_bytes: int | None = None,
    cpu_name: str | None = None,
    physical_cores: int | None = None,
    logical_cores: int | None = None,
    accelerators: tuple[Accelerator, ...] = (),
    warnings: tuple[str, ...] = (),
    detected_at: float = 0.0,
)

What we could learn about this machine.

Every field may be None: absence means "not determined", never "zero".

Attributes:

Name	Type	Description
os_name	str	"windows", "linux", "darwin", or the raw platform string.
arch	str	Machine architecture as reported by platform.machine().
total_ram_bytes	int | None	Physical RAM.
available_ram_bytes	int | None	RAM currently free, when the platform reports it.
cpu_name	str | None	Processor model string.
physical_cores	int | None	Physical core count, preferred for thread tuning.
logical_cores	int | None	Logical processor count.
accelerators	tuple[Accelerator, ...]	Detected accelerators, strongest first.
warnings	tuple[str, ...]	Everything that could not be determined, and why.
detected_at	float	Unix timestamp of the probe, for cache display.

primary_accelerator property

primary_accelerator: Accelerator | None

The accelerator a server should target, or None for CPU-only.

has_accelerator property

has_accelerator: bool

Whether any non-CPU accelerator was detected.

total_vram_bytes property

total_vram_bytes: int | None

Total memory of the primary accelerator, when known.

user_supplied property

user_supplied: bool

Whether this profile came from from_user_input rather than a probe.

from_user_input classmethod

from_user_input(
    *,
    ram_gb: float | None = None,
    vram_gb: float | None = None,
    accelerator: AcceleratorKind | None = None,
    accelerator_name: str | None = None,
    compute_capability: str | None = None,
    os_name: str = "",
    arch: str = "",
) -> HardwareProfile

Build a profile from specs a person supplied, in familiar units.

The remote-Ollama case: local probing describes the wrong machine, and no Ollama API reports its host's specs, so asking the user is the only honest source. Values arrive in gigabytes because that is what a user reads off a spec sheet; anything left out stays None and keeps its "not determined" meaning.

The profile is marked as self-reported in warnings, so any advice derived from it can say so.

to_json

to_json() -> dict[str, Any]

Serialize for the disk cache.

from_json classmethod

from_json(data: dict[str, Any]) -> HardwareProfile

Deserialize from the disk cache.

anyinfer.local.Accelerator dataclass

Accelerator(
    kind: AcceleratorKind,
    name: str | None = None,
    total_vram_bytes: int | None = None,
    free_vram_bytes: int | None = None,
    unified_memory: bool = False,
    compute_capability: str | None = None,
    driver_version: str | None = None,
)

One detected accelerator.

Attributes:

Name	Type	Description
kind	AcceleratorKind	Which runtime family can drive it.
name	str | None	Human-readable device name, when reported.
total_vram_bytes	int | None	Total device memory, or None when unknown.
free_vram_bytes	int | None	Free device memory at probe time, or None.
unified_memory	bool	True when device memory is shared with system RAM (Apple Silicon), which makes VRAM budgeting a different calculation entirely.
compute_capability	str | None	NVIDIA compute capability as reported by the driver ("8.9"), or None. Quantized kernels gate on this — FP8 needs 8.9, the Marlin GPTQ kernel needs 8.0, and an unknown capability must exclude a gated variant rather than optimistically permit it.
driver_version	str | None	Vendor driver version string, when reported. Used to check that a downloadable CUDA runtime's toolkit version is supported before installing it.

compute_capability_value property

compute_capability_value: float | None

compute_capability as a comparable number, or None when unparseable.

driver_major property

driver_major: int | None

The major component of driver_version, or None.

anyinfer.local.AcceleratorKind module-attribute

AcceleratorKind = Literal[
    "cuda", "rocm", "metal", "vulkan", "cpu"
]

Accelerator families we can detect and target.

anyinfer.local.probe_signature

probe_signature() -> str

Fingerprint the probe tooling, so the cache invalidates when it changes.

Keyed on the resolved path and mtime of each probe executable plus the interpreter's platform: installing a GPU driver, or moving to different hardware, changes this.

anyinfer.local.cache_path

cache_path() -> Path

Where the detection cache lives.

anyinfer.local.CACHE_BYPASS_ENV module-attribute

CACHE_BYPASS_ENV = 'ANYINFER_HARDWARE_CACHE_BYPASS'

Set to skip the cache entirely (read and write).

anyinfer.local.CACHE_REFRESH_ENV module-attribute

CACHE_REFRESH_ENV = 'ANYINFER_HARDWARE_CACHE_REFRESH'

Set to ignore a cached result and re-probe, then rewrite the cache.

Resource Sampling and Storage

Lightweight host metrics used by benchmarks and local-capacity reporting.

anyinfer.local.ResourceSample dataclass

ResourceSample(
    cpu_percent: float | None = None,
    ram_percent: float | None = None,
    gpu_percent: float | None = None,
    vram_percent: float | None = None,
    ram_used_bytes: int | None = None,
    vram_used_bytes: int | None = None,
)

One instantaneous host-utilization observation.

Percentages range from 0 to 100. None means the platform did not expose a safe, dependency-free reading.

anyinfer.local.SystemSampler

SystemSampler()

Stateful sampler for CPU, RAM, GPU, and VRAM utilization.

sample

sample() -> ResourceSample

Read one sample, leaving unsupported values unknown.

anyinfer.local.StorageProfile dataclass

StorageProfile(
    path: str,
    total_bytes: int | None = None,
    free_bytes: int | None = None,
)

Capacity facts for the filesystem holding a path.

anyinfer.local.storage_profile

storage_profile(path: Path | str) -> StorageProfile

Return capacity/free-space facts for path without performing a speed test.

Backends

anyinfer.local.available_backends

available_backends(
    *,
    search_paths: list[Path] | None = None,
    hardware: HardwareProfile | None = None,
    runtime_root: Path | None = None,
    include_runtime_root: bool = True,
) -> list[Backend]

Find installed llama-server binaries, best first.

Three sources, in descending order of how much they can be trusted:

1. Manifest-validated variants under the well-known runtime root. The manifest states the backend, so nothing is inferred.
2. Caller-supplied directories, where the backend is guessed from the directory name — a convention, not a fact.
3. PATH, where the backend is guessed from the hardware, because a binary on PATH says nothing at all about what it was compiled against.

The distinction is recorded in Backend.detail rather than hidden, so a surprising selection can be explained.

Parameters:

Name	Type	Description	Default
search_paths	list[Path] | None	Extra directories to search, each expected to hold a runtime variant named after its backend (.../cuda/llama-server).	None
hardware	HardwareProfile | None	Detected hardware, used to label what a found binary can actually drive.	None
runtime_root	Path | None	Override the well-known runtime root.	None
include_runtime_root	bool	Search the well-known runtime root at all.	True

Returns:

Type	Description
list[Backend]	Usable backends, ranked. Empty when no binary is found at all.

anyinfer.local.select_backend

select_backend(
    hardware: HardwareProfile,
    *,
    preferred: AcceleratorKind | None = None,
    search_paths: list[Path] | None = None,
    runtime_root: Path | None = None,
    include_runtime_root: bool = True,
) -> Backend | None

Pick the requested, or best, backend this machine can actually use.

A CUDA build on a machine with no NVIDIA device is useless, so the selection is the intersection of what is installed and what the hardware can drive. When the best drivable variant is not the best variant the hardware could theoretically use — a Vulkan build on an NVIDIA card because the CUDA add-on is not installed — the returned Backend.detail says so, which is what turns a silent degradation into a discoverable recommendation.

anyinfer.local.Backend dataclass

Backend(
    kind: AcceleratorKind,
    binary: Path,
    rank: int = 0,
    detail: str = "",
)

One usable llama.cpp runtime variant.

Attributes:

Name	Type	Description
kind	AcceleratorKind	The acceleration family this build targets.
binary	Path	Path to its llama-server executable.
rank	int	Preference score from BACKEND_RANK.
detail	str	Why this backend was or was not selected.

anyinfer.local.BACKEND_RANK module-attribute

BACKEND_RANK: dict[AcceleratorKind, int] = {
    "cuda": 30,
    "metal": 25,
    "rocm": 22,
    "vulkan": 20,
    "cpu": 10,
}

Preference order across runtime variants; higher wins.

Runtime Variants

AnyInfer ships no llama.cpp binaries. These fetch, validate, and select them; CUDA is an explicit opt-in, never installed on a user's behalf.

anyinfer.local.install_runtime

install_runtime(
    kind: AcceleratorKind | None = None,
    *,
    hardware: HardwareProfile | None = None,
    root: Path | None = None,
    table: RuntimeTable | None = None,
    progress: ProgressCallback | None = None,
    client: Client | None = None,
    force: bool = False,
) -> InstallReport

Fetch, verify, and unpack a llama-server runtime variant.

Parameters:

Name	Type	Description	Default
kind	AcceleratorKind | None	Which backend to install. None picks default_runtime_kind, which is never CUDA.	None
hardware	HardwareProfile | None	Detected hardware, needed for the default choice and the CUDA gate.	None
root	Path | None	Runtime root; defaults to runtime_root().	None
table	RuntimeTable | None	Pinned artifact table; defaults to the bundled one.	None
progress	ProgressCallback | None	Download progress callback.	None
client	Client | None	An httpx2.Client, for tests or custom transports.	None
force	bool	Install CUDA even when the precondition checks object. Never skips digest verification — only the hardware gate.	False

Returns:

Type	Description
InstallReport	A report naming the executable to launch.

Raises:

Type	Description
LocalRuntimeError	If no build exists for this platform and backend, if a CUDA precondition fails without force, or if the archive fails verification or cannot be unpacked.

anyinfer.local.installed_runtimes

installed_runtimes(
    root: Path | None = None, *, build: str | None = None
) -> list[RuntimeManifest]

Every validated runtime variant under the runtime root.

anyinfer.local.remove_runtime

remove_runtime(
    kind: AcceleratorKind, *, root: Path | None = None
) -> bool

Delete an installed runtime variant. Returns whether anything was removed.

anyinfer.local.default_runtime_kind

default_runtime_kind(
    hardware: HardwareProfile | None,
) -> AcceleratorKind

Which variant to install when the caller expresses no preference.

Never CUDA. The vendor-neutral small builds cover every GPU well enough to be useful immediately, and a several-hundred-megabyte download is a decision a user makes, not one a library makes on their behalf:

• Apple Silicon → Metal, which needs no vendor runtime and is the native path.
• Intel Mac → CPU; llama.cpp's Metal backend targets Apple Silicon.
• Windows or Linux with any GPU → Vulkan, which drives NVIDIA, AMD, and Intel alike.
• Anything else → CPU.

anyinfer.local.check_cuda_preconditions

check_cuda_preconditions(
    hardware: HardwareProfile, table: RuntimeTable
) -> tuple[tuple[str, ...], tuple[str, ...]]

Return (blocking_reasons, warnings) for installing the CUDA add-on.

The pinned build implies a CUDA toolkit version, which implies a minimum driver and a minimum compute capability. Checking them up front turns "your GPU is too old" into a clear refusal before a 400 MB download instead of an incomprehensible crash at load.

anyinfer.local.install_hint

install_hint(
    hardware: HardwareProfile | None,
    table: RuntimeTable | None = None,
) -> str

A one-line suggestion of which runtime this machine should install.

anyinfer.local.load_runtime_table

load_runtime_table(
    path: Path | None = None,
) -> RuntimeTable

Load the pinned runtime table.

Raises:

Type	Description
LocalRuntimeError	If the bundled table is unreadable or malformed. Unlike the probes, this is real data shipped with the package: a broken table is a build defect, not a property of the user's machine.

anyinfer.local.runtime_root

runtime_root() -> Path

Where runtime variants are installed.

Follows the same per-OS data-dir convention as the model directory, so a user who knows where one lives can find the other. Overridable with ANYINFER_RUNTIME_DIR.

anyinfer.local.RuntimeTable dataclass

RuntimeTable(
    build: str,
    generated: str = "",
    release_url: str = "",
    cuda_toolkit: str = "",
    min_cuda_driver_major: int = 0,
    min_compute_capability: float = 0.0,
    warn_below_vram_bytes: int = 0,
    artifacts: tuple[RuntimeArtifact, ...] = (),
)

The pinned set of fetchable runtime builds.

Attributes:

Name	Type	Description
build	str	The llama.cpp release tag every variant here comes from.
generated	str	When the table was pinned.
release_url	str	The upstream release page.
cuda_toolkit	str	CUDA version the pinned CUDA build links against.
min_cuda_driver_major	int	Driver major version that toolkit requires.
min_compute_capability	float	Lowest GPU compute capability the build supports.
warn_below_vram_bytes	int	Below this, CUDA works but is not worth the download.
artifacts	tuple[RuntimeArtifact, ...]	Every pinned variant.

for_platform

for_platform(
    key: str | None = None,
) -> tuple[RuntimeArtifact, ...]

Variants available for one platform, best backend first.

artifact

artifact(
    backend: str, *, key: str | None = None
) -> RuntimeArtifact | None

One variant by backend, for this platform.

anyinfer.local.RuntimeArtifact dataclass

RuntimeArtifact(
    platform: str,
    backend: AcceleratorKind,
    filename: str,
    url: str,
    sha256: str,
    size_bytes: int | None = None,
    companions: tuple[RuntimeArtifact, ...] = (),
)

One downloadable runtime archive.

Attributes:

Name	Type	Description
platform	str	Platform key this build targets ("win32-amd64").
backend	AcceleratorKind	Acceleration family the build was compiled for.
filename	str	Archive file name.
url	str	Where to fetch it.
sha256	str	Expected digest of the archive.
size_bytes	int | None	Expected archive size.
companions	tuple[RuntimeArtifact, ...]	Extra archives unpacked into the same directory — the CUDA runtime libraries ship separately from the llama.cpp build.

total_bytes property

total_bytes: int

Bytes to transfer including companions.

anyinfer.local.RuntimeManifest dataclass

RuntimeManifest(
    backend: AcceleratorKind,
    build: str,
    architecture: str,
    executable: Path,
    directory: Path,
)

A validated runtime.json describing an installed variant.

Attributes:

Name	Type	Description
backend	AcceleratorKind	The acceleration family this build targets.
build	str	The llama.cpp build id it was cut from.
architecture	str	The machine architecture it runs on.
executable	Path	Absolute path to llama-server.
directory	Path	The variant directory.

anyinfer.local.InstallReport dataclass

InstallReport(
    backend: AcceleratorKind,
    build: str,
    directory: Path,
    executable: Path,
    downloaded_bytes: int = 0,
    reused: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of installing a runtime variant.

Attributes:

Name	Type	Description
backend	AcceleratorKind	Which variant was installed.
build	str	The build id it came from.
directory	Path	Where it landed.
executable	Path	The llama-server inside it.
downloaded_bytes	int	Bytes actually transferred (zero when everything was cached).
reused	bool	Whether an already-valid install was kept.
warnings	tuple[str, ...]	Non-blocking notes, including CUDA precondition warnings.

Tuning

anyinfer.local.plan_server

plan_server(
    hardware: HardwareProfile,
    model: TuningInputs,
    *,
    posture: Posture = "balanced",
) -> ServerPlan

Derive a server plan from hardware, model facts, and posture.

Parameters:

Name	Type	Description	Default
hardware	HardwareProfile	The detected profile. Unknown fields make the plan more conservative, never more optimistic.	required
model	TuningInputs	What is known about the model being served.	required
posture	Posture	How much of the machine to commit.	'balanced'

Returns:

Type	Description
ServerPlan	A plan whose memory estimate the caller can check before spawning.

anyinfer.local.TuningInputs dataclass

TuningInputs(
    artifact_size_bytes: int | None = None,
    parameter_size: str | None = None,
    max_context: int | None = None,
    requested_context: int | None = None,
)

What the tuner needs to know about the model being served.

Attributes:

Name	Type	Description
artifact_size_bytes	int | None	On-disk size of the weights; they must be resident too.
parameter_size	str | None	Parameter class ("7B"), keying the KV-cost table.
max_context	int | None	Upper bound from the model itself, when known.
requested_context	int | None	An explicit context the caller wants, overriding the ladder.

anyinfer.local.ServerPlan dataclass

ServerPlan(
    context_size: int,
    parallel: int = 1,
    threads: int = 4,
    batch_size: int = 512,
    ubatch_size: int = 128,
    gpu_layers: int = 0,
    cache_type_k: str = "f16",
    cache_type_v: str = "f16",
    flash_attention: bool = False,
    estimated_kv_bytes: int = 0,
    estimated_total_bytes: int = 0,
    posture: Posture = "balanced",
    rationale: tuple[str, ...] = (),
    projector_path: str | None = None,
    embeddings: bool = False,
)

A concrete, explainable llama-server configuration.

Attributes:

Name	Type	Description
context_size	int	--ctx-size, the total context across all slots.
parallel	int	--parallel, concurrent request slots.
threads	int	--threads for CPU work.
batch_size	int	--batch-size.
ubatch_size	int	--ubatch-size.
gpu_layers	int	--n-gpu-layers; 0 means CPU-only.
cache_type_k	str	--cache-type-k.
cache_type_v	str	--cache-type-v.
flash_attention	bool	Whether to request flash attention.
estimated_kv_bytes	int	Predicted KV-cache footprint, for admission control.
estimated_total_bytes	int	Weights plus KV cache.
posture	Posture	The posture this plan was derived under.
rationale	tuple[str, ...]	Human-readable notes explaining the choices.
embeddings	bool	--embeddings. Live-verified 2026-08-14: llama-server refuses every embedding request with a 501 ("This server does not support embeddings. Start it with --embeddings") unless this was set at startup — it cannot be toggled on an already-running server, so a plan's embedding intent must be decided before the process is spawned, not after.

context_per_slot property

context_per_slot: int

Usable context for a single request.

server_arguments

server_arguments(
    model_path: str, *, host: str, port: int
) -> list[str]

Render the plan as llama-server CLI arguments.

--jinja is always on: without it llama-server cannot apply a model's chat template, and tool calling silently does not work at all.

anyinfer.local.Posture module-attribute

Posture = Literal['conservative', 'balanced', 'aggressive']

How much of the machine the user is willing to spend on inference.

anyinfer.local.CONTEXT_LADDER module-attribute

CONTEXT_LADDER: tuple[int, ...] = (
    8192,
    16384,
    32768,
    65536,
)

Context sizes to consider, smallest first. The largest that fits wins.

anyinfer.local.kv_bytes_per_token

kv_bytes_per_token(
    parameter_size: str | None, cache_type: str
) -> int

Estimate KV-cache bytes per token for a model class and cache precision.

Fit and Variant Selection

Whether a model will run on this machine, and which quantization to acquire for it. Both are advisory and both explain themselves.

anyinfer.local.classify_fit

classify_fit(
    entry: SizedEntry,
    hardware: HardwareProfile | None,
    *,
    posture: Posture = "balanced",
    backend: Backend | None = None,
) -> ModelFit

Classify one catalog entry against a machine.

Parameters:

Name	Type	Description	Default
entry	SizedEntry	The catalog model, with its stored memory estimates.	required
hardware	HardwareProfile | None	The profile to budget against. None — the remote-host case — always yields unknown, because guessing someone else's machine is not advice.	required
posture	Posture	How much of the machine to commit; matches the tuner's postures.	'balanced'
backend	Backend | None	The runtime variant that would actually drive this. Used only to surface the upgrade path when a faster one is available but not installed.	None

Returns:

Type	Description
ModelFit	A fit level with reasons. Never raises.

anyinfer.local.ModelFit dataclass

ModelFit(
    level: FitLevel,
    reasons: tuple[str, ...] = (),
    headroom_bytes: int | None = None,
)

How a model relates to a machine's memory.

Attributes:

Name	Type	Description
level	FitLevel	The classification.
reasons	tuple[str, ...]	Human-readable notes, mirroring ServerPlan.rationale. Always non-empty.
headroom_bytes	int | None	Budget minus requirement for the level that was chosen; negative when nothing fit, None when the numbers were unknown.

runnable property

runnable: bool

Whether this machine can plausibly run the model at all.

rank property

rank: int

Sort key for best-fit-first ordering; higher is better.

anyinfer.local.FitLevel module-attribute

FitLevel = Literal['gpu', 'cpu', 'tight', 'no', 'unknown']

How well a model fits: fully offloaded, CPU-resident, marginal, impossible, or unknown.

anyinfer.local.memory_budget

memory_budget(
    hardware: HardwareProfile,
    *,
    posture: Posture = "balanced",
) -> tuple[int | None, int | None]

Return (vram_budget, ram_budget) in bytes for a posture.

None means "not determinable". Free memory is preferred over total when the platform reported it, because a device already hosting a desktop compositor does not have its nameplate VRAM available. Unified memory reports no separate VRAM budget: it is the RAM budget, and counting it twice is how a plan overcommits an Apple Silicon machine.

anyinfer.local.sort_by_fit

sort_by_fit(
    pairs: Sequence[tuple[_EntryT, ModelFit]],
) -> list[tuple[_EntryT, ModelFit]]

Order entries best-fit-first, then by descending headroom, then by id.

Ties broken deterministically so the same catalog and the same machine always produce the same listing — a browsing UI that reshuffles between calls is unusable. Generic in the entry type so a caller keeps whatever it put in, rather than having its rows widened to the protocol.

anyinfer.local.SizedEntry

Bases: Protocol

The subset of a catalog model this module needs.

id property

id: str

Catalog model id.

parameter_size property

parameter_size: str | None

Parameter class ("7B"), when stated.

est_ram_bytes property

est_ram_bytes: int | None

Memory needed on the CPU-only path.

est_vram_bytes property

est_vram_bytes: int | None

Memory needed when fully offloaded.

anyinfer.local.select_variant

select_variant(
    variants: Sequence[SelectableVariant],
    hardware: HardwareProfile | None,
    *,
    engine: str | None = None,
    parameter_size: str | None = None,
    backend: Backend | None = None,
    prefs: VariantPrefs | None = None,
) -> VariantChoice | None

Choose the best quantization this machine can actually run.

Parameters:

Name	Type	Description	Default
variants	Sequence[SelectableVariant]	The model's ladder, in any order.	required
hardware	HardwareProfile | None	The machine to budget against. None yields None: guessing a quantization for an unknown machine is exactly the kind of confident wrong answer this module exists to avoid.	required
engine	str | None	Restrict to one engine's variants.	None
parameter_size	str | None	Parameter class, for the KV-cache cost.	None
backend	Backend | None	The runtime that would drive this, used to surface an upgrade path.	None
prefs	VariantPrefs | None	Selection preferences.	None

Returns:

Type	Description
VariantChoice | None	The chosen rung, or None when nothing acceptable fits. Use
VariantChoice | None	evaluate_variants when the rejection reasons matter too.

anyinfer.local.evaluate_variants

evaluate_variants(
    variants: Sequence[SelectableVariant],
    hardware: HardwareProfile | None,
    *,
    engine: str | None = None,
    parameter_size: str | None = None,
    backend: Backend | None = None,
    prefs: VariantPrefs | None = None,
) -> tuple[
    VariantChoice | None, tuple[tuple[str, str], ...]
]

Choose a rung and return why every other rung was passed over.

The rejections matter even — especially, when nothing was chosen: "no quantization fits" is only useful advice if it comes with the numbers behind it. select_variant is the convenience wrapper for callers that only need the choice.

Returns:

Type	Description
VariantChoice | None	(choice, rejections). choice is None when nothing acceptable fits,
tuple[tuple[str, str], ...]	which is a real answer, not a failure — the caller should then offer a smaller
tuple[VariantChoice | None, tuple[tuple[str, str], ...]]	model.

anyinfer.local.VariantChoice dataclass

VariantChoice(
    variant_id: str,
    quantization: str,
    fit: ModelFit,
    engine: str = "llama.cpp",
    est_file_bytes: int | None = None,
    reasons: tuple[str, ...] = (),
    rejected: tuple[tuple[str, str], ...] = (),
    tensor_parallel_size: int = 1,
    gpu_memory_utilization: float | None = None,
)

The chosen rung, and why the others were not.

Attributes:

Name	Type	Description
variant_id	str	The chosen variant.
quantization	str	What it ships.
fit	ModelFit	How it fits this machine.
engine	str	Which engine it targets.
est_file_bytes	int | None	What it will cost to download.
reasons	tuple[str, ...]	Why this rung, and why not the next one up.
rejected	tuple[tuple[str, str], ...]	(variant_id, why not) for every rung that was passed over.
tensor_parallel_size	int	How many devices a vLLM launch should span.
gpu_memory_utilization	float | None	The utilization the budget assumed.

anyinfer.local.VariantPrefs dataclass

VariantPrefs(
    posture: Posture = "balanced",
    context: int = _DEFAULT_CONTEXT,
    allow_low_quality: bool = False,
    allow_multi_gpu: bool = False,
    gpu_memory_utilization: float = 0.9,
    max_download_bytes: int | None = None,
)

How aggressively to choose.

Attributes:

Name	Type	Description
posture	Posture	Memory posture, matching the tuner's.
context	int	Context length to budget the KV cache for.
allow_low_quality	bool	Permit rungs below Q4_K_M.
allow_multi_gpu	bool	Let vLLM sum VRAM across identical devices, and emit a tensor_parallel_size hint. llama.cpp never sums by default, because its split is layer-wise and much easier to get wrong.
gpu_memory_utilization	float	vLLM's fraction of device memory to plan against.
max_download_bytes	int | None	Refuse variants larger than this, whatever fits in memory.

Acquisition and the Model Store

Getting weights onto this disk and finding them again. Model acquisition lives here, never in a provider adapter.

anyinfer.local.acquire

Getting model weights onto this disk: plan → preflight → fetch → verify → place.

One engine for both artifact shapes. A GGUF variant is a file set whose handle is the first shard; a Hugging Face snapshot is a directory whose handle is the directory. The file list and the handle differ; the machinery — resume, digest verification, locking, cancellation, and progress accounting — does not, and those are exactly the things that are expensive to get right twice.

Progress is reported for the whole acquisition, not per file. A sharded artifact whose byte counter restarts at zero on every shard is worse than no progress bar, because a user cannot tell a restart from a stall. AcquisitionProgress therefore carries aggregate totals, counts bytes that were already on disk, and knows its full size before the first byte arrives — the payoff for pinning sizes in the catalog.

AcquisitionPhase module-attribute

AcquisitionPhase = Literal[
    "resolving",
    "planning",
    "downloading",
    "verifying",
    "placing",
    "done",
]

Where an acquisition is. Every transition produces a callback, unthrottled.

ProgressSink module-attribute

ProgressSink = Callable[[AcquisitionProgress], None]

Receives AcquisitionProgress. See its docstring for the threading contract.

AcquisitionProgress dataclass

AcquisitionProgress(
    model_id: str,
    variant_id: str,
    phase: AcquisitionPhase,
    file_index: int = 0,
    file_count: int = 0,
    filename: str = "",
    file_downloaded_bytes: int = 0,
    file_total_bytes: int | None = None,
    total_downloaded_bytes: int = 0,
    total_bytes: int | None = None,
    total_is_estimate: bool = False,
    session_bytes: int = 0,
    bytes_per_second: float | None = None,
    eta_seconds: float | None = None,
)

One progress report for a whole acquisition.

The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run.

Attributes:

Name	Type	Description
model_id	str	The catalog model being acquired.
variant_id	str	The variant being acquired.
phase	AcquisitionPhase	Which stage this report came from.
file_index	int	1-based index of the file that most recently advanced.
file_count	int	How many files this acquisition covers.
filename	str	Name of the file that most recently advanced.
file_downloaded_bytes	int	Bytes present for that file, including a resumed prefix.
file_total_bytes	int | None	Expected size of that file, when known.
total_downloaded_bytes	int	Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%.
total_bytes	int | None	Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes.
total_is_estimate	bool	True when any file's size came from a guess rather than a pinned or reported figure.
session_bytes	int	Bytes this run actually transferred, which is what rate and ETA are derived from.
bytes_per_second	float | None	Transfer rate, or None until there is a real sample.
eta_seconds	float | None	Seconds remaining, or None on the same condition. A wildly wrong ETA in the first second is worse than no ETA.

fraction property

fraction: float | None

Completion as a fraction, or None when the total is unknown.

AcquisitionPlan dataclass

AcquisitionPlan(
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str,
    engine: str,
    quantization: str,
    directory: str,
    handle: str,
    files: tuple[RemoteFile, ...],
    already_have_bytes: int = 0,
    total_bytes: int | None = None,
    warnings: tuple[str, ...] = (),
    satisfied: bool = False,
    revision: str | None = None,
    repo: str | None = None,
    license: str = "",
)

What an acquisition will do, before it does any of it.

Attributes:

Name	Type	Description
entry_id	str	The store entry that will be written.
model_id	str	The catalog model.
variant_id	str	The catalog variant.
kind	str	"gguf" or "hf_repo".
engine	str	Which engine the result is for.
quantization	str	What will be on disk.
directory	str	Store-relative destination.
handle	str	Store-relative engine handle.
files	tuple[RemoteFile, ...]	Every file, resolved.
already_have_bytes	int	Bytes already on disk — verified files plus resumable .part prefixes.
total_bytes	int | None	Total size, or None when any file's size is unknown.
warnings	tuple[str, ...]	Notes from resolution.
satisfied	bool	True when everything is already present and verified.

remaining_bytes property

remaining_bytes: int | None

Bytes still to transfer, or None when the total is unknown.

total_is_estimate property

total_is_estimate: bool

Whether any file's size was unknown, making the total a floor.

AcquisitionReport dataclass

AcquisitionReport(
    plan: AcquisitionPlan,
    entry: StoreEntry | None = None,
    downloaded_bytes: int = 0,
    reused: bool = False,
    cancelled: bool = False,
    dry_run: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of an acquisition.

Attributes:

Name	Type	Description
plan	AcquisitionPlan	What was planned.
entry	StoreEntry | None	The registered store entry, or None for a dry run or a cancellation.
downloaded_bytes	int	Bytes transferred by this run.
reused	bool	True when nothing had to be transferred.
cancelled	bool	True when the caller stopped it. Partial transfers are kept.
dry_run	bool	True when nothing was written.
warnings	tuple[str, ...]	Everything the caller should know.

path property

path: Path | None

The engine handle, when one was registered.

AcquisitionRequest dataclass

AcquisitionRequest(
    ref: SourceRef,
    model_id: str,
    variant_id: str = "",
    kind: str = "gguf",
    engine: str = "llama.cpp",
    quantization: str = "",
    license: str = "",
    token: str | None = None,
    max_concurrent_files: int = 3,
    allow_unverified: bool = False,
    enforce_license: bool = False,
    launch_hints: Mapping[str, Any] = dict(),
)

Everything one acquisition needs.

Attributes:

Name	Type	Description
ref	SourceRef	Where the bytes come from.
model_id	str	Catalog model id, for the index and for progress reports.
variant_id	str	Catalog variant id.
kind	str	"gguf" or "hf_repo".
engine	str	Which engine the result is for.
quantization	str	What will be on disk.
license	str	License id, checked when enforce_license is set.
token	str | None	Credential for the source, when it needs one.
max_concurrent_files	int	How many transfers may run at once.
allow_unverified	bool	Accept files no digest can check.
enforce_license	bool	Refuse licenses outside the allowlist.
launch_hints	Mapping[str, Any]	Advisory engine arguments to attach to the result.

plan_acquisition async

plan_acquisition(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
) -> AcquisitionPlan

Resolve a source and work out exactly what would be transferred.

Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards.

acquire async

acquire(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
    progress: ProgressSink | None = None,
    plan: AcquisitionPlan | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Acquire a model variant into the store.

Parameters:

Name	Type	Description	Default
request	AcquisitionRequest	What to acquire and from where.	required
store	ModelStore | None	The destination store; defaults to the standard root.	None
client	AsyncClient | None	An httpx2.AsyncClient, for tests or custom transports.	None
progress	ProgressSink | None	Aggregate progress sink.	None
plan	AcquisitionPlan | None	A plan from plan_acquisition, to avoid resolving twice.	None
dry_run	bool	Resolve and report sizes without writing anything.	False
cancel_check	Callable[[], bool] | None	Polled between chunks; returning True stops the acquisition cooperatively. Task cancellation works too and is the primary mechanism — this exists for the synchronous facade.	None

Returns:

Type	Description
AcquisitionReport	The report, naming the registered entry.

Raises:

Type	Description
LocalRuntimeError	On a transfer failure, a digest mismatch, or insufficient disk.

acquire_sync

acquire_sync(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    progress: ProgressSink | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Blocking wrapper around acquire, for callers with no event loop.

Raises:

Type	Description
RuntimeError	If called from inside a running event loop, where it would deadlock. Use acquire there.

launch_hints_for

launch_hints_for(
    entry: StoreEntry,
    *,
    path: Path,
    context_size: int | None = None,
    gpu_layers: int | None = None,
    tensor_parallel_size: int | None = None,
    gpu_memory_utilization: float | None = None,
) -> dict[str, Any]

Build the advisory engine arguments that accompany a located model.

Data, not process control. These are keys a caller — the llama.cpp supervisor, a future vLLM launcher, or a user pasting a command line — turns into arguments. Producing them from numbers already computed is translation; launching is not, and nothing here starts a process.

anyinfer.local.acquire_sync

acquire_sync(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    progress: ProgressSink | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Blocking wrapper around acquire, for callers with no event loop.

Raises:

Type	Description
RuntimeError	If called from inside a running event loop, where it would deadlock. Use acquire there.

anyinfer.local.plan_acquisition async

plan_acquisition(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
) -> AcquisitionPlan

Resolve a source and work out exactly what would be transferred.

Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards.

anyinfer.local.AcquisitionPlan dataclass

AcquisitionPlan(
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str,
    engine: str,
    quantization: str,
    directory: str,
    handle: str,
    files: tuple[RemoteFile, ...],
    already_have_bytes: int = 0,
    total_bytes: int | None = None,
    warnings: tuple[str, ...] = (),
    satisfied: bool = False,
    revision: str | None = None,
    repo: str | None = None,
    license: str = "",
)

What an acquisition will do, before it does any of it.

Attributes:

Name	Type	Description
entry_id	str	The store entry that will be written.
model_id	str	The catalog model.
variant_id	str	The catalog variant.
kind	str	"gguf" or "hf_repo".
engine	str	Which engine the result is for.
quantization	str	What will be on disk.
directory	str	Store-relative destination.
handle	str	Store-relative engine handle.
files	tuple[RemoteFile, ...]	Every file, resolved.
already_have_bytes	int	Bytes already on disk — verified files plus resumable .part prefixes.
total_bytes	int | None	Total size, or None when any file's size is unknown.
warnings	tuple[str, ...]	Notes from resolution.
satisfied	bool	True when everything is already present and verified.

remaining_bytes property

remaining_bytes: int | None

Bytes still to transfer, or None when the total is unknown.

total_is_estimate property

total_is_estimate: bool

Whether any file's size was unknown, making the total a floor.

anyinfer.local.AcquisitionProgress dataclass

AcquisitionProgress(
    model_id: str,
    variant_id: str,
    phase: AcquisitionPhase,
    file_index: int = 0,
    file_count: int = 0,
    filename: str = "",
    file_downloaded_bytes: int = 0,
    file_total_bytes: int | None = None,
    total_downloaded_bytes: int = 0,
    total_bytes: int | None = None,
    total_is_estimate: bool = False,
    session_bytes: int = 0,
    bytes_per_second: float | None = None,
    eta_seconds: float | None = None,
)

One progress report for a whole acquisition.

The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run.

Attributes:

Name	Type	Description
model_id	str	The catalog model being acquired.
variant_id	str	The variant being acquired.
phase	AcquisitionPhase	Which stage this report came from.
file_index	int	1-based index of the file that most recently advanced.
file_count	int	How many files this acquisition covers.
filename	str	Name of the file that most recently advanced.
file_downloaded_bytes	int	Bytes present for that file, including a resumed prefix.
file_total_bytes	int | None	Expected size of that file, when known.
total_downloaded_bytes	int	Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%.
total_bytes	int | None	Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes.
total_is_estimate	bool	True when any file's size came from a guess rather than a pinned or reported figure.
session_bytes	int	Bytes this run actually transferred, which is what rate and ETA are derived from.
bytes_per_second	float | None	Transfer rate, or None until there is a real sample.
eta_seconds	float | None	Seconds remaining, or None on the same condition. A wildly wrong ETA in the first second is worse than no ETA.

fraction property

fraction: float | None

Completion as a fraction, or None when the total is unknown.

anyinfer.local.AcquisitionReport dataclass

AcquisitionReport(
    plan: AcquisitionPlan,
    entry: StoreEntry | None = None,
    downloaded_bytes: int = 0,
    reused: bool = False,
    cancelled: bool = False,
    dry_run: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of an acquisition.

Attributes:

Name	Type	Description
plan	AcquisitionPlan	What was planned.
entry	StoreEntry | None	The registered store entry, or None for a dry run or a cancellation.
downloaded_bytes	int	Bytes transferred by this run.
reused	bool	True when nothing had to be transferred.
cancelled	bool	True when the caller stopped it. Partial transfers are kept.
dry_run	bool	True when nothing was written.
warnings	tuple[str, ...]	Everything the caller should know.

path property

path: Path | None

The engine handle, when one was registered.

anyinfer.local.AcquisitionPhase module-attribute

AcquisitionPhase = Literal[
    "resolving",
    "planning",
    "downloading",
    "verifying",
    "placing",
    "done",
]

Where an acquisition is. Every transition produces a callback, unthrottled.

anyinfer.local.ProgressSink module-attribute

ProgressSink = Callable[[AcquisitionProgress], None]

Receives AcquisitionProgress. See its docstring for the threading contract.

anyinfer.local.ModelStore

ModelStore(root: Path | None = None)

A directory of acquired models, with an index over it.

Not thread-safe by construction; correctness across processes comes from the same cooperative file lock downloads use, taken around every index mutation.

root property

root: Path

The store root.

index_path property

index_path: Path

Where the index document lives.

staging_dir

staging_dir(entry_id: str) -> Path

Where an in-progress acquisition writes its .part files.

lock_path

lock_path(entry_id: str) -> Path

The cross-process lock guarding one entry.

entry_dir

entry_dir(entry: StoreEntry) -> Path

Absolute path to an entry's directory.

resolve_within

resolve_within(entry: StoreEntry, relative: str) -> Path

Resolve a path inside an entry, refusing anything that escapes it.

Called for every file before it is opened. Names come from a remote API, so this is the containment gate, applied after resolution so a symlink cannot step outside.

Raises:

Type	Description
ConfigError	If the path is unsafe or resolves outside the entry directory.

load_index

load_index() -> dict[str, StoreEntry]

Read the index, tolerating absence and corruption.

A store whose index cannot be parsed reports as empty rather than raising: the files are still there, rebuild_index can recover them, and refusing to work at all because a cache file got truncated would be the wrong trade.

register

register(entry: StoreEntry) -> StoreEntry

Add or replace an index entry.

Called only after every file has been verified: a half-complete multi-file set is never registered, which is what makes locate trustworthy.

unregister

unregister(entry_id: str) -> StoreEntry | None

Drop an index entry without touching files.

rebuild_index

rebuild_index() -> list[StoreEntry]

Drop entries whose files are gone, and re-stat the rest.

The recovery path for a user who deleted a directory by hand. It does not invent entries for unknown directories — an unrecognized tree could be anything, and registering it would be a claim about bytes nobody checked.

list_installed

list_installed() -> list[StoreEntry]

Every registered entry, id-ordered.

get

get(entry_id: str) -> StoreEntry | None

One entry by id.

find

find(
    model_id: str,
    *,
    variant_id: str | None = None,
    quantization: str | None = None,
    engine: str | None = None,
) -> StoreEntry | None

The best registered entry matching a model and optional constraints.

"Best" is the most recently installed match, so re-acquiring at a different quantization changes what a bare model id resolves to, which is what a user who just downloaded something expects.

locate

locate(
    model_id: str,
    *,
    variant_id: str | None = None,
    quantization: str | None = None,
    engine: str | None = None,
    verify: bool = False,
    launch_hints: Mapping[str, Any] | None = None,
) -> ResolvedModel | None

Find a stored model and return a path an engine can be launched against.

No network I/O, ever. Verification is the deliberate exception to "always check": hashing forty gigabytes on every request would be absurd, so the rule is verify on install and on adoption, then on lookup compare size and mtime against the index and re-hash only on a mismatch. verify=True forces a full re-hash.

Returns:

Type	Description
ResolvedModel | None	The located model, or None when it is absent or fails its check.

check

check(
    entry: StoreEntry, *, deep: bool = False
) -> tuple[str, ...]

Report what is wrong with a stored entry, cheaply by default.

The shallow check compares size and mtime against the index; the deep check re-hashes. Either way an empty result means "as installed".

disk_usage

disk_usage() -> int

Total bytes the registered entries occupy, excluding external ones.

remove

remove(entry_id: str) -> RemovalReport

Delete an entry's files and unregister it.

An external entry — one adopted from somebody else's cache — is only unregistered. Deleting files this store never wrote would be overstepping.

plan_prune

plan_prune(
    *,
    keep_bytes: int | None = None,
    older_than_days: float | None = None,
    now: float | None = None,
) -> PrunePlan

Propose least-recently-used deletions, without deleting anything.

Two ways to say what "too much" means, and exactly one must be given:

• keep_bytes is a disk budget. Entries are proposed least-recently-used first until what remains fits, so the newest-used models survive.
• older_than_days is an age cut. Every entry idle for longer is proposed, regardless of what that leaves behind.

Neither is a default. A prune with no stated limit would have to invent one, and an invented disk budget silently deleting multi-gigabyte downloads is exactly the automatic eviction this store does not do.

Externally adopted entries are never proposed. Their bytes belong to another tool's cache, and remove would only unregister them anyway — proposing a deletion that frees nothing would misreport the space a prune reclaims.

An entry that has never been located since installation is ordered by its install time and reports idle_days of None: it is a plausible thing to evict, but calling it "idle for N days" would overstate what is known about it.

Parameters:

Name	Type	Description	Default
keep_bytes	int | None	Disk budget to fit within, in bytes.	None
older_than_days	float | None	Idle age beyond which an entry is proposed.	None
now	float | None	Clock override for tests; defaults to the current time.	None

Returns:

Type	Description
PrunePlan	The plan. An empty plan means nothing needs deleting, which is a success.

Raises:

Type	Description
ValueError	If neither limit or both were given, or a limit is negative.

apply_prune

apply_prune(plan: PrunePlan) -> tuple[RemovalReport, ...]

Delete everything a plan proposed, in the plan's own order.

Re-reads each entry through remove, so a plan computed against a store that has since changed deletes what is still there and reports removed=False for what is not, rather than failing partway through.

Parameters:

Name	Type	Description	Default
plan	PrunePlan	A plan from plan_prune.	required

Returns:

Type	Description
tuple[RemovalReport, ...]	One report per proposal, in plan order.

clear_staging

clear_staging(entry_id: str) -> None

Remove an entry's staging directory and its partial transfers.

adopt_legacy_flat

adopt_legacy_flat(
    artifacts: Sequence[Any],
) -> list[StoreEntry]

Register pre-existing flat-layout GGUF files without moving or re-fetching them.

Earlier builds wrote <root>/<filename>.gguf with no revision in the path. Those files are perfectly good bytes a user paid bandwidth for, so the store adopts them where they lie, but only after verifying each one against its catalog hash, so adoption is never a lie about what is on disk.

Parameters:

Name	Type	Description	Default
artifacts	Sequence[Any]	Pinned catalog artifacts to look for, each with id and files.	required

Returns:

Type	Description
list[StoreEntry]	Newly registered entries.

adopt_external

adopt_external(
    directory: Path,
    *,
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str = "hf_repo",
    engine: str = "vllm",
    quantization: str = "",
    source: Mapping[str, Any] | None = None,
    expected: Mapping[str, str] | None = None,
) -> StoreEntry | None

Register a directory this store does not own, if every file checks out.

The Hugging Face cache case. We do not adopt another library's layout for our own writes — it is their private implementation detail, but re-downloading forty gigabytes the user already has is user-hostile. Adopted entries are marked StoreEntry.external, are never written to, and are never deleted by remove.

Parameters:

Name	Type	Description	Default
directory	Path	The existing snapshot directory.	required
entry_id	str	Id to register it under.	required
model_id	str	Catalog model this realizes.	required
variant_id	str	Catalog variant this realizes.	required
kind	str	Artifact kind.	'hf_repo'
engine	str	Engine the variant targets.	'vllm'
quantization	str	What is on disk.	''
source	Mapping[str, Any] | None	Provenance to record.	None
expected	Mapping[str, str] | None	Per-relative-path sha256 that every file must match. Adoption is refused outright when this is empty — an unverified adoption is a guess.	None

Returns:

Type	Description
StoreEntry | None	The registered entry, or None when verification failed.

anyinfer.local.StoreEntry dataclass

StoreEntry(
    id: str,
    kind: str = "gguf",
    model_id: str = "",
    variant_id: str = "",
    quantization: str = "",
    engine: str = "llama.cpp",
    source: Mapping[str, Any] = dict(),
    directory: str = "",
    handle: str = "",
    files: tuple[StoredFile, ...] = (),
    license: str = "",
    installed_at: float = 0.0,
    last_used_at: float = 0.0,
    external: bool = False,
    warnings: tuple[str, ...] = (),
)

One acquired model in the store.

Attributes:

Name	Type	Description
id	str	Stable entry id, also the lock and staging name.
kind	str	"gguf" (a file set) or "hf_repo" (a directory snapshot).
model_id	str	The catalog model this realizes, when it came from the catalog.
variant_id	str	The catalog variant, when it came from the catalog.
quantization	str	The quantization on disk.
engine	str	Which engine this variant is for.
source	Mapping[str, Any]	How it was acquired, including the resolved immutable revision.
directory	str	Where the files live, relative to the store root.
handle	str	The path an engine is pointed at, relative to the store root — the first shard for GGUF, the directory for a snapshot.
files	tuple[StoredFile, ...]	Every file, with digests.
license	str	License id recorded at acquisition.
installed_at	float	Unix timestamp of successful registration.
last_used_at	float	Unix timestamp of the most recent ModelStore.locate.
external	bool	True when the bytes are owned by something else (an adopted Hugging Face cache). Removal only unregisters an external entry; it never deletes.
warnings	tuple[str, ...]	Anything the user should know — unverified files, most importantly.

total_bytes property

total_bytes: int

Bytes this entry occupies.

verified property

verified: bool

Whether every file was verified against a digest at install time.

to_json

to_json() -> dict[str, Any]

Serialize for the index.

from_json classmethod

from_json(data: Mapping[str, Any]) -> StoreEntry

Parse one index entry.

anyinfer.local.ResolvedModel dataclass

ResolvedModel(
    entry_id: str,
    kind: str,
    path: Path,
    quantization: str | None = None,
    engine: str = "llama.cpp",
    verified: bool = False,
    warnings: tuple[str, ...] = (),
    launch_hints: Mapping[str, Any] = dict(),
)

A located model, ready to launch an engine against.

Attributes:

Name	Type	Description
entry_id	str	The store entry.
kind	str	"gguf" or "hf_repo".
path	Path	The engine handle — a file for GGUF, a directory for a snapshot.
quantization	str | None	What is actually on disk.
engine	str	Which engine this variant is for.
verified	bool	Whether every file was verified.
warnings	tuple[str, ...]	Notes carried from the entry.
launch_hints	Mapping[str, Any]	Engine-shaped arguments a caller can turn into a command line. Advisory data, not process control: this module locates weights, it does not start servers.

anyinfer.local.RemovalReport dataclass

RemovalReport(
    entry_id: str,
    removed: bool = False,
    freed_bytes: int = 0,
    external: bool = False,
)

The outcome of removing an entry.

Attributes:

Name	Type	Description
entry_id	str	What was removed.
removed	bool	Whether an entry was found and unregistered.
freed_bytes	int	Bytes reclaimed; zero for an external entry, which is only unregistered.
external	bool	Whether the files were left alone because something else owns them.

anyinfer.local.PrunePlan dataclass

PrunePlan(
    proposals: tuple[PruneProposal, ...] = (),
    keep_bytes: int | None = None,
    total_bytes: int = 0,
    protected: Mapping[str, str] = dict(),
)

What a prune would delete, computed but not yet performed.

Separating the plan from the deletion is the whole design. Eviction here is guided, never automatic: the store proposes, a person confirms, and the same plan object is what the CLI renders and what a programmatic caller inspects. Nothing in this type touches the filesystem.

Attributes:

Name	Type	Description
proposals	tuple[PruneProposal, ...]	Entries to delete, least-recently-used first — the order they would be deleted in, so a caller taking a prefix gets the same answer as a smaller budget would have produced.
keep_bytes	int | None	The disk budget the plan was computed against, or None when the caller asked for an age cut instead.
total_bytes	int	Bytes the store occupies now, counting only entries this store owns.
protected	Mapping[str, str]	Entries excluded from consideration, with the reason — externally adopted bytes another tool owns, most of all.

freed_bytes property

freed_bytes: int

Bytes the whole plan would reclaim.

remaining_bytes property

remaining_bytes: int

Bytes the store would occupy after the plan ran.

__bool__

__bool__() -> bool

Whether the plan proposes anything at all.

anyinfer.local.PruneProposal dataclass

PruneProposal(
    entry: StoreEntry,
    freed_bytes: int,
    idle_days: float | None,
)

One entry a prune plan suggests deleting, and why.

Attributes:

Name	Type	Description
entry	StoreEntry	The store entry proposed for deletion.
freed_bytes	int	Bytes this deletion would reclaim.
idle_days	float | None	Days since the entry was last located, or None when it has never been used since installation — which is not the same as "idle forever" and is shown differently, since a model downloaded an hour ago and never run should not read as the stalest thing in the store.

Sources

Where weights come from. Adding an internal mirror is a resolver, not a dependency.

anyinfer.local.SourceRef dataclass

SourceRef(
    resolver: str = "huggingface",
    repo: str | None = None,
    revision: str | None = None,
    files: tuple[str, ...] = (),
    digests: Mapping[str, str] = dict(),
    sizes: Mapping[str, int] = dict(),
    roles: Mapping[str, str] = dict(),
    urls: tuple[str, ...] = (),
    include: tuple[str, ...] = (),
    exclude: tuple[str, ...] = (),
    path: str | None = None,
)

A declarative pointer at a set of remote (or already-local) files.

Attributes:

Name	Type	Description
resolver	str	Which resolver understands this reference.
repo	str | None	Repository id, for repository-shaped resolvers ("Qwen/Qwen2.5-7B-Instruct").
revision	str | None	Branch, tag, or — preferably — an immutable commit sha.
files	tuple[str, ...]	Explicit file list. Empty means "whatever the include globs match".
digests	Mapping[str, str]	Per-file expected sha256, when the catalog pinned them.
sizes	Mapping[str, int]	Per-file expected byte counts, when the catalog pinned them.
roles	Mapping[str, str]	Optional per-file roles for companion artifacts such as a vision projector.
urls	tuple[str, ...]	Direct download URLs, for the url resolver.
include	tuple[str, ...]	Glob patterns selecting files from a repository listing.
exclude	tuple[str, ...]	Glob patterns removing files the include list matched.
path	str | None	An existing on-disk location, for the local resolver.

to_json

to_json() -> dict[str, object]

Serialize for the store index.

anyinfer.local.SourceResolver

Bases: Protocol

Turns a SourceRef into a ResolvedArtifact.

scheme instance-attribute

scheme: str

The SourceRef.resolver value this implementation answers to.

resolve async

resolve(
    ref: SourceRef,
    *,
    token: str | None = None,
    client: Any | None = None,
) -> ResolvedArtifact

Expand a reference into a concrete file list.

client is an optional httpx2.AsyncClient the caller already owns. Passing it keeps resolution and the subsequent transfer on one connection pool, and is what makes a resolver testable against a mock transport.

anyinfer.local.ResolvedArtifact dataclass

ResolvedArtifact(
    resolver: str,
    files: tuple[RemoteFile, ...] = (),
    repo: str | None = None,
    revision: str | None = None,
    warnings: tuple[str, ...] = (),
)

A concrete, ordered file list ready for acquisition.

Attributes:

Name	Type	Description
resolver	str	Which resolver produced this.
files	tuple[RemoteFile, ...]	Every file to fetch, in acquisition order.
repo	str | None	The repository this came from, when applicable.
revision	str | None	The immutable revision this resolved to, when the resolver could determine one. A branch name is always resolved to a commit before use.
warnings	tuple[str, ...]	Anything the caller should know — unverifiable files, skipped pickle weights, and so on.

total_bytes property

total_bytes: int | None

Sum of every file size, or None when any one is unknown.

anyinfer.local.RemoteFile dataclass

RemoteFile(
    path: str,
    url: str,
    size_bytes: int | None = None,
    digest: str = "",
    digest_kind: DigestKind = "none",
)

One file a resolver decided belongs to an artifact.

Attributes:

Name	Type	Description
path	str	Destination path relative to the entry directory, POSIX-separated.
url	str	Where to fetch it.
size_bytes	int | None	Expected size, or None when genuinely unknown.
digest	str	Expected digest, lowercase hex.
digest_kind	DigestKind	How to compute digest. "none" means unverifiable.

filename property

filename: str

The final path component.

Artifacts and Downloads

anyinfer.local.GgufArtifact dataclass

GgufArtifact(
    id: str,
    files: tuple[GgufFile, ...],
    license: str = "",
    description: str = "",
    parameter_size: str | None = None,
    quantization: str | None = None,
    est_ram_bytes: int | None = None,
    est_vram_bytes: int | None = None,
    embedding: EmbeddingCapabilities | None = None,
)

A pinned, verifiable local model artifact.

Attributes:

Name	Type	Description
id	str	Stable artifact id, the handle alias targets and the llama.cpp adapter use.
files	tuple[GgufFile, ...]	Every file the artifact comprises — one entry, or several for a sharded model.
license	str	License id; checked against the download allowlist for application-supplied entries.
description	str	Free text for display.
parameter_size	str | None	Parameter class ("7B"), when known.
quantization	str | None	Quantization of the shipped weights ("Q4_K_M"), when known.
est_ram_bytes	int | None	Estimated memory needed on the CPU-only path.
est_vram_bytes	int | None	Estimated memory needed when fully offloaded.
embedding	EmbeddingCapabilities | None	Vector facts, when the artifact's weights are an embedding model rather than a chat model. Its presence is what marks the artifact as one llama-server must be started with --embeddings to serve.

total_size_bytes property

total_size_bytes: int | None

Sum of every file's size, or None when any is unknown.

is_sharded property

is_sharded: bool

Whether this artifact spans multiple files.

projector property

projector: GgufFile | None

Pinned multimodal projector file, when this model has one.

__post_init__

__post_init__() -> None

Require model weights first and at most one projector companion.

anyinfer.local.GgufFile dataclass

GgufFile(
    filename: str,
    url: str,
    sha256: str = "",
    size_bytes: int | None = None,
    role: Literal["model", "projector"] = "model",
)

One file of a (possibly sharded) GGUF artifact.

Attributes:

Name	Type	Description
filename	str	The name the file is stored under in the model directory.
url	str	Pinned download URL.
sha256	str	Expected content hash. Empty means the file cannot be verified, and the downloader warns instead of checking.
size_bytes	int | None	Expected size, when known; feeds download progress totals.

anyinfer.local.download_artifact

download_artifact(
    artifact: GgufArtifact,
    *,
    model_dir: Path | None = None,
    progress: ProgressCallback | None = None,
    client: Client | None = None,
    enforce_license: bool = False,
) -> DownloadReport

Ensure every file of an artifact is present and verified.

Parameters:

Name	Type	Description	Default
artifact	GgufArtifact	The pinned catalog entry.	required
model_dir	Path | None	Destination directory; defaults to default_model_dir().	None
progress	ProgressCallback | None	Called as bytes arrive.	None
client	Client | None	An httpx2.Client to use, for tests or custom transports.	None
enforce_license	bool	Reject artifacts whose license is not in ALLOWED_LICENSES. Applied to application-supplied entries.	False

Returns:

Type	Description
DownloadReport	A report naming the on-disk files.

Raises:

Type	Description
LocalRuntimeError	On a hash mismatch, a transfer failure, or a rejected license.

anyinfer.local.iter_missing

iter_missing(
    artifacts: Iterable[GgufArtifact],
    model_dir: Path | None = None,
) -> list[GgufArtifact]

Which artifacts are absent or fail verification.

anyinfer.local.verify_file

verify_file(path: Path, expected_sha256: str) -> bool

Whether a file matches its expected hash.

An artifact with no recorded hash cannot be verified; its mere existence is accepted, and the caller is warned.

anyinfer.local.artifact_paths

artifact_paths(
    artifact: GgufArtifact, model_dir: Path | None = None
) -> tuple[Path, ...]

Where an artifact's files would live on disk.

anyinfer.local.default_model_dir

default_model_dir() -> Path

Where downloaded artifacts live by default.

anyinfer.local.DownloadReport dataclass

DownloadReport(
    artifact_id: str,
    paths: tuple[Path, ...],
    downloaded_bytes: int = 0,
    reused: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of ensuring an artifact is present.

Attributes:

Name	Type	Description
artifact_id	str	The artifact this report describes.
paths	tuple[Path, ...]	Where each of the artifact's files lives on disk, in declaration order.
downloaded_bytes	int	Bytes actually transferred; zero when everything was reused.
reused	bool	Whether every file was already present and verified, so nothing was fetched.
warnings	tuple[str, ...]	Non-fatal notes — files with no recorded hash, or files that failed verification and were re-downloaded.

primary_path property

primary_path: Path

The file to hand to llama-server (the first shard of a sharded artifact).

anyinfer.local.ProgressCallback module-attribute

ProgressCallback = Callable[[str, int, int | None], None]

(artifact_id, downloaded_bytes, total_bytes_or_None).

anyinfer.local.ALLOWED_LICENSES module-attribute

ALLOWED_LICENSES = frozenset(
    {
        "apache-2.0",
        "falcon-llm-2.0",
        "gemma-terms",
        "llama-3.1-community",
        "llama-3.2-community",
        "llama-3.3-community",
        "mit",
        "openrail-m",
    }
)

Licenses permitted for catalog entries, compared case-insensitively.

The bundled catalog is curated; entries an application adds are checked so that a convenience feature cannot quietly redistribute weights under terms the user has not seen. Non-commercial and research-only terms are deliberately absent — an application that has accepted those terms adds the model through a catalog overlay, which is an explicit act.

anyinfer.local.license_allowed

license_allowed(license_id: str) -> bool

Whether a license id is in ALLOWED_LICENSES, ignoring case.

Server Supervision

anyinfer.local.ServerSupervisor

ServerSupervisor(
    *,
    binary: Path | str = "llama-server",
    hardware: HardwareProfile | None = None,
    runtime_backend: AcceleratorKind | None = None,
    idle_ttl_s: float | None = 900.0,
    max_resident: int = 1,
    allow_remote_exposure: bool = False,
    host: str = LOOPBACK_HOST,
    health_timeout_s: float = _HEALTH_TIMEOUT_S,
    on_lifecycle: LifecycleCallback | None = None,
)

Owns the llama-server processes for one client.

Parameters:

Name	Type	Description	Default
binary	Path | str	Path to the llama-server executable.	'llama-server'
hardware	HardwareProfile | None	Detected hardware, used for VRAM admission control.	None
runtime_backend	AcceleratorKind | None	A required installed backend family, or None to select the best runtime the detected hardware can drive.	None
idle_ttl_s	float | None	Unload a server after this long with no active streams. None keeps servers until the supervisor closes.	900.0
max_resident	int	How many servers may run at once. Exceeding it evicts the least-recently-used idle server.	1
allow_remote_exposure	bool	Bind a non-loopback address. Off by default: a local model server is loopback-only unless deliberately exposed.	False
on_lifecycle	LifecycleCallback | None	Called with lifecycle events.	None

resident_models property

resident_models: tuple[str, ...]

Model keys with a running server.

resident_plans property

resident_plans: Mapping[str, ServerPlan]

The launch plan each running server was started with.

What the tuner decided, which is not always what the caller assumed: a plan that offloaded no layers explains a local model running an order of magnitude slower than the same weights did on the same machine last week.

acquire async

acquire(
    model_key: str,
    model_path: Path,
    plan: ServerPlan,
    *,
    persist: bool = False,
    provenance: WeightsProvenance | None = None,
) -> ManagedServer

Get a ready server for a model, starting or reusing one.

Blocks until the server answers its health probe. Concurrent callers requesting different models are serialized, so loads never overlap.

Parameters:

Name	Type	Description	Default
model_key	str	Cache key for the running server.	required
model_path	Path	The weights to load.	required
plan	ServerPlan	The rendered llama-server configuration.	required
persist	bool	Keep the server alive past the usual idle reaping.	False
provenance	WeightsProvenance | None	Tier 4 — a signed manifest and the vendor key. When given, the weights are verified inside this call, immediately before the process is started, and their identity is re-confirmed in the instant before exec. Verifying anywhere else leaves a gap of unbounded length between the check and the load; this parameter exists so there is no such gap.	None

Raises:

Type	Description
LocalRuntimeError	If the model cannot fit, the binary is missing, or the server fails to become ready.
ConfidentialExecutionError	provenance was supplied and the weights do not match it, or changed between verification and start. Nothing is spawned.

set_hardware

set_hardware(hardware: HardwareProfile) -> None

Late-bind the detected hardware profile.

Detection is deliberately lazy — probing at construction would tax clients that never run a local model, so the adapter hands the profile over once it has one. Admission control and backend fallback stay disabled until then.

set_runtime_backend

set_runtime_backend(
    backend: AcceleratorKind | None,
) -> None

Select a named installed backend, or return to automatic selection.

Existing child processes keep the executable they started with; this affects only later server starts.

resolve_binary

resolve_binary() -> Path

Locate the llama-server executable without starting anything.

Public so a health probe can answer "could a server start?" cheaply.

Falls back to the best installed backend variant for this hardware when the configured name is not on PATH (a CUDA build in a known runtime directory beats a missing PATH entry).

Raises:

Type	Description
LocalRuntimeError	When no usable binary exists anywhere.

collect_idle async

collect_idle() -> int

Stop servers idle beyond the TTL. Returns how many were stopped.

Call periodically. Servers with active streams are never collected, however long the generation has been running.

aclose async

aclose() -> None

Stop every supervised server.

anyinfer.local.ManagedServer

ManagedServer(handle: ServerHandle)

A context manager marking a server busy for the duration of a request.

This is what makes the idle timer honest: the server is busy while a stream is open, not merely while a request is arriving.

base_url property

base_url: str

The server's base URL.

__enter__

__enter__() -> ManagedServer

Mark the server busy.

take_load_ms

take_load_ms() -> float | None

This request's share of a cold start: the load it caused, or None.

__exit__

__exit__(*exc: object) -> None

Release the server and restart its idle clock.

anyinfer.local.ServerHandle dataclass

ServerHandle(
    model_key: str,
    model_path: Path,
    plan: ServerPlan,
    host: str,
    port: int,
    process: Popen[bytes],
    started_at: float,
    load_ms: float | None = None,
    log_tail: deque[str] = (
        lambda: deque(maxlen=_LOG_TAIL_LINES)
    )(),
    active_streams: int = 0,
    last_activity: float = time.monotonic(),
    persist: bool = False,
    stopping: bool = False,
)

A running llama-server and everything known about it.

model_key instance-attribute

model_key: str

The model this server serves; also its key in the supervisor's server table.

model_path instance-attribute

model_path: Path

The GGUF file the server was started with.

plan instance-attribute

plan: ServerPlan

The tuned launch plan; its memory estimate is what admission control committed.

host instance-attribute

host: str

Interface the server is bound to — loopback unless exposure was explicitly allowed.

port instance-attribute

port: int

TCP port the server listens on, allocated just before spawning.

process instance-attribute

process: Popen[bytes]

The supervised child, polled for liveness and terminated on stop.

started_at instance-attribute

started_at: float

Monotonic time the child was spawned.

load_ms class-attribute instance-attribute

load_ms: float | None = None

How long this server took to become ready, in milliseconds, or None once that has been reported.

The supervised runtime's equivalent of a hosted engine's load duration. It is a property of the request that caused the start, not of the server, so it is consumed exactly once — every later request on the same server is warm by definition, and re-reporting the original load would turn one cold start into a permanent one.

log_tail class-attribute instance-attribute

log_tail: deque[str] = field(
    default_factory=lambda: deque(maxlen=_LOG_TAIL_LINES)
)

The child's most recent output lines, kept so failures can explain themselves.

active_streams class-attribute instance-attribute

active_streams: int = 0

Open response streams. Nonzero means busy: the idle clock and eviction ignore it.

last_activity class-attribute instance-attribute

last_activity: float = field(default_factory=time.monotonic)

Monotonic time of the last request or stream release; the idle clock's baseline.

persist class-attribute instance-attribute

persist: bool = False

Exempt this server from idle collection and capacity eviction.

stopping class-attribute instance-attribute

stopping: bool = False

Set while the supervisor is tearing this server down.

base_url property

base_url: str

The OpenAI-compatible base URL this server serves.

is_running property

is_running: bool

Whether the child process is still alive.

is_idle property

is_idle: bool

Whether no request is currently streaming from this server.

idle_seconds

idle_seconds() -> float

How long this server has been idle. Zero while any stream is active.

touch

touch() -> None

Mark activity, resetting the idle clock.

take_load_ms

take_load_ms() -> float | None

Return this server's load duration once, then forget it.

anyinfer.local.LifecycleCallback module-attribute

LifecycleCallback = Callable[[ServerLifecycle], None]

Receives every lifecycle transition of a supervised server.

anyinfer.local.allocate_port

allocate_port(host: str = LOOPBACK_HOST) -> int

Reserve an ephemeral port by binding and immediately releasing it.

Inherently racy, but the alternative — letting llama-server pick and then discovering which port it chose — requires parsing its log output, which is far more fragile.

anyinfer.local.LOOPBACK_HOST module-attribute

LOOPBACK_HOST = '127.0.0.1'

Local servers bind loopback only, unless the caller explicitly opts out.

anyinfer.local.is_loopback

is_loopback(base_url: str | None) -> bool

Whether a base URL points at this machine.

Used by two callers with the same underlying question — "is the thing at the other end of this URL running on hardware I can probe?". A remote Ollama daemon answers no, and everything downstream (hardware detection, fit classification, zero-cost pricing) depends on not pretending otherwise.

A URL that cannot be parsed is treated as not loopback, because the safe default is to assume someone else's machine.

Recommendation

anyinfer.local.recommend_alias

recommend_alias(
    hardware: HardwareProfile,
    catalog: TierSource,
    *,
    prefer_accelerated: bool = True,
) -> Recommendation

Recommend the largest catalog tier this machine can comfortably run.

Parameters:

Name	Type	Description	Default
hardware	HardwareProfile	The detected profile.	required
catalog	TierSource	The catalog whose aliases carry min_ram_bytes/min_vram_bytes.	required
prefer_accelerated	bool	Budget against VRAM when an accelerator is present. With unified memory, system RAM is the budget regardless.	True

Returns:

Type	Description
Recommendation	A recommendation. When memory is unknown, the smallest tier is proposed with
Recommendation	confident=False rather than guessing upward.

anyinfer.local.Recommendation dataclass

Recommendation(
    alias: str | None, reason: str, confident: bool = True
)

A recommended tier and the reasoning behind it.

Attributes:

Name	Type	Description
alias	str | None	The recommended alias, or None when nothing fits.
reason	str	Why this tier was chosen, for display to a user.
confident	bool	False when the machine's memory could not be determined, so the recommendation is a floor rather than a fit.

anyinfer.local.Tier

Bases: Protocol

The subset of a catalog alias this module needs.

Structural rather than nominal so the local subsystem does not import the catalog: artifacts (local data) are depended on by the catalog, and a reverse dependency here would make that a cycle.

name property

name: str

The alias name.

min_ram_bytes property

min_ram_bytes: int | None

System RAM this tier needs, when stated.

min_vram_bytes property

min_vram_bytes: int | None

Accelerator memory this tier needs, when stated.

anyinfer.local.TierSource

Bases: Protocol

The subset of a catalog needed to recommend a tier.

alias_names

alias_names() -> tuple[str, ...]

Every alias name.

alias

alias(name: str) -> Tier

Look up one alias.

Discovery

What this machine can already use: engines answering on loopback, and credential variables that are actually set. This is what anyinfer init composes into a configuration file.

anyinfer.local.discover async

discover(
    registry: ProviderRegistry,
    *,
    timeout_s: float = _PROBE_TIMEOUT_S,
    probe: bool = True,
    keyring: bool = False,
    environ: Mapping[str, str] | None = None,
    transports: Mapping[str, Any] | None = None,
) -> tuple[DiscoveredProvider, ...]

Report every provider this machine can already use.

Parameters:

Name	Type	Description	Default
registry	ProviderRegistry	Which providers to consider. Endpoints and variables both come from the descriptors it holds, so a registry carrying third-party providers discovers them on equal terms.	required
timeout_s	float	Wall clock for each endpoint probe. Endpoints are probed concurrently, so this bounds the whole endpoint phase rather than summing across it.	_PROBE_TIMEOUT_S
probe	bool	Whether to contact endpoints at all. False restricts discovery to credential evidence, which touches no socket.	True
keyring	bool	Whether to consult the OS credential vault. Off by default: an environment variable is already in this process, while reading a vault can prompt the user to unlock it, so vault evidence is asked for rather than collected for free.	False
environ	Mapping[str, str] | None	Environment to inspect; defaults to this process's.	None
transports	Mapping[str, Any] | None	Test seam — httpx2 transports keyed by provider id, used when building the probe adapter so a test can prove the probe logic without opening a socket.	None

Returns:

Type	Description
DiscoveredProvider	The evidence, endpoint findings first and each in registry order. At most one
...	entry per provider: an engine that is both running and holds a key in the
tuple[DiscoveredProvider, ...]	environment is reported as running, since that is the stronger observation.

Raises:

Type	Description
ConfigError	If keyring=True and the [keyring] extra is not installed. Asked for a vault and unable to open one, reporting "nothing found" would be a lie by omission.

anyinfer.local.DiscoveredProvider dataclass

DiscoveredProvider(
    provider_id: str,
    base_url: str | None,
    evidence: DiscoveryEvidence,
    detail: str,
    models: tuple[str, ...] = (),
    embedding_models: tuple[str, ...] = (),
    credential_key: str = "",
    credential_ref: str = "",
)

A provider found usable on this machine, and the evidence for it.

Attributes:

Name	Type	Description
provider_id	str	Registered id of the provider this evidence is for.
base_url	str | None	The endpoint that answered, or the provider's default when the evidence is a credential rather than a running service. None when the provider has no default endpoint.
evidence	DiscoveryEvidence	What was observed; see DiscoveryEvidence.
detail	str	One line naming the observation, for display — "4 models", "ANTHROPIC_API_KEY set". Never contains a credential value.
models	tuple[str, ...]	Model ids the endpoint listed, when it listed any. Empty for credential evidence, which says nothing about what a provider serves.
embedding_models	tuple[str, ...]	The subset of models the endpoint stamped with the embedding operation (LM Studio and Cohere discovery do this; most providers report generation only, so this is empty far more often than models is). A separate field rather than replacing models — the ids stay the flat list every existing caller expects, and this is additive evidence for a caller that specifically wants to route embedding traffic.
credential_key	str	Setup-field key this credential satisfies ("api_key"), or empty for endpoint evidence.
credential_ref	str	The reference a configuration file should carry for that field — "env://ANTHROPIC_API_KEY", "credential://system/openai-api-key". A reference, never a value: this is the field a config writer copies, and it is built here precisely so no caller is ever tempted to resolve one first.

anyinfer.local.DiscoveryEvidence module-attribute

DiscoveryEvidence = Literal[
    "endpoint", "environment", "credential-store"
]

How a provider was found to be usable.

endpoint A loopback address the provider declares as its default answered a model listing. environment A variable the provider declares as its conventional credential source is set and non-blank. Its value was not read. credential-store A secret is stored in the OS vault under a conventional identifier. Only ever produced when a caller passed keyring=True.

anyinfer.local.endpoint_candidates

endpoint_candidates(
    registry: ProviderRegistry,
) -> tuple[tuple[str, ...], ...]

The loopback endpoints discover would contact, grouped by shared address.

Returns:

Type	Description
tuple[str, ...]	One entry per distinct endpoint, as (base_url, provider_id, provider_id, …)
...	in registry order. Several engines share a port — llamafile, localai,
tuple[tuple[str, ...], ...]	ramalama and llama-swap all default to 8080, so an address that answers
tuple[tuple[str, ...], ...]	cannot be attributed to one of them by probing alone, and grouping is how that
tuple[tuple[str, ...], ...]	stays visible instead of becoming a coin flip.

A caller that wants to tell a user exactly what was contacted reads this; the same grouping is what keeps discover to one request per address.

anyinfer.local.KEYRING_IDENTIFIER_SUFFIX module-attribute

KEYRING_IDENTIFIER_SUFFIX = '-api-key'

Suffix of the vault identifier discovery looks under, after the provider id.

There is no protocol here to follow — a vault entry is whatever someone chose to call it — so discovery looks under two conventional spellings per provider (openai and openai-api-key) and finds nothing otherwise. A caller who named their entry something else writes the credential:// reference themselves; that is one line of configuration, and it is better than a command that rummages through a credential store by prefix.

Engine-Managed Models

anyinfer.PullRequest dataclass

PullRequest(
    model: str,
    base_url: str,
    timeout_s: float = PULL_TIMEOUT_S,
    transport: Any | None = None,
    progress: Callable[[DownloadProgress], None]
    | None = None,
)

What a puller needs to make one model available on one engine.

Attributes:

Name	Type	Description
model	str	The model name in the engine's own namespace ("qwen3:8b").
base_url	str	The engine's endpoint, after defaults and shorthand expansion.
timeout_s	float	Wall clock for the whole transfer.
transport	Any | None	Optional httpx2 transport override, for tests.
progress	Callable[[DownloadProgress], None] | None	Sink for DownloadProgress events, or None for a silent pull.

anyinfer.PullReport dataclass

PullReport(
    model: str,
    already_present: bool = False,
    bytes_transferred: int = 0,
    detail: str = "",
)

What a pull did.

Attributes:

Name	Type	Description
model	str	The model that is now available.
already_present	bool	Whether the engine reported it was already there, so nothing was transferred. Worth distinguishing: "took two seconds" is reassuring when it means already installed and alarming when it means downloaded 8 GB.
bytes_transferred	int	Bytes the engine reported pulling, when it reported any.
detail	str	The engine's final status line.

Confidential Execution Attestation

Tier 3 of the Confidentiality Tiers: whether this host can back an attested-local-execution guarantee, and does it, right now. Advisory detection only; enforcement is anyinfer.providers.confidential_execution.ConfidentialExecutionAdapter, which calls the same function this section documents.

anyinfer.local.confidential_execution_status

confidential_execution_status(
    *,
    backend: Backend,
    model: ResolvedModel | None = None,
    use_cache: bool = True,
    manifest: ModelManifest | None = None,
    vendor_public_key: bytes | None = None,
) -> ConfidentialExecutionStatus

Detect what this box can guarantee for confidential local execution.

Never raises: anything that cannot be determined becomes "not detected," never a guess. Callers — including ConfidentialExecutionAdapter's own fail-closed check — treat the result as advice about a hardware fact, not as a decision.

What this proves, and to whom. This is TEE detection: it reports what the guest kernel believes about its own environment. No attestation quote is generated or verified against AMD's, Intel's, or NVIDIA's roots of trust, so nothing here rules out a lying hypervisor and nothing here is evidence a remote relying party can check. end_to_end=True is a local precondition, not a cryptographic guarantee.

Parameters:

Name	Type	Description	Default
backend	Backend	The selected local backend (used only to know whether this run targets a GPU-accelerated build at all, alongside model).	required
model	ResolvedModel | None	The selected model, when known. Its launch_hints["n_gpu_layers"] determines gpu_offload_required. When None (a caller checking capability before choosing a model), gpu_offload_required is conservatively True unless backend itself is a CPU-only build — a capability check must never over-promise before a model is even chosen.	None
use_cache	bool	Read and write the disk cache for the hardware-detection portion of the result. Overridden by ATTESTATION_CACHE_BYPASS_ENV and ATTESTATION_CACHE_REFRESH_ENV. Tier 4 verification is never cached, regardless of this flag — a swapped model file must be caught on the very next call, not masked by a stale cache entry.	True
manifest	ModelManifest | None	Tier 4 — a vendor-signed provenance.ModelManifest to verify model's weights against. Requires model and vendor_public_key too; ignored otherwise.	None
vendor_public_key	bytes | None	The vendor's Ed25519 public key for manifest's signature.	None

Returns:

Type	Description
ConfidentialExecutionStatus	The detected status.

Raises:

Type	Description
ConfigError	manifest was supplied but the attest extra (pip install anyinfer[attest]) is not installed.

anyinfer.local.ConfidentialExecutionStatus dataclass

ConfidentialExecutionStatus(
    cpu_tee: CpuTeeKind | None,
    gpu_cc_capable: bool,
    gpu_cc_enabled: bool,
    gpu_offload_required: bool,
    end_to_end: bool,
    detail: str,
    model_verified: bool | None = None,
)

What this box can actually guarantee for confidential local execution right now.

Attributes:

Name	Type	Description
cpu_tee	CpuTeeKind | None	Detected CPU TEE, or None.
gpu_cc_capable	bool	The primary detected GPU supports CC mode at all (a hardware fact); False when no GPU was detected.
gpu_cc_enabled	bool	CC mode is actually active in the current driver/runtime configuration, when gpu_cc_capable is True.
gpu_offload_required	bool	Whether the selected model's launch plan offloads to GPU at all; when False, gpu_cc_capable/gpu_cc_enabled do not gate end_to_end — a CPU-only backend has no PCIe bridge to worry about.
end_to_end	bool	The one field most callers branch on — see the module docstring for the exact definition. Advisory, local-only. It is derived from TEE detection (guest device nodes, nvidia-smi output), not from a verified attestation quote, so it is evidence this process can act on and not something a remote party can be asked to trust. A fabricated device node satisfies every probe. Treat it as a precondition for running, never as proof to anyone else, until a quote_verified field exists alongside it.
detail	str	Human-readable why, the same role Backend.detail already plays.
model_verified	bool | None	Tier 4 — whether confidential_execution_status's optional manifest/vendor_public_key arguments were supplied and verified against the running model's weights on disk. None means "not evaluated" (no manifest was supplied), never "failed." This field alone is not a Tier 4 claim — a hash-and-signature check on an unattested host is a real but weaker guarantee; only model_verified is True and end_to_end is True together is the full Tier 4 claim (see provenance.py's module docstring). It is also point-in-time: it describes the weights when this status was computed, not the weights a server later loads. To bind verification to the load itself, pass a WeightsProvenance to LocalServerSupervisor.acquire, which verifies inside the start path.

anyinfer.local.CpuTeeKind module-attribute

CpuTeeKind = Literal['sev-snp', 'tdx', 'nitro', 'sgx']

CPU TEE families this module can detect.

sgx and nitro are detected and reported for completeness — a caller asking "what did you find" deserves the whole answer — but neither is part of the v1 end_to_end claim: SGX's enclave-shaped programming model is not the lift-and-shift story SEV-SNP/TDX give, and Nitro Enclaves have no persistent storage or general networking, so serving a model inside one needs real integration work this module does not attempt to paper over (see the market findings in DESIGN.md §30.4).

anyinfer.local.attestation_cache_path

attestation_cache_path() -> Path

Where the detection cache lives — the same directory hardware.py uses.

anyinfer.local.ATTESTATION_CACHE_BYPASS_ENV module-attribute

ATTESTATION_CACHE_BYPASS_ENV = (
    "ANYINFER_ATTESTATION_CACHE_BYPASS"
)

Set to skip the cache entirely (read and write).

anyinfer.local.ATTESTATION_CACHE_REFRESH_ENV module-attribute

ATTESTATION_CACHE_REFRESH_ENV = (
    "ANYINFER_ATTESTATION_CACHE_REFRESH"
)

Set to ignore a cached result and re-probe, then rewrite the cache.

ConfidentialExecutionAdapter

anyinfer.providers.confidential_execution.ConfidentialExecutionAdapter

ConfidentialExecutionAdapter(
    inner: ProviderAdapter,
    *,
    backend: Backend,
    model: ResolvedModel | None = None,
)

Wraps a local ProviderAdapter, refusing generate() unless attestation succeeds.

Discovery and health pass straight through to the inner adapter unchanged — attestation is a property of execution, not of what models are discoverable or whether the process is reachable at all.

Bind the wrapper to one inner adapter and the backend/model it will attest.

Parameters:

Name	Type	Description	Default
inner	ProviderAdapter	An already-configured local adapter instance to delegate to once attestation succeeds.	required
backend	Backend	The local backend inner runs — passed straight to confidential_execution_status on every generate() call.	required
model	ResolvedModel | None	The selected model, when known; also passed straight through. See confidential_execution_status's own docstring for what a missing model means for the check.	None

list_models async

list_models() -> Sequence[DiscoveredModel]

Delegate to the inner adapter unchanged.

health async

health() -> Health

Delegate to the inner adapter unchanged.

aclose async

aclose() -> None

Delegate to the inner adapter unchanged.

generate async

generate(req: WireRequest) -> AsyncIterator[AdapterEvent]

Attest, then generate — or refuse, and never touch the inner adapter at all.

Raises:

Type	Description
ConfidentialExecutionError	The attested guarantee is not available on this host right now. Carries ConfidentialExecutionStatus.detail so a caller can render why.

Model Provenance Verification (Tier 4)

Whether the model weights actually on disk are the exact artifact a vendor signed (verification only, never signing); see the module docstring for why that boundary is absolute. Only a Tier 4 claim in combination with ConfidentialExecutionStatus.end_to_end; see the Confidentiality Tiers guide.

anyinfer.local.ModelManifest dataclass

ModelManifest(
    model_id: str,
    weight_hash: str,
    vendor_key_id: str,
    signed_at: str,
    signature: bytes,
)

A vendor-signed record of what one set of model weights should hash to.

Attributes:

Name	Type	Description
model_id	str	The vendor's identifier for this model variant.
weight_hash	str	SHA-256 of the weight file (or, for a multi-file snapshot, of the sorted relative_path:sha256 listing — see hash_model_weights), as a hex string.
vendor_key_id	str	Which vendor key this manifest was signed with, for a caller managing more than one registered vendor public key.
signed_at	str	ISO-8601 signing timestamp, for audit trails.
signature	bytes	The vendor's signature over this manifest's canonical payload.

to_dict

to_dict() -> dict[str, Any]

A JSON-safe mapping — the on-disk manifest format.

from_dict classmethod

from_dict(data: dict[str, Any]) -> ModelManifest

Load a manifest previously written by to_dict.

anyinfer.local.hash_model_weights

hash_model_weights(path: Path) -> str

Hash the weights at path, hex-encoded SHA-256.

A single file (the GGUF case) is hashed directly. A directory (an hf_repo snapshot) is hashed as the SHA-256 of a sorted relative_path:sha256\n listing over every file it contains — deterministic regardless of filesystem enumeration order, and sensitive to every file's content, name, and presence.

anyinfer.local.verify_model_manifest

verify_model_manifest(
    manifest: ModelManifest,
    *,
    weights_path: Path,
    vendor_public_key: bytes,
) -> bool

Verify a manifest's signature and that it matches the weights on disk.

Parameters:

Name	Type	Description	Default
manifest	ModelManifest	The vendor-signed manifest to check.	required
weights_path	Path	Where the model weights actually are — re-hashed and compared against manifest.weight_hash; a manifest is never trusted for the hash alone, since that would make the signature pointless.	required
vendor_public_key	bytes	The registered vendor's Ed25519 public key.	required

Returns:

Type	Description
bool	True only when the signature verifies against vendor_public_key and the
bool	recomputed hash of weights_path matches manifest.weight_hash exactly.

Raises:

Type	Description
ConfigError	The attest extra is not installed.

Note

This is a point-in-time answer, not a load-bound one. It reports what was on disk when it was read, and says nothing about what a loader opens afterwards — the gap between the two is however long the caller makes it.

For anything that is about to load the weights, use open_verified_weights, or pass a WeightsProvenance to LocalServerSupervisor.acquire, which verifies inside the start path and re-confirms file identity in the instant before the process is spawned. This function remains the right call for reporting on weights nobody is loading right now — which is exactly what confidential_execution_status does with it.

anyinfer.local.WeightsProvenance dataclass

WeightsProvenance(
    manifest: ModelManifest, vendor_public_key: bytes
)

A signed manifest plus the key that checks it, carried to the point of load.

Exists so a caller can hand the whole provenance requirement to whatever starts the server, instead of verifying somewhere else and hoping the two agree. Passing this to LocalServerSupervisor.acquire is what makes verification and load one operation.

Attributes:

Name	Type	Description
manifest	ModelManifest	The vendor-signed manifest describing the expected weights.
vendor_public_key	bytes	The vendor's Ed25519 public key.

anyinfer.local.open_verified_weights

open_verified_weights(
    provenance: WeightsProvenance, weights_path: Path
) -> Iterator[VerifiedWeights]

Verify weights and keep them pinned for the length of the block.

The difference from verify_model_manifest is when and from what. This opens every file first and hashes through those descriptors, so the digest describes the bytes of specific inodes rather than of whatever the path named at the time. The descriptors stay open for the block, and VerifiedWeights.assert_unchanged re-checks the paths still resolve to them — so a caller can verify, do its remaining setup, and re-confirm identity in the instant before it starts a loader.

Parameters:

Name	Type	Description	Default
provenance	WeightsProvenance	The manifest and the key to check it with.	required
weights_path	Path	A GGUF file, or a snapshot directory.	required

Yields:

Type	Description
VerifiedWeights	A VerifiedWeights whose descriptors are open for the duration.

Raises:

Type	Description
ConfidentialExecutionError	The signature does not verify, or the weights do not match the manifest. Fails closed — nothing should be loaded.
ConfigError	The attest extra is missing, or the path is absent.

Note

This closes the swap-at-the-path window, not every window. llama-server opens the path itself, so the microseconds between assert_unchanged and that open are not covered, and because llama.cpp maps weights lazily, a writer with access to the same inode can still alter pages that have not been faulted in yet. Both residuals need the bytes to be immutable during load — a read-only mount, or a directory only root can write — which is a property of the deployment, not of this function. See verify_model_manifest for the wider note.

anyinfer.local.VerifiedWeights

VerifiedWeights(
    digest: str,
    identities: tuple[_FileIdentity, ...],
    fds: tuple[int, ...],
)

Weights that have been verified, with the descriptors still open.

Holding the descriptors matters for a reason that is easy to miss: inode numbers are recycled. If verification only recorded (dev, ino) and closed the file, an attacker could delete it and create a replacement that happens to be assigned the same inode number, and an identity check would pass. An open descriptor keeps that inode alive, so its number cannot be handed to anything else while this object lives.

Instances come from open_verified_weights and are not constructed directly.

digest property

digest: str

The hex SHA-256 that was verified, read from the open descriptors.

assert_unchanged

assert_unchanged() -> None

Re-resolve every path and confirm it still names the bytes we verified.

Call immediately before handing the path to a loader. Catches the whole swap-at-the-path class — rename over, delete and recreate, repointed symlink — because a replacement is a different inode no matter how identical it looks.

Raises:

Type	Description
ConfidentialExecutionError	A path now resolves somewhere else, or has gone missing. Fails closed: the caller must not load.

close

close() -> None

Release the descriptors. The identity guarantee ends here.