Local inference¶

The anyinfer.local subsystem: hardware detection, backend selection, runtime acquisition, tuning, fit classification, model acquisition and storage, server supervision, and hardware→tier recommendation. Concepts: the local subsystem · the model catalog · acquiring models · guides: run a model locally · choose and download a local model.

from anyinfer import local

Hardware¶

anyinfer.local.detect ¶

detect(*, use_cache: bool = True) -> HardwareProfile

Detect this machine's hardware.

Never raises: anything that could not be determined becomes a warning and a None field. Callers treat the result as advice, not as fact.

Parameters:

Name	Type	Description	Default
`use_cache`	`bool`	Read and write the disk cache. Overridden by `CACHE_BYPASS_ENV` and `CACHE_REFRESH_ENV`.	`True`

Returns:

Type	Description
`HardwareProfile`	The detected profile.

anyinfer.local.HardwareProfile `dataclass` ¶

HardwareProfile(
    os_name: str,
    arch: str,
    total_ram_bytes: int | None = None,
    available_ram_bytes: int | None = None,
    cpu_name: str | None = None,
    physical_cores: int | None = None,
    logical_cores: int | None = None,
    accelerators: tuple[Accelerator, ...] = (),
    warnings: tuple[str, ...] = (),
    detected_at: float = 0.0,
)

What we could learn about this machine.

Every field may be None: absence means "not determined", never "zero".

Attributes:

Name	Type	Description
`os_name`	`str`	`"windows"`, `"linux"`, `"darwin"`, or the raw platform string.
`arch`	`str`	Machine architecture as reported by `platform.machine()`.
`total_ram_bytes`	`int \| None`	Physical RAM.
`available_ram_bytes`	`int \| None`	RAM currently free, when the platform reports it.
`cpu_name`	`str \| None`	Processor model string.
`physical_cores`	`int \| None`	Physical core count, preferred for thread tuning.
`logical_cores`	`int \| None`	Logical processor count.
`accelerators`	`tuple[Accelerator, ...]`	Detected accelerators, strongest first.
`warnings`	`tuple[str, ...]`	Everything that could not be determined, and why.
`detected_at`	`float`	Unix timestamp of the probe, for cache display.

primary_accelerator `property` ¶

primary_accelerator: Accelerator | None

The accelerator a server should target, or None for CPU-only.

has_accelerator `property` ¶

has_accelerator: bool

Whether any non-CPU accelerator was detected.

total_vram_bytes `property` ¶

total_vram_bytes: int | None

Total memory of the primary accelerator, when known.

user_supplied `property` ¶

user_supplied: bool

Whether this profile came from from_user_input rather than a probe.

from_user_input `classmethod` ¶

from_user_input(
    *,
    ram_gb: float | None = None,
    vram_gb: float | None = None,
    accelerator: AcceleratorKind | None = None,
    accelerator_name: str | None = None,
    compute_capability: str | None = None,
    os_name: str = "",
    arch: str = "",
) -> HardwareProfile

Build a profile from specs a person supplied, in familiar units.

The remote-Ollama case: local probing describes the wrong machine, and no Ollama API reports its host's specs, so asking the user is the only honest source. Values arrive in gigabytes because that is what a user reads off a spec sheet; anything left out stays None and keeps its "not determined" meaning.

The profile is marked as self-reported in warnings, so any advice derived from it can say so.

to_json ¶

to_json() -> dict[str, Any]

Serialize for the disk cache.

from_json `classmethod` ¶

from_json(data: dict[str, Any]) -> HardwareProfile

Deserialize from the disk cache.

anyinfer.local.Accelerator `dataclass` ¶

Accelerator(
    kind: AcceleratorKind,
    name: str | None = None,
    total_vram_bytes: int | None = None,
    free_vram_bytes: int | None = None,
    unified_memory: bool = False,
    compute_capability: str | None = None,
    driver_version: str | None = None,
)

One detected accelerator.

Attributes:

Name	Type	Description
`kind`	`AcceleratorKind`	Which runtime family can drive it.
`name`	`str \| None`	Human-readable device name, when reported.
`total_vram_bytes`	`int \| None`	Total device memory, or `None` when unknown.
`free_vram_bytes`	`int \| None`	Free device memory at probe time, or `None`.
`unified_memory`	`bool`	True when device memory is shared with system RAM (Apple Silicon), which makes VRAM budgeting a different calculation entirely.
`compute_capability`	`str \| None`	NVIDIA compute capability as reported by the driver (`"8.9"`), or `None`. Quantized kernels gate on this — FP8 needs 8.9, the Marlin GPTQ kernel needs 8.0 — and an unknown capability must exclude a gated variant rather than optimistically permit it.
`driver_version`	`str \| None`	Vendor driver version string, when reported. Used to check that a downloadable CUDA runtime's toolkit version is supported before installing it.

compute_capability_value `property` ¶

compute_capability_value: float | None

compute_capability as a comparable number, or None when unparseable.

driver_major `property` ¶

driver_major: int | None

The major component of driver_version, or None.

anyinfer.local.AcceleratorKind `module-attribute` ¶

AcceleratorKind = Literal[
    "cuda", "rocm", "metal", "vulkan", "cpu"
]

Accelerator families we can detect and target.

anyinfer.local.probe_signature ¶

probe_signature() -> str

Fingerprint the probe tooling, so the cache invalidates when it changes.

Keyed on the resolved path and mtime of each probe executable plus the interpreter's platform: installing a GPU driver, or moving to different hardware, changes this.

anyinfer.local.cache_path ¶

cache_path() -> Path

Where the detection cache lives.

anyinfer.local.CACHE_BYPASS_ENV `module-attribute` ¶

CACHE_BYPASS_ENV = 'ANYINFER_HARDWARE_CACHE_BYPASS'

Set to skip the cache entirely (read and write).

anyinfer.local.CACHE_REFRESH_ENV `module-attribute` ¶

CACHE_REFRESH_ENV = 'ANYINFER_HARDWARE_CACHE_REFRESH'

Set to ignore a cached result and re-probe, then rewrite the cache.

Backends¶

anyinfer.local.available_backends ¶

available_backends(
    *,
    search_paths: list[Path] | None = None,
    hardware: HardwareProfile | None = None,
    runtime_root: Path | None = None,
    include_runtime_root: bool = True,
) -> list[Backend]

Find installed llama-server binaries, best first.

Three sources, in descending order of how much they can be trusted:

Manifest-validated variants under the well-known runtime root. The manifest states the backend, so nothing is inferred.
Caller-supplied directories, where the backend is guessed from the directory name — a convention, not a fact.
PATH, where the backend is guessed from the hardware, because a binary on PATH says nothing at all about what it was compiled against.

The distinction is recorded in Backend.detail rather than hidden, so a surprising selection can be explained.

Parameters:

Name	Type	Description	Default
`search_paths`	`list[Path] \| None`	Extra directories to search, each expected to hold a runtime variant named after its backend (`.../cuda/llama-server`).	`None`
`hardware`	`HardwareProfile \| None`	Detected hardware, used to label what a found binary can actually drive.	`None`
`runtime_root`	`Path \| None`	Override the well-known runtime root.	`None`
`include_runtime_root`	`bool`	Search the well-known runtime root at all.	`True`

Returns:

Type	Description
`list[Backend]`	Usable backends, ranked. Empty when no binary is found at all.

anyinfer.local.select_backend ¶

select_backend(
    hardware: HardwareProfile,
    *,
    search_paths: list[Path] | None = None,
    runtime_root: Path | None = None,
    include_runtime_root: bool = True,
) -> Backend | None

Pick the best backend this machine can actually use.

A CUDA build on a machine with no NVIDIA device is useless, so the selection is the intersection of what is installed and what the hardware can drive. When the best drivable variant is not the best variant the hardware could theoretically use — a Vulkan build on an NVIDIA card because the CUDA add-on is not installed — the returned Backend.detail says so, which is what turns a silent degradation into a discoverable recommendation.

anyinfer.local.Backend `dataclass` ¶

Backend(
    kind: AcceleratorKind,
    binary: Path,
    rank: int = 0,
    detail: str = "",
)

One usable llama.cpp runtime variant.

Attributes:

Name	Type	Description
`kind`	`AcceleratorKind`	The acceleration family this build targets.
`binary`	`Path`	Path to its `llama-server` executable.
`rank`	`int`	Preference score from `BACKEND_RANK`.
`detail`	`str`	Why this backend was or was not selected.

anyinfer.local.BACKEND_RANK `module-attribute` ¶

BACKEND_RANK: dict[AcceleratorKind, int] = {
    "cuda": 30,
    "metal": 25,
    "rocm": 22,
    "vulkan": 20,
    "cpu": 10,
}

Preference order across runtime variants; higher wins.

Runtime variants¶

AnyInfer ships no llama.cpp binaries. These fetch, validate, and select them; CUDA is an explicit opt-in, never installed on a user's behalf.

anyinfer.local.install_runtime ¶

install_runtime(
    kind: AcceleratorKind | None = None,
    *,
    hardware: HardwareProfile | None = None,
    root: Path | None = None,
    table: RuntimeTable | None = None,
    progress: ProgressCallback | None = None,
    client: Client | None = None,
    force: bool = False,
) -> InstallReport

Fetch, verify, and unpack a llama-server runtime variant.

Parameters:

Name	Type	Description	Default
`kind`	`AcceleratorKind \| None`	Which backend to install. `None` picks `default_runtime_kind`, which is never CUDA.	`None`
`hardware`	`HardwareProfile \| None`	Detected hardware, needed for the default choice and the CUDA gate.	`None`
`root`	`Path \| None`	Runtime root; defaults to `runtime_root()`.	`None`
`table`	`RuntimeTable \| None`	Pinned artifact table; defaults to the bundled one.	`None`
`progress`	`ProgressCallback \| None`	Download progress callback.	`None`
`client`	`Client \| None`	An `httpx2.Client`, for tests or custom transports.	`None`
`force`	`bool`	Install CUDA even when the precondition checks object. Never skips digest verification — only the hardware gate.	`False`

Returns:

Type	Description
`InstallReport`	A report naming the executable to launch.

Raises:

Type	Description
`LocalRuntimeError`	If no build exists for this platform and backend, if a CUDA precondition fails without `force`, or if the archive fails verification or cannot be unpacked.

anyinfer.local.installed_runtimes ¶

installed_runtimes(
    root: Path | None = None, *, build: str | None = None
) -> list[RuntimeManifest]

Every validated runtime variant under the runtime root.

anyinfer.local.remove_runtime ¶

remove_runtime(
    kind: AcceleratorKind, *, root: Path | None = None
) -> bool

Delete an installed runtime variant. Returns whether anything was removed.

anyinfer.local.default_runtime_kind ¶

default_runtime_kind(
    hardware: HardwareProfile | None,
) -> AcceleratorKind

Which variant to install when the caller expresses no preference.

Never CUDA. The vendor-neutral small builds cover every GPU well enough to be useful immediately, and a several-hundred-megabyte download is a decision a user makes, not one a library makes on their behalf:

Apple Silicon → Metal, which needs no vendor runtime and is the native path.
Intel Mac → CPU; llama.cpp's Metal backend targets Apple Silicon.
Windows or Linux with any GPU → Vulkan, which drives NVIDIA, AMD, and Intel alike.
Anything else → CPU.

anyinfer.local.check_cuda_preconditions ¶

check_cuda_preconditions(
    hardware: HardwareProfile, table: RuntimeTable
) -> tuple[tuple[str, ...], tuple[str, ...]]

Return (blocking_reasons, warnings) for installing the CUDA add-on.

The pinned build implies a CUDA toolkit version, which implies a minimum driver and a minimum compute capability. Checking them up front turns "your GPU is too old" into a clear refusal before a 400 MB download instead of an incomprehensible crash at load.

anyinfer.local.install_hint ¶

install_hint(
    hardware: HardwareProfile | None,
    table: RuntimeTable | None = None,
) -> str

A one-line suggestion of which runtime this machine should install.

anyinfer.local.load_runtime_table ¶

load_runtime_table(
    path: Path | None = None,
) -> RuntimeTable

Load the pinned runtime table.

Raises:

Type	Description
`LocalRuntimeError`	If the bundled table is unreadable or malformed. Unlike the probes, this is real data shipped with the package: a broken table is a build defect, not a property of the user's machine.

anyinfer.local.runtime_root ¶

runtime_root() -> Path

Where runtime variants are installed.

Follows the same per-OS data-dir convention as the model directory, so a user who knows where one lives can find the other. Overridable with ANYINFER_RUNTIME_DIR.

anyinfer.local.RuntimeTable `dataclass` ¶

RuntimeTable(
    build: str,
    generated: str = "",
    release_url: str = "",
    cuda_toolkit: str = "",
    min_cuda_driver_major: int = 0,
    min_compute_capability: float = 0.0,
    warn_below_vram_bytes: int = 0,
    artifacts: tuple[RuntimeArtifact, ...] = (),
)

The pinned set of fetchable runtime builds.

Attributes:

Name	Type	Description
`build`	`str`	The llama.cpp release tag every variant here comes from.
`generated`	`str`	When the table was pinned.
`release_url`	`str`	The upstream release page.
`cuda_toolkit`	`str`	CUDA version the pinned CUDA build links against.
`min_cuda_driver_major`	`int`	Driver major version that toolkit requires.
`min_compute_capability`	`float`	Lowest GPU compute capability the build supports.
`warn_below_vram_bytes`	`int`	Below this, CUDA works but is not worth the download.
`artifacts`	`tuple[RuntimeArtifact, ...]`	Every pinned variant.

for_platform ¶

for_platform(
    key: str | None = None,
) -> tuple[RuntimeArtifact, ...]

Variants available for one platform, best backend first.

artifact ¶

artifact(
    backend: str, *, key: str | None = None
) -> RuntimeArtifact | None

One variant by backend, for this platform.

anyinfer.local.RuntimeArtifact `dataclass` ¶

RuntimeArtifact(
    platform: str,
    backend: AcceleratorKind,
    filename: str,
    url: str,
    sha256: str,
    size_bytes: int | None = None,
    companions: tuple[RuntimeArtifact, ...] = (),
)

One downloadable runtime archive.

Attributes:

Name	Type	Description
`platform`	`str`	Platform key this build targets (`"win32-amd64"`).
`backend`	`AcceleratorKind`	Acceleration family the build was compiled for.
`filename`	`str`	Archive file name.
`url`	`str`	Where to fetch it.
`sha256`	`str`	Expected digest of the archive.
`size_bytes`	`int \| None`	Expected archive size.
`companions`	`tuple[RuntimeArtifact, ...]`	Extra archives unpacked into the same directory — the CUDA runtime libraries ship separately from the llama.cpp build.

total_bytes `property` ¶

total_bytes: int

Bytes to transfer including companions.

anyinfer.local.RuntimeManifest `dataclass` ¶

RuntimeManifest(
    backend: AcceleratorKind,
    build: str,
    architecture: str,
    executable: Path,
    directory: Path,
)

A validated runtime.json describing an installed variant.

Attributes:

Name	Type	Description
`backend`	`AcceleratorKind`	The acceleration family this build targets.
`build`	`str`	The llama.cpp build id it was cut from.
`architecture`	`str`	The machine architecture it runs on.
`executable`	`Path`	Absolute path to `llama-server`.
`directory`	`Path`	The variant directory.

anyinfer.local.InstallReport `dataclass` ¶

InstallReport(
    backend: AcceleratorKind,
    build: str,
    directory: Path,
    executable: Path,
    downloaded_bytes: int = 0,
    reused: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of installing a runtime variant.

Attributes:

Name	Type	Description
`backend`	`AcceleratorKind`	Which variant was installed.
`build`	`str`	The build id it came from.
`directory`	`Path`	Where it landed.
`executable`	`Path`	The `llama-server` inside it.
`downloaded_bytes`	`int`	Bytes actually transferred (zero when everything was cached).
`reused`	`bool`	Whether an already-valid install was kept.
`warnings`	`tuple[str, ...]`	Non-blocking notes, including CUDA precondition warnings.

Tuning¶

anyinfer.local.plan_server ¶

plan_server(
    hardware: HardwareProfile,
    model: TuningInputs,
    *,
    posture: Posture = "balanced",
) -> ServerPlan

Derive a server plan from hardware, model facts, and posture.

Parameters:

Name	Type	Description	Default
`hardware`	`HardwareProfile`	The detected profile. Unknown fields make the plan more conservative, never more optimistic.	required
`model`	`TuningInputs`	What is known about the model being served.	required
`posture`	`Posture`	How much of the machine to commit.	`'balanced'`

Returns:

Type	Description
`ServerPlan`	A plan whose memory estimate the caller can check before spawning.

anyinfer.local.TuningInputs `dataclass` ¶

TuningInputs(
    artifact_size_bytes: int | None = None,
    parameter_size: str | None = None,
    max_context: int | None = None,
    requested_context: int | None = None,
)

What the tuner needs to know about the model being served.

Attributes:

Name	Type	Description
`artifact_size_bytes`	`int \| None`	On-disk size of the weights; they must be resident too.
`parameter_size`	`str \| None`	Parameter class (`"7B"`), keying the KV-cost table.
`max_context`	`int \| None`	Upper bound from the model itself, when known.
`requested_context`	`int \| None`	An explicit context the caller wants, overriding the ladder.

anyinfer.local.ServerPlan `dataclass` ¶

ServerPlan(
    context_size: int,
    parallel: int = 1,
    threads: int = 4,
    batch_size: int = 512,
    ubatch_size: int = 128,
    gpu_layers: int = 0,
    cache_type_k: str = "f16",
    cache_type_v: str = "f16",
    flash_attention: bool = False,
    estimated_kv_bytes: int = 0,
    estimated_total_bytes: int = 0,
    posture: Posture = "balanced",
    rationale: tuple[str, ...] = (),
)

A concrete, explainable llama-server configuration.

Attributes:

Name	Type	Description
`context_size`	`int`	`--ctx-size`, the total context across all slots.
`parallel`	`int`	`--parallel`, concurrent request slots.
`threads`	`int`	`--threads` for CPU work.
`batch_size`	`int`	`--batch-size`.
`ubatch_size`	`int`	`--ubatch-size`.
`gpu_layers`	`int`	`--n-gpu-layers`; `0` means CPU-only.
`cache_type_k`	`str`	`--cache-type-k`.
`cache_type_v`	`str`	`--cache-type-v`.
`flash_attention`	`bool`	Whether to request flash attention.
`estimated_kv_bytes`	`int`	Predicted KV-cache footprint, for admission control.
`estimated_total_bytes`	`int`	Weights plus KV cache.
`posture`	`Posture`	The posture this plan was derived under.
`rationale`	`tuple[str, ...]`	Human-readable notes explaining the choices.

context_per_slot `property` ¶

context_per_slot: int

Usable context for a single request.

server_arguments ¶

server_arguments(
    model_path: str, *, host: str, port: int
) -> list[str]

Render the plan as llama-server CLI arguments.

--jinja is always on: without it llama-server cannot apply a model's chat template, and tool calling silently does not work at all.

anyinfer.local.Posture `module-attribute` ¶

Posture = Literal['conservative', 'balanced', 'aggressive']

How much of the machine the user is willing to spend on inference.

anyinfer.local.CONTEXT_LADDER `module-attribute` ¶

CONTEXT_LADDER: tuple[int, ...] = (
    8192,
    16384,
    32768,
    65536,
)

Context sizes to consider, smallest first. The largest that fits wins.

anyinfer.local.kv_bytes_per_token ¶

kv_bytes_per_token(
    parameter_size: str | None, cache_type: str
) -> int

Estimate KV-cache bytes per token for a model class and cache precision.

Fit and variant selection¶

Whether a model will run on this machine, and which quantization to acquire for it. Both are advisory and both explain themselves.

anyinfer.local.classify_fit ¶

classify_fit(
    entry: SizedEntry,
    hardware: HardwareProfile | None,
    *,
    posture: Posture = "balanced",
    backend: Backend | None = None,
) -> ModelFit

Classify one catalog entry against a machine.

Parameters:

Name	Type	Description	Default
`entry`	`SizedEntry`	The catalog model, with its stored memory estimates.	required
`hardware`	`HardwareProfile \| None`	The profile to budget against. `None` — the remote-host case — always yields `unknown`, because guessing someone else's machine is not advice.	required
`posture`	`Posture`	How much of the machine to commit; matches the tuner's postures.	`'balanced'`
`backend`	`Backend \| None`	The runtime variant that would actually drive this. Used only to surface the upgrade path when a faster one is available but not installed.	`None`

Returns:

Type	Description
`ModelFit`	A fit level with reasons. Never raises.

anyinfer.local.ModelFit `dataclass` ¶

ModelFit(
    level: FitLevel,
    reasons: tuple[str, ...] = (),
    headroom_bytes: int | None = None,
)

How a model relates to a machine's memory.

Attributes:

Name	Type	Description
`level`	`FitLevel`	The classification.
`reasons`	`tuple[str, ...]`	Human-readable notes, mirroring `ServerPlan.rationale`. Always non-empty.
`headroom_bytes`	`int \| None`	Budget minus requirement for the level that was chosen; negative when nothing fit, `None` when the numbers were unknown.

runnable `property` ¶

runnable: bool

Whether this machine can plausibly run the model at all.

rank `property` ¶

rank: int

Sort key for best-fit-first ordering; higher is better.

anyinfer.local.FitLevel `module-attribute` ¶

FitLevel = Literal['gpu', 'cpu', 'tight', 'no', 'unknown']

How well a model fits: fully offloaded, CPU-resident, marginal, impossible, or unknown.

anyinfer.local.memory_budget ¶

memory_budget(
    hardware: HardwareProfile,
    *,
    posture: Posture = "balanced",
) -> tuple[int | None, int | None]

Return (vram_budget, ram_budget) in bytes for a posture.

None means "not determinable". Free memory is preferred over total when the platform reported it, because a device already hosting a desktop compositor does not have its nameplate VRAM available. Unified memory reports no separate VRAM budget: it is the RAM budget, and counting it twice is how a plan overcommits an Apple Silicon machine.

anyinfer.local.sort_by_fit ¶

sort_by_fit(
    pairs: Sequence[tuple[_EntryT, ModelFit]],
) -> list[tuple[_EntryT, ModelFit]]

Order entries best-fit-first, then by descending headroom, then by id.

Ties broken deterministically so the same catalog and the same machine always produce the same listing — a browsing UI that reshuffles between calls is unusable. Generic in the entry type so a caller keeps whatever it put in, rather than having its rows widened to the protocol.

anyinfer.local.SizedEntry ¶

Bases: Protocol

The subset of a catalog model this module needs.

id `property` ¶

id: str

Catalog model id.

parameter_size `property` ¶

parameter_size: str | None

Parameter class ("7B"), when stated.

est_ram_bytes `property` ¶

est_ram_bytes: int | None

Memory needed on the CPU-only path.

est_vram_bytes `property` ¶

est_vram_bytes: int | None

Memory needed when fully offloaded.

anyinfer.local.select_variant ¶

select_variant(
    variants: Sequence[SelectableVariant],
    hardware: HardwareProfile | None,
    *,
    engine: str | None = None,
    parameter_size: str | None = None,
    backend: Backend | None = None,
    prefs: VariantPrefs | None = None,
) -> VariantChoice | None

Choose the best quantization this machine can actually run.

Parameters:

Name	Type	Description	Default
`variants`	`Sequence[SelectableVariant]`	The model's ladder, in any order.	required
`hardware`	`HardwareProfile \| None`	The machine to budget against. `None` yields `None`: guessing a quantization for an unknown machine is exactly the kind of confident wrong answer this module exists to avoid.	required
`engine`	`str \| None`	Restrict to one engine's variants.	`None`
`parameter_size`	`str \| None`	Parameter class, for the KV-cache cost.	`None`
`backend`	`Backend \| None`	The runtime that would drive this, used to surface an upgrade path.	`None`
`prefs`	`VariantPrefs \| None`	Selection preferences.	`None`

Returns:

Type	Description
`VariantChoice \| None`	The chosen rung, or `None` when nothing acceptable fits. Use
`VariantChoice \| None`	`evaluate_variants` when the rejection reasons matter too.

anyinfer.local.evaluate_variants ¶

evaluate_variants(
    variants: Sequence[SelectableVariant],
    hardware: HardwareProfile | None,
    *,
    engine: str | None = None,
    parameter_size: str | None = None,
    backend: Backend | None = None,
    prefs: VariantPrefs | None = None,
) -> tuple[
    VariantChoice | None, tuple[tuple[str, str], ...]
]

Choose a rung and return why every other rung was passed over.

The rejections matter even — especially — when nothing was chosen: "no quantization fits" is only useful advice if it comes with the numbers behind it. select_variant is the convenience wrapper for callers that only need the choice.

Returns:

Type	Description
`VariantChoice \| None`	`(choice, rejections)`. `choice` is `None` when nothing acceptable fits,
`tuple[tuple[str, str], ...]`	which is a real answer, not a failure — the caller should then offer a smaller
`tuple[VariantChoice \| None, tuple[tuple[str, str], ...]]`	model.

anyinfer.local.VariantChoice `dataclass` ¶

VariantChoice(
    variant_id: str,
    quantization: str,
    fit: ModelFit,
    engine: str = "llama.cpp",
    est_file_bytes: int | None = None,
    reasons: tuple[str, ...] = (),
    rejected: tuple[tuple[str, str], ...] = (),
    tensor_parallel_size: int = 1,
    gpu_memory_utilization: float | None = None,
)

The chosen rung, and why the others were not.

Attributes:

Name	Type	Description
`variant_id`	`str`	The chosen variant.
`quantization`	`str`	What it ships.
`fit`	`ModelFit`	How it fits this machine.
`engine`	`str`	Which engine it targets.
`est_file_bytes`	`int \| None`	What it will cost to download.
`reasons`	`tuple[str, ...]`	Why this rung, and why not the next one up.
`rejected`	`tuple[tuple[str, str], ...]`	`(variant_id, why not)` for every rung that was passed over.
`tensor_parallel_size`	`int`	How many devices a vLLM launch should span.
`gpu_memory_utilization`	`float \| None`	The utilization the budget assumed.

anyinfer.local.VariantPrefs `dataclass` ¶

VariantPrefs(
    posture: Posture = "balanced",
    context: int = _DEFAULT_CONTEXT,
    allow_low_quality: bool = False,
    allow_multi_gpu: bool = False,
    gpu_memory_utilization: float = 0.9,
    max_download_bytes: int | None = None,
)

How aggressively to choose.

Attributes:

Name	Type	Description
`posture`	`Posture`	Memory posture, matching the tuner's.
`context`	`int`	Context length to budget the KV cache for.
`allow_low_quality`	`bool`	Permit rungs below Q4_K_M.
`allow_multi_gpu`	`bool`	Let vLLM sum VRAM across identical devices, and emit a `tensor_parallel_size` hint. llama.cpp never sums by default, because its split is layer-wise and much easier to get wrong.
`gpu_memory_utilization`	`float`	vLLM's fraction of device memory to plan against.
`max_download_bytes`	`int \| None`	Refuse variants larger than this, whatever fits in memory.

Acquisition and the model store¶

Getting weights onto this disk and finding them again. Model acquisition lives here, never in a provider adapter.

anyinfer.local.acquire ¶

Getting model weights onto this disk: plan → preflight → fetch → verify → place.

One engine for both artifact shapes. A GGUF variant is a file set whose handle is the first shard; a Hugging Face snapshot is a directory whose handle is the directory. The file list and the handle differ; the machinery — resume, digest verification, locking, cancellation, and progress accounting — does not, and those are exactly the things that are expensive to get right twice.

Progress is reported for the whole acquisition, not per file. A sharded artifact whose byte counter restarts at zero on every shard is worse than no progress bar, because a user cannot tell a restart from a stall. AcquisitionProgress therefore carries aggregate totals, counts bytes that were already on disk, and knows its full size before the first byte arrives — the payoff for pinning sizes in the catalog.

AcquisitionPhase `module-attribute` ¶

AcquisitionPhase = Literal[
    "resolving",
    "planning",
    "downloading",
    "verifying",
    "placing",
    "done",
]

Where an acquisition is. Every transition produces a callback, unthrottled.

ProgressSink `module-attribute` ¶

ProgressSink = Callable[[AcquisitionProgress], None]

Receives AcquisitionProgress. See its docstring for the threading contract.

AcquisitionProgress `dataclass` ¶

AcquisitionProgress(
    model_id: str,
    variant_id: str,
    phase: AcquisitionPhase,
    file_index: int = 0,
    file_count: int = 0,
    filename: str = "",
    file_downloaded_bytes: int = 0,
    file_total_bytes: int | None = None,
    total_downloaded_bytes: int = 0,
    total_bytes: int | None = None,
    total_is_estimate: bool = False,
    session_bytes: int = 0,
    bytes_per_second: float | None = None,
    eta_seconds: float | None = None,
)

One progress report for a whole acquisition.

The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run.

Attributes:

Name	Type	Description
`model_id`	`str`	The catalog model being acquired.
`variant_id`	`str`	The variant being acquired.
`phase`	`AcquisitionPhase`	Which stage this report came from.
`file_index`	`int`	1-based index of the file that most recently advanced.
`file_count`	`int`	How many files this acquisition covers.
`filename`	`str`	Name of the file that most recently advanced.
`file_downloaded_bytes`	`int`	Bytes present for that file, including a resumed prefix.
`file_total_bytes`	`int \| None`	Expected size of that file, when known.
`total_downloaded_bytes`	`int`	Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%.
`total_bytes`	`int \| None`	Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes.
`total_is_estimate`	`bool`	True when any file's size came from a guess rather than a pinned or reported figure.
`session_bytes`	`int`	Bytes this run actually transferred, which is what rate and ETA are derived from.
`bytes_per_second`	`float \| None`	Transfer rate, or `None` until there is a real sample.
`eta_seconds`	`float \| None`	Seconds remaining, or `None` on the same condition. A wildly wrong ETA in the first second is worse than no ETA.

fraction `property` ¶

fraction: float | None

Completion as a fraction, or None when the total is unknown.

AcquisitionPlan `dataclass` ¶

AcquisitionPlan(
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str,
    engine: str,
    quantization: str,
    directory: str,
    handle: str,
    files: tuple[RemoteFile, ...],
    already_have_bytes: int = 0,
    total_bytes: int | None = None,
    warnings: tuple[str, ...] = (),
    satisfied: bool = False,
    revision: str | None = None,
    repo: str | None = None,
    license: str = "",
)

What an acquisition will do, before it does any of it.

Attributes:

Name	Type	Description
`entry_id`	`str`	The store entry that will be written.
`model_id`	`str`	The catalog model.
`variant_id`	`str`	The catalog variant.
`kind`	`str`	`"gguf"` or `"hf_repo"`.
`engine`	`str`	Which engine the result is for.
`quantization`	`str`	What will be on disk.
`directory`	`str`	Store-relative destination.
`handle`	`str`	Store-relative engine handle.
`files`	`tuple[RemoteFile, ...]`	Every file, resolved.
`already_have_bytes`	`int`	Bytes already on disk — verified files plus resumable `.part` prefixes.
`total_bytes`	`int \| None`	Total size, or `None` when any file's size is unknown.
`warnings`	`tuple[str, ...]`	Notes from resolution.
`satisfied`	`bool`	True when everything is already present and verified.

remaining_bytes `property` ¶

remaining_bytes: int | None

Bytes still to transfer, or None when the total is unknown.

total_is_estimate `property` ¶

total_is_estimate: bool

Whether any file's size was unknown, making the total a floor.

AcquisitionReport `dataclass` ¶

AcquisitionReport(
    plan: AcquisitionPlan,
    entry: StoreEntry | None = None,
    downloaded_bytes: int = 0,
    reused: bool = False,
    cancelled: bool = False,
    dry_run: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of an acquisition.

Attributes:

Name	Type	Description
`plan`	`AcquisitionPlan`	What was planned.
`entry`	`StoreEntry \| None`	The registered store entry, or `None` for a dry run or a cancellation.
`downloaded_bytes`	`int`	Bytes transferred by this run.
`reused`	`bool`	True when nothing had to be transferred.
`cancelled`	`bool`	True when the caller stopped it. Partial transfers are kept.
`dry_run`	`bool`	True when nothing was written.
`warnings`	`tuple[str, ...]`	Everything the caller should know.

path `property` ¶

path: Path | None

The engine handle, when one was registered.

AcquisitionRequest `dataclass` ¶

AcquisitionRequest(
    ref: SourceRef,
    model_id: str,
    variant_id: str = "",
    kind: str = "gguf",
    engine: str = "llama.cpp",
    quantization: str = "",
    license: str = "",
    token: str | None = None,
    max_concurrent_files: int = 3,
    allow_unverified: bool = False,
    enforce_license: bool = False,
    launch_hints: Mapping[str, Any] = dict(),
)

Everything one acquisition needs.

Attributes:

Name	Type	Description
`ref`	`SourceRef`	Where the bytes come from.
`model_id`	`str`	Catalog model id, for the index and for progress reports.
`variant_id`	`str`	Catalog variant id.
`kind`	`str`	`"gguf"` or `"hf_repo"`.
`engine`	`str`	Which engine the result is for.
`quantization`	`str`	What will be on disk.
`license`	`str`	License id, checked when `enforce_license` is set.
`token`	`str \| None`	Credential for the source, when it needs one.
`max_concurrent_files`	`int`	How many transfers may run at once.
`allow_unverified`	`bool`	Accept files no digest can check.
`enforce_license`	`bool`	Refuse licenses outside the allowlist.
`launch_hints`	`Mapping[str, Any]`	Advisory engine arguments to attach to the result.

plan_acquisition `async` ¶

plan_acquisition(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
) -> AcquisitionPlan

Resolve a source and work out exactly what would be transferred.

Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards.

acquire `async` ¶

acquire(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
    progress: ProgressSink | None = None,
    plan: AcquisitionPlan | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Acquire a model variant into the store.

Parameters:

Name	Type	Description	Default
`request`	`AcquisitionRequest`	What to acquire and from where.	required
`store`	`ModelStore \| None`	The destination store; defaults to the standard root.	`None`
`client`	`AsyncClient \| None`	An `httpx2.AsyncClient`, for tests or custom transports.	`None`
`progress`	`ProgressSink \| None`	Aggregate progress sink.	`None`
`plan`	`AcquisitionPlan \| None`	A plan from `plan_acquisition`, to avoid resolving twice.	`None`
`dry_run`	`bool`	Resolve and report sizes without writing anything.	`False`
`cancel_check`	`Callable[[], bool] \| None`	Polled between chunks; returning `True` stops the acquisition cooperatively. Task cancellation works too and is the primary mechanism — this exists for the synchronous facade.	`None`

Returns:

Type	Description
`AcquisitionReport`	The report, naming the registered entry.

Raises:

Type	Description
`LocalRuntimeError`	On a transfer failure, a digest mismatch, or insufficient disk.

acquire_sync ¶

acquire_sync(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    progress: ProgressSink | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Blocking wrapper around acquire, for callers with no event loop.

Raises:

Type	Description
`RuntimeError`	If called from inside a running event loop, where it would deadlock. Use `acquire` there.

launch_hints_for ¶

launch_hints_for(
    entry: StoreEntry,
    *,
    path: Path,
    context_size: int | None = None,
    gpu_layers: int | None = None,
    tensor_parallel_size: int | None = None,
    gpu_memory_utilization: float | None = None,
) -> dict[str, Any]

Build the advisory engine arguments that accompany a located model.

Data, not process control. These are keys a caller — the llama.cpp supervisor, a future vLLM launcher, or a user pasting a command line — turns into arguments. Producing them from numbers already computed is translation; launching is not, and nothing here starts a process.

anyinfer.local.acquire_sync ¶

acquire_sync(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    progress: ProgressSink | None = None,
    dry_run: bool = False,
    cancel_check: Callable[[], bool] | None = None,
) -> AcquisitionReport

Blocking wrapper around acquire, for callers with no event loop.

Raises:

Type	Description
`RuntimeError`	If called from inside a running event loop, where it would deadlock. Use `acquire` there.

anyinfer.local.plan_acquisition `async` ¶

plan_acquisition(
    request: AcquisitionRequest,
    *,
    store: ModelStore | None = None,
    client: AsyncClient | None = None,
) -> AcquisitionPlan

Resolve a source and work out exactly what would be transferred.

Separated from the transfer so an application can put a real confirmation dialog in front of a forty-gigabyte download instead of discovering the size afterwards.

anyinfer.local.AcquisitionPlan `dataclass` ¶

AcquisitionPlan(
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str,
    engine: str,
    quantization: str,
    directory: str,
    handle: str,
    files: tuple[RemoteFile, ...],
    already_have_bytes: int = 0,
    total_bytes: int | None = None,
    warnings: tuple[str, ...] = (),
    satisfied: bool = False,
    revision: str | None = None,
    repo: str | None = None,
    license: str = "",
)

What an acquisition will do, before it does any of it.

Attributes:

Name	Type	Description
`entry_id`	`str`	The store entry that will be written.
`model_id`	`str`	The catalog model.
`variant_id`	`str`	The catalog variant.
`kind`	`str`	`"gguf"` or `"hf_repo"`.
`engine`	`str`	Which engine the result is for.
`quantization`	`str`	What will be on disk.
`directory`	`str`	Store-relative destination.
`handle`	`str`	Store-relative engine handle.
`files`	`tuple[RemoteFile, ...]`	Every file, resolved.
`already_have_bytes`	`int`	Bytes already on disk — verified files plus resumable `.part` prefixes.
`total_bytes`	`int \| None`	Total size, or `None` when any file's size is unknown.
`warnings`	`tuple[str, ...]`	Notes from resolution.
`satisfied`	`bool`	True when everything is already present and verified.

remaining_bytes `property` ¶

remaining_bytes: int | None

Bytes still to transfer, or None when the total is unknown.

total_is_estimate `property` ¶

total_is_estimate: bool

Whether any file's size was unknown, making the total a floor.

anyinfer.local.AcquisitionProgress `dataclass` ¶

AcquisitionProgress(
    model_id: str,
    variant_id: str,
    phase: AcquisitionPhase,
    file_index: int = 0,
    file_count: int = 0,
    filename: str = "",
    file_downloaded_bytes: int = 0,
    file_total_bytes: int | None = None,
    total_downloaded_bytes: int = 0,
    total_bytes: int | None = None,
    total_is_estimate: bool = False,
    session_bytes: int = 0,
    bytes_per_second: float | None = None,
    eta_seconds: float | None = None,
)

One progress report for a whole acquisition.

The sink may be invoked from a worker thread. It must not block, must not raise, and must not re-enter the client — a progress bar that can deadlock a download is a bug with no upside. A sink that raises anyway is caught, recorded once as a warning on the report, and then dropped for the rest of the run.

Attributes:

Name	Type	Description
`model_id`	`str`	The catalog model being acquired.
`variant_id`	`str`	The variant being acquired.
`phase`	`AcquisitionPhase`	Which stage this report came from.
`file_index`	`int`	1-based index of the file that most recently advanced.
`file_count`	`int`	How many files this acquisition covers.
`filename`	`str`	Name of the file that most recently advanced.
`file_downloaded_bytes`	`int`	Bytes present for that file, including a resumed prefix.
`file_total_bytes`	`int \| None`	Expected size of that file, when known.
`total_downloaded_bytes`	`int`	Bytes present across every file — including what was already on disk, so resuming a 90%-complete transfer reports 90%, not 0%.
`total_bytes`	`int \| None`	Total expected bytes, known before the first byte arrives whenever the catalog or the listing API supplied sizes.
`total_is_estimate`	`bool`	True when any file's size came from a guess rather than a pinned or reported figure.
`session_bytes`	`int`	Bytes this run actually transferred, which is what rate and ETA are derived from.
`bytes_per_second`	`float \| None`	Transfer rate, or `None` until there is a real sample.
`eta_seconds`	`float \| None`	Seconds remaining, or `None` on the same condition. A wildly wrong ETA in the first second is worse than no ETA.

fraction `property` ¶

fraction: float | None

Completion as a fraction, or None when the total is unknown.

anyinfer.local.AcquisitionReport `dataclass` ¶

AcquisitionReport(
    plan: AcquisitionPlan,
    entry: StoreEntry | None = None,
    downloaded_bytes: int = 0,
    reused: bool = False,
    cancelled: bool = False,
    dry_run: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of an acquisition.

Attributes:

Name	Type	Description
`plan`	`AcquisitionPlan`	What was planned.
`entry`	`StoreEntry \| None`	The registered store entry, or `None` for a dry run or a cancellation.
`downloaded_bytes`	`int`	Bytes transferred by this run.
`reused`	`bool`	True when nothing had to be transferred.
`cancelled`	`bool`	True when the caller stopped it. Partial transfers are kept.
`dry_run`	`bool`	True when nothing was written.
`warnings`	`tuple[str, ...]`	Everything the caller should know.

path `property` ¶

path: Path | None

The engine handle, when one was registered.

anyinfer.local.AcquisitionPhase `module-attribute` ¶

AcquisitionPhase = Literal[
    "resolving",
    "planning",
    "downloading",
    "verifying",
    "placing",
    "done",
]

Where an acquisition is. Every transition produces a callback, unthrottled.

anyinfer.local.ProgressSink `module-attribute` ¶

ProgressSink = Callable[[AcquisitionProgress], None]

Receives AcquisitionProgress. See its docstring for the threading contract.

anyinfer.local.ModelStore ¶

ModelStore(root: Path | None = None)

A directory of acquired models, with an index over it.

Not thread-safe by construction; correctness across processes comes from the same cooperative file lock downloads use, taken around every index mutation.

root `property` ¶

root: Path

The store root.

index_path `property` ¶

index_path: Path

Where the index document lives.

staging_dir ¶

staging_dir(entry_id: str) -> Path

Where an in-progress acquisition writes its .part files.

lock_path ¶

lock_path(entry_id: str) -> Path

The cross-process lock guarding one entry.

entry_dir ¶

entry_dir(entry: StoreEntry) -> Path

Absolute path to an entry's directory.

resolve_within ¶

resolve_within(entry: StoreEntry, relative: str) -> Path

Resolve a path inside an entry, refusing anything that escapes it.

Called for every file before it is opened. Names come from a remote API, so this is the containment gate, applied after resolution so a symlink cannot step outside.

Raises:

Type	Description
`ConfigError`	If the path is unsafe or resolves outside the entry directory.

load_index ¶

load_index() -> dict[str, StoreEntry]

Read the index, tolerating absence and corruption.

A store whose index cannot be parsed reports as empty rather than raising: the files are still there, rebuild_index can recover them, and refusing to work at all because a cache file got truncated would be the wrong trade.

register ¶

register(entry: StoreEntry) -> StoreEntry

Add or replace an index entry.

Called only after every file has been verified: a half-complete multi-file set is never registered, which is what makes locate trustworthy.

unregister ¶

unregister(entry_id: str) -> StoreEntry | None

Drop an index entry without touching files.

rebuild_index ¶

rebuild_index() -> list[StoreEntry]

Drop entries whose files are gone, and re-stat the rest.

The recovery path for a user who deleted a directory by hand. It does not invent entries for unknown directories — an unrecognized tree could be anything, and registering it would be a claim about bytes nobody checked.

list_installed ¶

list_installed() -> list[StoreEntry]

Every registered entry, id-ordered.

get ¶

get(entry_id: str) -> StoreEntry | None

One entry by id.

find ¶

find(
    model_id: str,
    *,
    variant_id: str | None = None,
    quantization: str | None = None,
    engine: str | None = None,
) -> StoreEntry | None

The best registered entry matching a model and optional constraints.

"Best" is the most recently installed match, so re-acquiring at a different quantization changes what a bare model id resolves to — which is what a user who just downloaded something expects.

locate ¶

locate(
    model_id: str,
    *,
    variant_id: str | None = None,
    quantization: str | None = None,
    engine: str | None = None,
    verify: bool = False,
    launch_hints: Mapping[str, Any] | None = None,
) -> ResolvedModel | None

Find a stored model and return a path an engine can be launched against.

No network I/O, ever. Verification is the deliberate exception to "always check": hashing forty gigabytes on every request would be absurd, so the rule is verify on install and on adoption, then on lookup compare size and mtime against the index and re-hash only on a mismatch. verify=True forces a full re-hash.

Returns:

Type	Description
`ResolvedModel \| None`	The located model, or `None` when it is absent or fails its check.

check ¶

check(
    entry: StoreEntry, *, deep: bool = False
) -> tuple[str, ...]

Report what is wrong with a stored entry, cheaply by default.

The shallow check compares size and mtime against the index; the deep check re-hashes. Either way an empty result means "as installed".

disk_usage ¶

disk_usage() -> int

Total bytes the registered entries occupy, excluding external ones.

remove ¶

remove(entry_id: str) -> RemovalReport

Delete an entry's files and unregister it.

An external entry — one adopted from somebody else's cache — is only unregistered. Deleting files this store never wrote would be overstepping.

clear_staging ¶

clear_staging(entry_id: str) -> None

Remove an entry's staging directory and its partial transfers.

adopt_legacy_flat ¶

adopt_legacy_flat(
    artifacts: Sequence[Any],
) -> list[StoreEntry]

Register pre-existing flat-layout GGUF files without moving or re-fetching them.

Earlier builds wrote <root>/<filename>.gguf with no revision in the path. Those files are perfectly good bytes a user paid bandwidth for, so the store adopts them where they lie — but only after verifying each one against its catalog hash, so adoption is never a lie about what is on disk.

Parameters:

Name	Type	Description	Default
`artifacts`	`Sequence[Any]`	Pinned catalog artifacts to look for, each with `id` and `files`.	required

Returns:

Type	Description
`list[StoreEntry]`	Newly registered entries.

adopt_external ¶

adopt_external(
    directory: Path,
    *,
    entry_id: str,
    model_id: str,
    variant_id: str,
    kind: str = "hf_repo",
    engine: str = "vllm",
    quantization: str = "",
    source: Mapping[str, Any] | None = None,
    expected: Mapping[str, str] | None = None,
) -> StoreEntry | None

Register a directory this store does not own, if every file checks out.

The Hugging Face cache case. We do not adopt another library's layout for our own writes — it is their private implementation detail — but re-downloading forty gigabytes the user already has is user-hostile. Adopted entries are marked StoreEntry.external, are never written to, and are never deleted by remove.

Parameters:

Name	Type	Description	Default
`directory`	`Path`	The existing snapshot directory.	required
`entry_id`	`str`	Id to register it under.	required
`model_id`	`str`	Catalog model this realizes.	required
`variant_id`	`str`	Catalog variant this realizes.	required
`kind`	`str`	Artifact kind.	`'hf_repo'`
`engine`	`str`	Engine the variant targets.	`'vllm'`
`quantization`	`str`	What is on disk.	`''`
`source`	`Mapping[str, Any] \| None`	Provenance to record.	`None`
`expected`	`Mapping[str, str] \| None`	Per-relative-path sha256 that every file must match. Adoption is refused outright when this is empty — an unverified adoption is a guess.	`None`

Returns:

Type	Description
`StoreEntry \| None`	The registered entry, or `None` when verification failed.

anyinfer.local.StoreEntry `dataclass` ¶

StoreEntry(
    id: str,
    kind: str = "gguf",
    model_id: str = "",
    variant_id: str = "",
    quantization: str = "",
    engine: str = "llama.cpp",
    source: Mapping[str, Any] = dict(),
    directory: str = "",
    handle: str = "",
    files: tuple[StoredFile, ...] = (),
    license: str = "",
    installed_at: float = 0.0,
    last_used_at: float = 0.0,
    external: bool = False,
    warnings: tuple[str, ...] = (),
)

One acquired model in the store.

Attributes:

Name	Type	Description
`id`	`str`	Stable entry id, also the lock and staging name.
`kind`	`str`	`"gguf"` (a file set) or `"hf_repo"` (a directory snapshot).
`model_id`	`str`	The catalog model this realizes, when it came from the catalog.
`variant_id`	`str`	The catalog variant, when it came from the catalog.
`quantization`	`str`	The quantization on disk.
`engine`	`str`	Which engine this variant is for.
`source`	`Mapping[str, Any]`	How it was acquired, including the resolved immutable revision.
`directory`	`str`	Where the files live, relative to the store root.
`handle`	`str`	The path an engine is pointed at, relative to the store root — the first shard for GGUF, the directory for a snapshot.
`files`	`tuple[StoredFile, ...]`	Every file, with digests.
`license`	`str`	License id recorded at acquisition.
`installed_at`	`float`	Unix timestamp of successful registration.
`last_used_at`	`float`	Unix timestamp of the most recent `ModelStore.locate`.
`external`	`bool`	True when the bytes are owned by something else (an adopted Hugging Face cache). Removal only unregisters an external entry; it never deletes.
`warnings`	`tuple[str, ...]`	Anything the user should know — unverified files, most importantly.

total_bytes `property` ¶

total_bytes: int

Bytes this entry occupies.

verified `property` ¶

verified: bool

Whether every file was verified against a digest at install time.

to_json ¶

to_json() -> dict[str, Any]

Serialize for the index.

from_json `classmethod` ¶

from_json(data: Mapping[str, Any]) -> StoreEntry

Parse one index entry.

anyinfer.local.ResolvedModel `dataclass` ¶

ResolvedModel(
    entry_id: str,
    kind: str,
    path: Path,
    quantization: str | None = None,
    engine: str = "llama.cpp",
    verified: bool = False,
    warnings: tuple[str, ...] = (),
    launch_hints: Mapping[str, Any] = dict(),
)

A located model, ready to launch an engine against.

Attributes:

Name	Type	Description
`entry_id`	`str`	The store entry.
`kind`	`str`	`"gguf"` or `"hf_repo"`.
`path`	`Path`	The engine handle — a file for GGUF, a directory for a snapshot.
`quantization`	`str \| None`	What is actually on disk.
`engine`	`str`	Which engine this variant is for.
`verified`	`bool`	Whether every file was verified.
`warnings`	`tuple[str, ...]`	Notes carried from the entry.
`launch_hints`	`Mapping[str, Any]`	Engine-shaped arguments a caller can turn into a command line. Advisory data, not process control: this module locates weights, it does not start servers.

anyinfer.local.RemovalReport `dataclass` ¶

RemovalReport(
    entry_id: str,
    removed: bool = False,
    freed_bytes: int = 0,
    external: bool = False,
)

The outcome of removing an entry.

Attributes:

Name	Type	Description
`entry_id`	`str`	What was removed.
`removed`	`bool`	Whether an entry was found and unregistered.
`freed_bytes`	`int`	Bytes reclaimed; zero for an external entry, which is only unregistered.
`external`	`bool`	Whether the files were left alone because something else owns them.

Sources¶

Where weights come from. Adding an internal mirror is a resolver, not a dependency.

anyinfer.local.SourceRef `dataclass` ¶

SourceRef(
    resolver: str = "huggingface",
    repo: str | None = None,
    revision: str | None = None,
    files: tuple[str, ...] = (),
    digests: Mapping[str, str] = dict(),
    sizes: Mapping[str, int] = dict(),
    urls: tuple[str, ...] = (),
    include: tuple[str, ...] = (),
    exclude: tuple[str, ...] = (),
    path: str | None = None,
)

A declarative pointer at a set of remote (or already-local) files.

Attributes:

Name	Type	Description
`resolver`	`str`	Which resolver understands this reference.
`repo`	`str \| None`	Repository id, for repository-shaped resolvers (`"Qwen/Qwen2.5-7B-Instruct"`).
`revision`	`str \| None`	Branch, tag, or — preferably — an immutable commit sha.
`files`	`tuple[str, ...]`	Explicit file list. Empty means "whatever the include globs match".
`digests`	`Mapping[str, str]`	Per-file expected sha256, when the catalog pinned them.
`sizes`	`Mapping[str, int]`	Per-file expected byte counts, when the catalog pinned them.
`urls`	`tuple[str, ...]`	Direct download URLs, for the `url` resolver.
`include`	`tuple[str, ...]`	Glob patterns selecting files from a repository listing.
`exclude`	`tuple[str, ...]`	Glob patterns removing files the include list matched.
`path`	`str \| None`	An existing on-disk location, for the `local` resolver.

to_json ¶

to_json() -> dict[str, object]

Serialize for the store index.

anyinfer.local.SourceResolver ¶

Bases: Protocol

Turns a SourceRef into a ResolvedArtifact.

scheme `instance-attribute` ¶

scheme: str

The SourceRef.resolver value this implementation answers to.

resolve `async` ¶

resolve(
    ref: SourceRef,
    *,
    token: str | None = None,
    client: Any | None = None,
) -> ResolvedArtifact

Expand a reference into a concrete file list.

client is an optional httpx2.AsyncClient the caller already owns. Passing it keeps resolution and the subsequent transfer on one connection pool — and is what makes a resolver testable against a mock transport.

anyinfer.local.ResolvedArtifact `dataclass` ¶

ResolvedArtifact(
    resolver: str,
    files: tuple[RemoteFile, ...] = (),
    repo: str | None = None,
    revision: str | None = None,
    warnings: tuple[str, ...] = (),
)

A concrete, ordered file list ready for acquisition.

Attributes:

Name	Type	Description
`resolver`	`str`	Which resolver produced this.
`files`	`tuple[RemoteFile, ...]`	Every file to fetch, in acquisition order.
`repo`	`str \| None`	The repository this came from, when applicable.
`revision`	`str \| None`	The immutable revision this resolved to, when the resolver could determine one. A branch name is always resolved to a commit before use.
`warnings`	`tuple[str, ...]`	Anything the caller should know — unverifiable files, skipped pickle weights, and so on.

total_bytes `property` ¶

total_bytes: int | None

Sum of every file size, or None when any one is unknown.

anyinfer.local.RemoteFile `dataclass` ¶

RemoteFile(
    path: str,
    url: str,
    size_bytes: int | None = None,
    digest: str = "",
    digest_kind: DigestKind = "none",
)

One file a resolver decided belongs to an artifact.

Attributes:

Name	Type	Description
`path`	`str`	Destination path relative to the entry directory, POSIX-separated.
`url`	`str`	Where to fetch it.
`size_bytes`	`int \| None`	Expected size, or `None` when genuinely unknown.
`digest`	`str`	Expected digest, lowercase hex.
`digest_kind`	`DigestKind`	How to compute `digest`. `"none"` means unverifiable.

filename `property` ¶

filename: str

The final path component.

Artifacts and downloads¶

anyinfer.local.GgufArtifact `dataclass` ¶

GgufArtifact(
    id: str,
    files: tuple[GgufFile, ...],
    license: str = "",
    description: str = "",
    parameter_size: str | None = None,
    quantization: str | None = None,
    est_ram_bytes: int | None = None,
    est_vram_bytes: int | None = None,
)

A pinned, verifiable local model artifact.

Attributes:

Name	Type	Description
`id`	`str`	Stable artifact id, the handle alias targets and the llama.cpp adapter use.
`files`	`tuple[GgufFile, ...]`	Every file the artifact comprises — one entry, or several for a sharded model.
`license`	`str`	License id; checked against the download allowlist for application-supplied entries.
`description`	`str`	Free text for display.
`parameter_size`	`str \| None`	Parameter class (`"7B"`), when known.
`quantization`	`str \| None`	Quantization of the shipped weights (`"Q4_K_M"`), when known.
`est_ram_bytes`	`int \| None`	Estimated memory needed on the CPU-only path.
`est_vram_bytes`	`int \| None`	Estimated memory needed when fully offloaded.

total_size_bytes `property` ¶

total_size_bytes: int | None

Sum of every file's size, or None when any is unknown.

is_sharded `property` ¶

is_sharded: bool

Whether this artifact spans multiple files.

anyinfer.local.GgufFile `dataclass` ¶

GgufFile(
    filename: str,
    url: str,
    sha256: str = "",
    size_bytes: int | None = None,
)

One file of a (possibly sharded) GGUF artifact.

Attributes:

Name	Type	Description
`filename`	`str`	The name the file is stored under in the model directory.
`url`	`str`	Pinned download URL.
`sha256`	`str`	Expected content hash. Empty means the file cannot be verified, and the downloader warns instead of checking.
`size_bytes`	`int \| None`	Expected size, when known; feeds download progress totals.

anyinfer.local.download_artifact ¶

download_artifact(
    artifact: GgufArtifact,
    *,
    model_dir: Path | None = None,
    progress: ProgressCallback | None = None,
    client: Client | None = None,
    enforce_license: bool = False,
) -> DownloadReport

Ensure every file of an artifact is present and verified.

Parameters:

Name	Type	Description	Default
`artifact`	`GgufArtifact`	The pinned catalog entry.	required
`model_dir`	`Path \| None`	Destination directory; defaults to `default_model_dir()`.	`None`
`progress`	`ProgressCallback \| None`	Called as bytes arrive.	`None`
`client`	`Client \| None`	An `httpx2.Client` to use, for tests or custom transports.	`None`
`enforce_license`	`bool`	Reject artifacts whose license is not in `ALLOWED_LICENSES`. Applied to application-supplied entries.	`False`

Returns:

Type	Description
`DownloadReport`	A report naming the on-disk files.

Raises:

Type	Description
`LocalRuntimeError`	On a hash mismatch, a transfer failure, or a rejected license.

anyinfer.local.iter_missing ¶

iter_missing(
    artifacts: Iterable[GgufArtifact],
    model_dir: Path | None = None,
) -> list[GgufArtifact]

Which artifacts are absent or fail verification.

anyinfer.local.verify_file ¶

verify_file(path: Path, expected_sha256: str) -> bool

Whether a file matches its expected hash.

An artifact with no recorded hash cannot be verified; its mere existence is accepted, and the caller is warned.

anyinfer.local.artifact_paths ¶

artifact_paths(
    artifact: GgufArtifact, model_dir: Path | None = None
) -> tuple[Path, ...]

Where an artifact's files would live on disk.

anyinfer.local.default_model_dir ¶

default_model_dir() -> Path

Where downloaded artifacts live by default.

anyinfer.local.DownloadReport `dataclass` ¶

DownloadReport(
    artifact_id: str,
    paths: tuple[Path, ...],
    downloaded_bytes: int = 0,
    reused: bool = False,
    warnings: tuple[str, ...] = (),
)

The outcome of ensuring an artifact is present.

Attributes:

Name	Type	Description
`artifact_id`	`str`	The artifact this report describes.
`paths`	`tuple[Path, ...]`	Where each of the artifact's files lives on disk, in declaration order.
`downloaded_bytes`	`int`	Bytes actually transferred; zero when everything was reused.
`reused`	`bool`	Whether every file was already present and verified, so nothing was fetched.
`warnings`	`tuple[str, ...]`	Non-fatal notes — files with no recorded hash, or files that failed verification and were re-downloaded.

primary_path `property` ¶

primary_path: Path

The file to hand to llama-server (the first shard of a sharded artifact).

anyinfer.local.ProgressCallback `module-attribute` ¶

ProgressCallback = Callable[[str, int, int | None], None]

(artifact_id, downloaded_bytes, total_bytes_or_None).

anyinfer.local.ALLOWED_LICENSES `module-attribute` ¶

ALLOWED_LICENSES = frozenset(
    {
        "apache-2.0",
        "falcon-llm-2.0",
        "gemma-terms",
        "llama-3.1-community",
        "llama-3.2-community",
        "llama-3.3-community",
        "mit",
        "openrail-m",
    }
)

Licenses permitted for catalog entries, compared case-insensitively.

The bundled catalog is curated; entries an application adds are checked so that a convenience feature cannot quietly redistribute weights under terms the user has not seen. Non-commercial and research-only terms are deliberately absent — an application that has accepted those terms adds the model through a catalog overlay, which is an explicit act.

anyinfer.local.license_allowed ¶

license_allowed(license_id: str) -> bool

Whether a license id is in ALLOWED_LICENSES, ignoring case.

Server supervision¶

anyinfer.local.ServerSupervisor ¶

ServerSupervisor(
    *,
    binary: Path | str = "llama-server",
    hardware: HardwareProfile | None = None,
    idle_ttl_s: float | None = 900.0,
    max_resident: int = 1,
    allow_remote_exposure: bool = False,
    host: str = LOOPBACK_HOST,
    health_timeout_s: float = _HEALTH_TIMEOUT_S,
    on_lifecycle: LifecycleCallback | None = None,
)

Owns the llama-server processes for one client.

Parameters:

Name	Type	Description	Default
`binary`	`Path \| str`	Path to the `llama-server` executable.	`'llama-server'`
`hardware`	`HardwareProfile \| None`	Detected hardware, used for VRAM admission control.	`None`
`idle_ttl_s`	`float \| None`	Unload a server after this long with no active streams. `None` keeps servers until the supervisor closes.	`900.0`
`max_resident`	`int`	How many servers may run at once. Exceeding it evicts the least-recently-used idle server.	`1`
`allow_remote_exposure`	`bool`	Bind a non-loopback address. Off by default: a local model server is loopback-only unless deliberately exposed.	`False`
`on_lifecycle`	`LifecycleCallback \| None`	Called with lifecycle events.	`None`

resident_models `property` ¶

resident_models: tuple[str, ...]

Model keys with a running server.

acquire `async` ¶

acquire(
    model_key: str,
    model_path: Path,
    plan: ServerPlan,
    *,
    persist: bool = False,
) -> ManagedServer

Get a ready server for a model, starting or reusing one.

Blocks until the server answers its health probe. Concurrent callers requesting different models are serialized, so loads never overlap.

Raises:

Type	Description
`LocalRuntimeError`	If the model cannot fit, the binary is missing, or the server fails to become ready.

set_hardware ¶

set_hardware(hardware: HardwareProfile) -> None

Late-bind the detected hardware profile.

Detection is deliberately lazy — probing at construction would tax clients that never run a local model — so the adapter hands the profile over once it has one. Admission control and backend fallback stay disabled until then.

resolve_binary ¶

resolve_binary() -> Path

Locate the llama-server executable without starting anything.

Public so a health probe can answer "could a server start?" cheaply.

Falls back to the best installed backend variant for this hardware when the configured name is not on PATH (a CUDA build in a known runtime directory beats a missing PATH entry).

Raises:

Type	Description
`LocalRuntimeError`	When no usable binary exists anywhere.

collect_idle `async` ¶

collect_idle() -> int

Stop servers idle beyond the TTL. Returns how many were stopped.

Call periodically. Servers with active streams are never collected, however long the generation has been running.

aclose `async` ¶

aclose() -> None

Stop every supervised server.

anyinfer.local.ManagedServer ¶

ManagedServer(handle: ServerHandle)

A context manager marking a server busy for the duration of a request.

This is what makes the idle timer honest: the server is busy while a stream is open, not merely while a request is arriving.

base_url `property` ¶

base_url: str

The server's base URL.

enter ¶

__enter__() -> ManagedServer

Mark the server busy.

exit ¶

__exit__(*exc: object) -> None

Release the server and restart its idle clock.

anyinfer.local.ServerHandle `dataclass` ¶

ServerHandle(
    model_key: str,
    model_path: Path,
    plan: ServerPlan,
    host: str,
    port: int,
    process: Popen[bytes],
    started_at: float,
    log_tail: deque[str] = (
        lambda: deque(maxlen=_LOG_TAIL_LINES)
    )(),
    active_streams: int = 0,
    last_activity: float = time.monotonic(),
    persist: bool = False,
    stopping: bool = False,
)

A running llama-server and everything known about it.

model_key `instance-attribute` ¶

model_key: str

The model this server serves; also its key in the supervisor's server table.

model_path `instance-attribute` ¶

model_path: Path

The GGUF file the server was started with.

plan `instance-attribute` ¶

plan: ServerPlan

The tuned launch plan; its memory estimate is what admission control committed.

host `instance-attribute` ¶

host: str

Interface the server is bound to — loopback unless exposure was explicitly allowed.

port `instance-attribute` ¶

port: int

TCP port the server listens on, allocated just before spawning.

process `instance-attribute` ¶

process: Popen[bytes]

The supervised child, polled for liveness and terminated on stop.

started_at `instance-attribute` ¶

started_at: float

Monotonic time the child was spawned.

log_tail `class-attribute` `instance-attribute` ¶

log_tail: deque[str] = field(
    default_factory=lambda: deque(maxlen=_LOG_TAIL_LINES)
)

The child's most recent output lines, kept so failures can explain themselves.

active_streams `class-attribute` `instance-attribute` ¶

active_streams: int = 0

Open response streams. Nonzero means busy: the idle clock and eviction ignore it.

last_activity `class-attribute` `instance-attribute` ¶

last_activity: float = field(default_factory=time.monotonic)

Monotonic time of the last request or stream release; the idle clock's baseline.

persist `class-attribute` `instance-attribute` ¶

persist: bool = False

Exempt this server from idle collection and capacity eviction.

stopping `class-attribute` `instance-attribute` ¶

stopping: bool = False

Set while the supervisor is tearing this server down.

base_url `property` ¶

base_url: str

The OpenAI-compatible base URL this server serves.

is_running `property` ¶

is_running: bool

Whether the child process is still alive.

is_idle `property` ¶

is_idle: bool

Whether no request is currently streaming from this server.

idle_seconds ¶

idle_seconds() -> float

How long this server has been idle. Zero while any stream is active.

touch ¶

touch() -> None

Mark activity, resetting the idle clock.

anyinfer.local.LifecycleCallback `module-attribute` ¶

LifecycleCallback = Callable[[ServerLifecycle], None]

Receives every lifecycle transition of a supervised server.

anyinfer.local.allocate_port ¶

allocate_port(host: str = LOOPBACK_HOST) -> int

Reserve an ephemeral port by binding and immediately releasing it.

Inherently racy, but the alternative — letting llama-server pick and then discovering which port it chose — requires parsing its log output, which is far more fragile.

anyinfer.local.LOOPBACK_HOST `module-attribute` ¶

LOOPBACK_HOST = '127.0.0.1'

Local servers bind loopback only, unless the caller explicitly opts out (D20).

anyinfer.local.is_loopback ¶

is_loopback(base_url: str | None) -> bool

Whether a base URL points at this machine.

Used by two callers with the same underlying question — "is the thing at the other end of this URL running on hardware I can probe?". A remote Ollama daemon answers no, and everything downstream (hardware detection, fit classification, zero-cost pricing) depends on not pretending otherwise.

A URL that cannot be parsed is treated as not loopback, because the safe default is to assume someone else's machine.

Recommendation¶

anyinfer.local.recommend_alias ¶

recommend_alias(
    hardware: HardwareProfile,
    catalog: TierSource,
    *,
    prefer_accelerated: bool = True,
) -> Recommendation

Recommend the largest catalog tier this machine can comfortably run.

Parameters:

Name	Type	Description	Default
`hardware`	`HardwareProfile`	The detected profile.	required
`catalog`	`TierSource`	The catalog whose aliases carry `min_ram_bytes`/`min_vram_bytes`.	required
`prefer_accelerated`	`bool`	Budget against VRAM when an accelerator is present. With unified memory, system RAM is the budget regardless.	`True`

Returns:

Type	Description
`Recommendation`	A recommendation. When memory is unknown, the smallest tier is proposed with
`Recommendation`	`confident=False` rather than guessing upward.

anyinfer.local.Recommendation `dataclass` ¶

Recommendation(
    alias: str | None, reason: str, confident: bool = True
)

A recommended tier and the reasoning behind it.

Attributes:

Name	Type	Description
`alias`	`str \| None`	The recommended alias, or `None` when nothing fits.
`reason`	`str`	Why this tier was chosen, for display to a user.
`confident`	`bool`	`False` when the machine's memory could not be determined, so the recommendation is a floor rather than a fit.

anyinfer.local.Tier ¶

Bases: Protocol

The subset of a catalog alias this module needs.

Structural rather than nominal so the local subsystem does not import the catalog: artifacts (local data) are depended on by the catalog, and a reverse dependency here would make that a cycle.

name `property` ¶

name: str

The alias name.

min_ram_bytes `property` ¶

min_ram_bytes: int | None

System RAM this tier needs, when stated.

min_vram_bytes `property` ¶

min_vram_bytes: int | None

Accelerator memory this tier needs, when stated.

anyinfer.local.TierSource ¶

Bases: Protocol

The subset of a catalog needed to recommend a tier.

alias_names ¶

alias_names() -> tuple[str, ...]

Every alias name.

alias ¶

alias(name: str) -> Tier

Look up one alias.

Local inference¶

Hardware¶

anyinfer.local.detect ¶

anyinfer.local.HardwareProfile dataclass ¶

primary_accelerator property ¶

has_accelerator property ¶

total_vram_bytes property ¶

user_supplied property ¶

from_user_input classmethod ¶

to_json ¶

from_json classmethod ¶

anyinfer.local.Accelerator dataclass ¶

compute_capability_value property ¶

driver_major property ¶

anyinfer.local.AcceleratorKind module-attribute ¶

anyinfer.local.probe_signature ¶

anyinfer.local.cache_path ¶

anyinfer.local.CACHE_BYPASS_ENV module-attribute ¶

anyinfer.local.CACHE_REFRESH_ENV module-attribute ¶

Backends¶

anyinfer.local.available_backends ¶

anyinfer.local.select_backend ¶

anyinfer.local.Backend dataclass ¶

anyinfer.local.BACKEND_RANK module-attribute ¶

Runtime variants¶

anyinfer.local.install_runtime ¶

anyinfer.local.installed_runtimes ¶

anyinfer.local.remove_runtime ¶

anyinfer.local.default_runtime_kind ¶

anyinfer.local.check_cuda_preconditions ¶

anyinfer.local.install_hint ¶

anyinfer.local.load_runtime_table ¶

anyinfer.local.runtime_root ¶

anyinfer.local.RuntimeTable dataclass ¶

for_platform ¶

artifact ¶

anyinfer.local.RuntimeArtifact dataclass ¶

total_bytes property ¶

anyinfer.local.RuntimeManifest dataclass ¶

anyinfer.local.InstallReport dataclass ¶

Tuning¶

anyinfer.local.plan_server ¶

anyinfer.local.TuningInputs dataclass ¶

anyinfer.local.ServerPlan dataclass ¶

context_per_slot property ¶

server_arguments ¶

anyinfer.local.Posture module-attribute ¶

anyinfer.local.CONTEXT_LADDER module-attribute ¶

anyinfer.local.kv_bytes_per_token ¶

Fit and variant selection¶

anyinfer.local.classify_fit ¶

anyinfer.local.ModelFit dataclass ¶

runnable property ¶

rank property ¶

anyinfer.local.FitLevel module-attribute ¶

anyinfer.local.memory_budget ¶

anyinfer.local.sort_by_fit ¶

anyinfer.local.SizedEntry ¶

id property ¶

parameter_size property ¶

est_ram_bytes property ¶

est_vram_bytes property ¶

anyinfer.local.select_variant ¶

anyinfer.local.evaluate_variants ¶

anyinfer.local.VariantChoice dataclass ¶

anyinfer.local.VariantPrefs dataclass ¶

Acquisition and the model store¶

anyinfer.local.acquire ¶

AcquisitionPhase module-attribute ¶

ProgressSink module-attribute ¶

AcquisitionProgress dataclass ¶

fraction property ¶

AcquisitionPlan dataclass ¶

remaining_bytes property ¶

total_is_estimate property ¶

AcquisitionReport dataclass ¶

path property ¶

AcquisitionRequest dataclass ¶

plan_acquisition async ¶

acquire async ¶

anyinfer.local.HardwareProfile `dataclass` ¶

primary_accelerator `property` ¶

has_accelerator `property` ¶

total_vram_bytes `property` ¶

user_supplied `property` ¶

from_user_input `classmethod` ¶

from_json `classmethod` ¶

anyinfer.local.Accelerator `dataclass` ¶

compute_capability_value `property` ¶

driver_major `property` ¶

anyinfer.local.AcceleratorKind `module-attribute` ¶

anyinfer.local.CACHE_BYPASS_ENV `module-attribute` ¶

anyinfer.local.CACHE_REFRESH_ENV `module-attribute` ¶

anyinfer.local.Backend `dataclass` ¶

anyinfer.local.BACKEND_RANK `module-attribute` ¶

anyinfer.local.RuntimeTable `dataclass` ¶

anyinfer.local.RuntimeArtifact `dataclass` ¶

total_bytes `property` ¶

anyinfer.local.RuntimeManifest `dataclass` ¶

anyinfer.local.InstallReport `dataclass` ¶

anyinfer.local.TuningInputs `dataclass` ¶

anyinfer.local.ServerPlan `dataclass` ¶

context_per_slot `property` ¶

anyinfer.local.Posture `module-attribute` ¶

anyinfer.local.CONTEXT_LADDER `module-attribute` ¶

anyinfer.local.ModelFit `dataclass` ¶

runnable `property` ¶

rank `property` ¶

anyinfer.local.FitLevel `module-attribute` ¶

id `property` ¶

parameter_size `property` ¶

est_ram_bytes `property` ¶

est_vram_bytes `property` ¶

anyinfer.local.VariantChoice `dataclass` ¶

anyinfer.local.VariantPrefs `dataclass` ¶

AcquisitionPhase `module-attribute` ¶

ProgressSink `module-attribute` ¶

AcquisitionProgress `dataclass` ¶

fraction `property` ¶

AcquisitionPlan `dataclass` ¶

remaining_bytes `property` ¶

total_is_estimate `property` ¶

AcquisitionReport `dataclass` ¶

path `property` ¶

AcquisitionRequest `dataclass` ¶

plan_acquisition `async` ¶

acquire `async` ¶

anyinfer.local.plan_acquisition `async` ¶

anyinfer.local.AcquisitionPlan `dataclass` ¶

remaining_bytes `property` ¶

total_is_estimate `property` ¶

anyinfer.local.AcquisitionProgress `dataclass` ¶

fraction `property` ¶

anyinfer.local.AcquisitionReport `dataclass` ¶

path `property` ¶

anyinfer.local.AcquisitionPhase `module-attribute` ¶

anyinfer.local.ProgressSink `module-attribute` ¶

root `property` ¶

index_path `property` ¶

anyinfer.local.StoreEntry `dataclass` ¶

total_bytes `property` ¶

verified `property` ¶

from_json `classmethod` ¶

anyinfer.local.ResolvedModel `dataclass` ¶

anyinfer.local.RemovalReport `dataclass` ¶

anyinfer.local.SourceRef `dataclass` ¶

scheme `instance-attribute` ¶

resolve `async` ¶

anyinfer.local.ResolvedArtifact `dataclass` ¶

total_bytes `property` ¶

anyinfer.local.RemoteFile `dataclass` ¶

filename `property` ¶

anyinfer.local.GgufArtifact `dataclass` ¶

total_size_bytes `property` ¶

is_sharded `property` ¶

anyinfer.local.GgufFile `dataclass` ¶

anyinfer.local.DownloadReport `dataclass` ¶

primary_path `property` ¶

anyinfer.local.ProgressCallback `module-attribute` ¶

anyinfer.local.ALLOWED_LICENSES `module-attribute` ¶