llama.cpp¶

A supervised llama-server subprocess speaking the OpenAI-compatible dialect over loopback. In-process llama-cpp-python is explicitly not supported: one wire protocol for every engine, crash isolation, and no GPU-wheel build matrix in the dependency tree.

streaming structured output (grammar) tool calls health discovery

Setup¶

client = ai.Client([
    ai.ProviderSettings.of(
        "llama-cpp",
        options={
            "catalog": ai.load_default_catalog(),
            "posture": "balanced",
            "idle_ttl_s": 900,
        },
    ),
])

result = client.generate(prompt, target="llama-cpp:qwen2.5-7b-instruct-q4-k-m")

The model reference is a catalog artifact id, not a file path. That one call resolves the artifact, downloads and verifies it, tunes a server for your hardware, starts it, and answers.

Install a pinned runtime with anyinfer runtime install, or supply your own llama-server through the binary option. AnyInfer fetches runtimes and weights on demand rather than embedding them in the wheel, and verifies catalog-managed downloads before use.

Aliases: llamacpp, llama.

Options¶

Option	Default	Meaning
`catalog`	—	Required. Resolves artifact ids to pinned downloads.
`binary`	`llama-server`	Optional executable override; otherwise the best installed AnyInfer runtime is selected.
`model_dir`	platform cache	Where GGUF files are stored.
`posture`	`balanced`	`conservative`, `balanced`, or `aggressive`.
`hardware`	detected	A pre-detected profile, to skip re-probing.
`idle_ttl_s`	900	Unload after this long with no active streams.
`max_resident`	1	Concurrent servers before eviction.
`auto_download`	`True`	Fetch a missing artifact rather than failing.
`allow_remote_exposure`	`False`	Bind a non-loopback address.
`progress`	—	Download progress callback.

The typed form of this table, for programmatic construction:

anyinfer.providers.llama_cpp.LlamaCppOptions `dataclass` ¶

LlamaCppOptions(
    catalog: Catalog | None = None,
    binary: str = "llama-server",
    model_dir: Path | None = None,
    posture: Posture = "balanced",
    hardware: HardwareProfile | None = None,
    idle_ttl_s: float | None = 900.0,
    max_resident: int = 1,
    auto_download: bool = True,
    allow_remote_exposure: bool = False,
    progress: ProgressCallback | None = None,
)

Adapter configuration, supplied through ProviderSettings.options.

Attributes:

Name	Type	Description
`catalog`	`Catalog \| None`	Catalog resolving artifact ids to pinned downloads.
`binary`	`str`	Path to `llama-server`.
`model_dir`	`Path \| None`	Where artifacts are stored.
`posture`	`Posture`	Tuning posture.
`hardware`	`HardwareProfile \| None`	Pre-detected hardware, to avoid re-probing.
`idle_ttl_s`	`float \| None`	Unload a server after this long with no active streams.
`max_resident`	`int`	How many servers may run at once.
`auto_download`	`bool`	Fetch a missing artifact rather than failing.
`allow_remote_exposure`	`bool`	Bind a non-loopback address; loopback-only by default.
`progress`	`ProgressCallback \| None`	Download progress callback.

from_mapping `classmethod` ¶

from_mapping(options: Mapping[str, Any]) -> LlamaCppOptions

Build options from a provider settings mapping, ignoring unknown keys.

Supported¶

Behavior	Support
Streaming	Native SSE
Structured output	Grammar (GBNF), compiled from your schema
Tools	Native, via `--jinja`
Usage	Input and output tokens
Cost	Free — a genuine zero, not an unknown

Structured output¶

llama.cpp compiles response_format.json_schema into a real GBNF grammar. As with Ollama, the schema is also injected into the prompt, because a grammar constrains form without conveying meaning.

Tool calling needs `--jinja`¶

The tuner always emits --jinja. Without it, llama-server cannot apply a model's chat template and tool calling silently does not work at all.

Supervision¶

Servers bind 127.0.0.1. A non-loopback bind requires allow_remote_exposure=True.
Model swaps are serialized — two unloaded models never race for the same VRAM.
Requests block until the server is ready rather than receiving a 503 mid-load.
VRAM admission is checked before spawning, so an oversized model is refused with a clear message instead of crashing the child process.
The idle timer keys on active streams, so a long generation is never mistaken for idle.

See the local subsystem.

Wire contract¶

For the exact request/response fields this adapter depends on, see contracts/llama-cpp.md.