Skip to content

Ollama

Uses Ollama's native /api/chat API, not its /v1 OpenAI-compatibility layer. The native API carries grammar-enforced structured output, per-phase nanosecond timings, keep_alive session retention, and reasoning via think; the /v1 layer does not implement all of these and drops the parameters it lacks without reporting it.

streaming structured output (grammar) tool calls health discovery embeddings

Setup

client = ai.Client([ai.ProviderSettings.of("ollama")])
result = client.generate(prompt, target="ollama:qwen3:8b")

Defaults to http://127.0.0.1:11434. A bare hostname expands automatically, so base_url="myserver" becomes http://myserver:11434.

Model names may contain colons: "ollama:qwen3:8b" is the provider ollama and the model qwen3:8b, because targets split on the first colon only.

Supported

Behavior Support
Streaming Native NDJSON
Structured output Grammar-enforced via format
Tools Native
Reasoning think, with effort levels
Usage Input and output tokens
Phase timings Model load, prefill, decode
Sessions keep_alive

Structured Output

Ollama compiles the schema to a decoding grammar. Two consequences AnyInfer handles:

  • Grammar-hostile keywords (minLength, maxLength, huge minItems/maxItems) are stripped for the wire only; the original schema still validates the response.
  • The schema is also injected into the prompt. A grammar guarantees well-formed JSON, not meaningful JSON; a model never shown the schema emits schema-shaped nonsense.

Phase Timings

result.timing.phases
# {"model_load_ms": 300.0, "prefill_ms": 200.0,
#  "decode_ms": 1000.0, "provider_total_ms": 1500.0}

A large model_load_ms on a first request is the model being read from disk.

GPU Spill Diagnostics

The slowest failure Ollama has is not a failure. A model that no longer fits in VRAM alongside whatever else the GPU is holding is loaded anyway, with the overflow served from system memory; the request succeeds, the answer is correct, and it takes an order of magnitude longer than the same model took yesterday. The wire says nothing about it.

/api/ps reports how much of each resident model is actually in VRAM, so the adapter reads it and says so:

for note in client.diagnostics("ollama"):
    print(note.code, note.message)
# ollama.gpu-spill  qwen3:8b is only 45% resident in VRAM; the rest runs on the CPU,
#                   which is far slower. Free GPU memory, or choose a smaller model
#                   or quantization.

The same text lands on result.warnings for any request that hit it, and as a ProviderDiagnostic event; see runtime diagnostics. Costs nothing: /api/ps is a local read, never a generation. A model within 5% of full residency is not reported; Ollama's own sizes wobble by a few megabytes, and a warning on every healthy load is one nobody reads.

Embeddings

Ollama's native POST /api/embed is a batch-capable embedding endpoint. (The older POST /api/embeddings route is deprecated and singular-input; this adapter does not speak it.)

result = client.embed(["Why is the sky blue?", "Why is the grass green?"], target="ollama:nomic-embed-text")
print(result.space.dimensions, len(result.vectors))

Batch input is native: every text in one call is sent as one array, not simulated with repeated requests. Requested dimensions are forwarded when the model supports native dimensionality reduction. Ollama documents no native rerank endpoint, so reranking is unsupported for this provider.

Multimodal Inputs

Vision-capable models receive inline images through the native message images field. Remote image URLs, documents, and audio are refused rather than silently dropped.

Notes

  • A missing model produces ModelNotFoundError hinting ollama pull <model>.
  • Usage arrives only on the terminal object; the core synthesizes a UsageUpdate event so streaming consumers see it exactly as they would from any other provider.
  • Native extras pass through provider options: provider_options = {"ollama": {"keep_alive": "10m", "num_ctx": 8192, "num_gpu": 99}}.

Wire Contract

For the exact request/response fields this adapter depends on, see contracts/ollama.md.

See Also