Cohere¶

The native v2 Chat API, not the OpenAI compatibility endpoint. The compatibility layer would work, but v2 is where the things worth choosing Cohere for live: grounded generation with document citations, a separate thinking channel, and usage that distinguishes what was processed from what was billed.

streaming structured output tool calls reasoning discovery (context lengths)

Setup¶

import anyinfer as ai

client = ai.Client([
    ai.ProviderSettings.of("cohere", api_key="env://CO_API_KEY"),
])

result = client.generate(prompt, target="cohere:command-a-03-2025")

Dialect differences you may notice¶

Cohere's API diverges from the OpenAI shape in ways the adapter normalizes, but which show up if you reach past it with provider_options:

AnyInfer	Cohere
`finish_reason == "stop"`	`COMPLETE` (uppercase enum)
`tool_choice="required"`	`"REQUIRED"`; there is no way to name one tool
`Sampling(top_p=...)`	`p`, not `top_p`
optional streaming	`stream` is required on every request

Unknown finish reasons normalize to "other" rather than propagating.

Reasoning¶

Cohere budgets thinking in tokens rather than naming levels, so normalized effort maps to a budget:

result = client.generate(prompt, target="cohere:command-a-03-2025", reasoning="high")

minimal disables thinking outright; low, medium, and high map to increasing token budgets. Thinking blocks arrive as ReasoningDelta events and stay out of result.text.

Usage: processed, not billed¶

Cohere reports both billed_units and tokens. AnyInfer's counts follow tokens — what the model actually processed, which is what a context window measures:

result = client.generate(prompt, target="cohere:command-a-03-2025")
print(result.usage.input_tokens)   # processed

If you need billed units for cost reconciliation, build the client with retain_raw=True and read them off result.raw.

Grounded generation¶

Cohere's document grounding and citations are reachable through the escape hatch, though citations are not yet surfaced as typed results:

client.generate(
    question,
    target="cohere:command-a-03-2025",
    provider_options={"cohere": {
        "documents": [{"id": "doc1", "data": {"text": "..."}}],
        "citation_options": {"mode": "ACCURATE"},
    }},
)

Read the citations from result.raw (with retain_raw=True) until they are modelled.

Discovery¶

The model listing reports real context lengths, so windows carry discovered provenance:

for model in client.models("cohere"):
    caps = model.capabilities
    if caps and caps.context_window:
        print(model.id, caps.context_window.value, caps.context_window.provenance)

Only chat-capable models are listed.