Skip to content

OpenAI-Compatible

The base dialect for any endpoint speaking POST /chat/completions: vLLM, LM Studio, an externally-run llama-server, a corporate gateway, or OpenAI itself.

streaming structured output (server-dependent) tool calls health discovery

Setup

client = ai.Client(
    [
        ai.ProviderSettings.of(
            "openai-compat",
            base_url="http://localhost:8000/v1",
            api_key="env://MY_API_KEY",  # optional for keyless local servers
        ),
    ]
)
result = client.generate(prompt, target="openai-compat:my-model")

base_url is required; there is no sensible default for "any server".

Aliases: openai-compatible, oai-compat.

Supported

Behavior Support
Streaming SSE
Structured output json_schema or json_object, where the server implements it
Tools Native
Usage When the server reports it

Capabilities Are Unknown by Default

AnyInfer cannot know what an arbitrary server supports, so features default to a conservative set and structured output falls back to prompt injection unless told otherwise. Client-side validation means the caller still gets a validated result either way.

In order to declare what a server actually does, register a descriptor with richer default_capabilities; see writing an adapter.

Servers That Ignore stream

Some endpoints accept stream: true and answer with a buffered body anyway. The adapter detects that and consumes the body rather than paying for a second request, so consumer code is unaffected.

Known Divergences

  • max_tokens vs max_completion_tokens: subclasses override this; the base sends max_tokens.
  • stream_options.include_usage is not universally implemented. Usage is simply absent when a server omits it, never fabricated.
  • response_format support varies widely between implementations, which is one of the reasons validation is always client-side.

Wire Contract

For the exact request/response fields this adapter depends on, see contracts/openai-compat.md.

See Also