0 Fork 0

Files

T

Hongsheng Liuandhsliu_ustc 44c8d1c449 docs: add concise API server endpoint guide (#6114 )

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>

2026-09-01 20:34:09 +08:00

3.8 KiB

Raw Permalink Blame History

Chat Completions API

Use /v1/chat/completions for conversational and multimodal pipelines that follow the OpenAI Chat API schema. Depending on the loaded model, a request can contain text, image, audio, or video input and can return text, audio, images, or other model-specific output.

Some diffusion pipelines also support image generation and editing through Chat Completions. Prefer the dedicated Image Generation API or Image Edit API for task-specific clients; their request fields and response contracts are more direct.

Basic Request

curl http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Describe vLLM-Omni briefly."}
    ],
    "modalities": ["text"]
  }'

Use stream: true for Server-Sent Events. Input media syntax, supported output modalities, and response choices depend on the model; see the model-specific online serving examples.

!!! tip Each server hosts one model. Query GET /v1/models for its served model name when your client requires the model field.

vLLM Extension Parameters

Send standard and vLLM-specific Chat parameters as top-level fields in a direct HTTP request. When using the OpenAI Python SDK, pass vLLM-specific fields through the SDK's extra_body keyword; the SDK merges them into the top-level JSON sent to the server.

=== "curl"

```bash
curl http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a short haiku."}],
    "top_k": 40
  }'
```

=== "OpenAI Python SDK"

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")

response = client.chat.completions.create(
    model="your-served-model",
    messages=[{"role": "user", "content": "Write a short haiku."}],
    extra_body={"top_k": 40},
)
```

??? note "Diffusion compatibility"

Diffusion models exposed through Chat Completions can accept supported
fields such as `num_inference_steps`, `seed`, `height`, and `width`. Send
them at the top level with direct HTTP, or in the SDK's `extra_body`
keyword argument.

A literal nested `"extra_body"` JSON object is accepted for compatibility,
but it is not recommended for new direct HTTP clients. Do not provide the
same parameter in multiple locations; the server returns a `400` error for
duplicate diffusion parameters. Prefer the dedicated [Image Generation
API](image_generation_api.md) or [Image Edit API](image_edit_api.md) when
either endpoint matches the task.

Batch Requests

POST /v1/chat/completions/batch accepts the same shared generation fields, but messages is a list of conversations. The response contains one choice per conversation in input order.

curl http://localhost:8091/v1/chat/completions/batch \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      [{"role": "user", "content": "Summarize vLLM in one sentence."}],
      [{"role": "user", "content": "Summarize vLLM-Omni in one sentence."}]
    ],
    "max_tokens": 64
  }'

Batch Chat Completions does not support streaming, tools, beam search, or n > 1.

Model-Specific Examples

For complete examples with model-specific inputs and outputs, see:

3.8 KiB Raw Permalink Blame History