anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: add API server endpoint guide

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-13 21:28:36 +08:00
1 parent f41451f64c
commit 5c6ed9b4a9
8 files changed
+279 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+2 -1
View File
@@ -5,7 +5,8 @@ nav:
- getting_started/quickstart.md
- getting_started/installation/*
- Serving:
- OpenAI-Compatible API:
- API Server:
- Overview: serving/README.md
- Chat Completions API: serving/chat_completions_api.md
- Image Generation: serving/image_generation_api.md
- Image Edit: serving/image_edit_api.md
docs/examples/README.md
+3 -1
View File
@@ -3,4 +3,6 @@
vLLM-Omni's examples are split into two categories:
- If you are using vLLM-Omni from within Python code, see the *Offline Inference* section.
- If you are using vLLM-Omni from an HTTP application or client, see the *Online Serving* section.
- If you are using vLLM-Omni from an HTTP application or client, start with
the [API Server guide](../serving/README.md) to choose an endpoint, then use
the *Online Serving* section for model-specific examples.
docs/getting_started/quickstart.md
+10 -14
View File
@@ -92,7 +92,7 @@ if __name__ == "__main__":
For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md)
## Online Serving with OpenAI-Completions API
## Online Serving with the API Server
Text-to-image generation quickstart with vLLM-Omni:
@@ -101,20 +101,16 @@ vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091
```
```bash
curl -s http://localhost:8091/v1/chat/completions \
curl -s http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "a cup of coffee on the table"}
],
"extra_body": {
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 4.0,
"seed": 42
}
}' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > coffee.png
"prompt": "a cup of coffee on the table",
"size": "1024x1024",
"response_format": "b64_json",
"seed": 42
}' | jq -r '.data[0].b64_json' | base64 -d > coffee.png
```
For more details, please refer to [online serving](../user_guide/examples/online_serving/text_to_image.md).
See the [API Server guide](../serving/README.md) to choose an endpoint. For
model-specific details, refer to the [text-to-image online serving
example](../user_guide/examples/online_serving/text_to_image.md).
docs/serving/README.md
+173
View File
@@ -0,0 +1,173 @@
# API Server
vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for
image, audio, video, and realtime workloads. Use this page to choose an
endpoint; follow the linked reference page for model-specific fields and
response details.
Each server instance hosts one model. An endpoint is usable only when the
loaded model supports that task.
This guide focuses on vLLM-Omni's primary task endpoints. Compatible models
may also expose standard routes inherited from vLLM, such as
`POST /v1/completions`; those routes remain model- and task-dependent.
## Start and Verify the Server
Start a model with the shared serving command:
```bash
vllm serve <model> --omni --port 8000
```
Some models require additional flags or a deployment configuration. Use the
model-specific guide when one is provided.
After the server starts, check its health and served model name:
```bash
export VLLM_OMNI_BASE_URL=http://localhost:8000
curl "$VLLM_OMNI_BASE_URL/health"
curl "$VLLM_OMNI_BASE_URL/v1/models" | jq .
```
Use `http://localhost:8000/v1` as the `base_url` for an OpenAI SDK client. If
the server was started with `--api-key`, include an `Authorization` header with
`Bearer <api-key>` in requests.
## Choose an Endpoint
Prefer a task-specific endpoint when one matches your workload. Use Chat
Completions for conversational or heterogeneous omni pipelines, rather than as
the default wrapper for every generation task.
| Task | Endpoint | Request | Response | Details |
|------|----------|---------|----------|---------|
| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) |
| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) |
| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) |
| Text-to-image | `POST /v1/images/generations` | JSON | Base64 JSON or an image file | [Image Generation](image_generation_api.md) |
| Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) |
| Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) |
| Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) |
| Live video understanding | `WS /v1/video/chat/stream` | WebSocket events | Streaming text/audio events | [Streaming Video Input](video_stream_api.md) |
`POST /v1/videos/sync` is intended for tests and simple scripts. Use the
asynchronous job API for production requests so clients can poll, download,
and delete generated videos independently.
## Minimal Requests
The examples below assume a compatible model is already running. TTS voices,
media inputs, generation controls, and output capabilities vary by model.
=== "Chat"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Describe vLLM-Omni briefly."}],
"modalities": ["text"]
}'
```
=== "Speech"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/audio/speech" \
-H "Content-Type: application/json" \
-d '{
"input": "Hello from vLLM-Omni.",
"voice": "vivian"
}' \
--output speech.wav
```
=== "Audio"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/audio/generate" \
-H "Content-Type: application/json" \
-d '{
"input": "Ocean waves on a quiet beach",
"audio_length": 5.0
}' \
--output audio.wav
```
=== "Image"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/images/generations" \
-H "Content-Type: application/json" \
-d '{
"prompt": "A small robot reading beside a window",
"size": "1024x1024",
"response_format": "file"
}' \
--output image.png
```
=== "Image edit"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/images/edits" \
-F "image=@input.png" \
-F "prompt=Turn the background into a snowy mountain" \
| jq -r '.data[0].b64_json' \
| base64 --decode > edited.png
```
=== "Video"
```bash
curl "$VLLM_OMNI_BASE_URL/v1/videos" \
-F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \
-F "size=1280x720" \
-F "seconds=5"
```
The response contains a video job `id`. Poll and download it with:
```bash
curl "$VLLM_OMNI_BASE_URL/v1/videos/<video_id>"
curl -L "$VLLM_OMNI_BASE_URL/v1/videos/<video_id>/content" --output video.mp4
```
## Related Endpoints
| Purpose | Endpoints | Reference |
|---------|-----------|-----------|
| Discovery and readiness | `GET /health`, `GET /v1/models` | This page |
| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
| TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) |
| Streaming text-to-speech | `WS /v1/audio/speech/stream` | [Streaming speech](speech_api.md#streaming-text-input-websocket) |
| Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) |
| Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) |
??? info "Specialized and experimental WebSocket endpoints"
These routes are enabled only for specific pipelines. Start with the
associated example instead of treating them as general-purpose APIs.
| Endpoint | Use case | Example |
|----------|----------|---------|
| `WS /v1/realtime` | OpenAI-style realtime audio interaction | [Qwen3-Omni realtime client](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) |
| `WS /v1/realtime/video` | Stream generated video chunks | [Streaming video generation](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) |
| `WS /v1/duplex` or `WS /v1/realtime?duplex=1` | Experimental full-duplex speech interaction | [Full-duplex examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo) |
| `WS /v1/realtime/robot/openpi` | OpenPI-compatible robot policy inference | [DreamZero](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) |
## Compatibility Notes
- A registered path does not mean every model supports it. Unsupported tasks
return an OpenAI-style error or a WebSocket `error` event.
- Use JSON for Chat, Speech, Audio Generation, and Image Generation. Use
`multipart/form-data` when uploading images, videos, or other files.
- Generation can take minutes for large diffusion models. Configure client
timeouts accordingly; prefer asynchronous video jobs over long blocking
requests.
- Model-specific parameters are vLLM-Omni extensions. Check the endpoint
reference before assuming an OpenAI SDK exposes them directly.
docs/serving/audio_generate_api.md
+3 -3
View File
@@ -4,14 +4,14 @@ vLLM-Omni provides an API for text-to-audio generation using diffusion-based mod
Unlike the [Speech API](speech_api.md) which targets text-to-speech synthesis, the Audio Generate API is designed for general-purpose audio generation from text descriptions (sound effects, music, ambient soundscapes, etc.).
Each server instance runs a single model (specified at startup via `vllm-omni serve <model> --omni`).
Each server instance runs a single model (specified at startup via `vllm serve <model> --omni`).
## Quick Start
### Start the Server
```bash
vllm-omni serve stabilityai/stable-audio-open-1.0 \
vllm serve stabilityai/stable-audio-open-1.0 \
--host 0.0.0.0 \
--port 8091 \
--gpu-memory-utilization 0.9 \
@@ -326,7 +326,7 @@ curl http://localhost:8091/health
Enable debug logging:
```bash
vllm-omni serve stabilityai/stable-audio-open-1.0 \
vllm serve stabilityai/stable-audio-open-1.0 \
--host 0.0.0.0 \
--port 8091 \
--gpu-memory-utilization 0.9 \
docs/serving/chat_completions_api.md
+86 -54
View File
@@ -1,78 +1,110 @@
# Chat Completions API
vLLM-Omni supports generating and editing images via the `/v1/chat/completions`
endpoint using diffusion models. This page explains how to pass generation
parameters (such as `num_inference_steps`, `height`, `width`) to diffusion
models through this endpoint.
Use `/v1/chat/completions` for conversational and multimodal pipelines that
follow the OpenAI Chat API schema. Depending on the loaded model, a request can
contain text, image, audio, or video input and can return text, audio, images,
or other model-specific output.
!!! tip
For dedicated endpoints that accept generation parameters as top-level
fields, see [Image Generation API](image_generation_api.md) and
[Image Edit API](image_edit_api.md).
Some diffusion pipelines also support image generation and editing through
Chat Completions. Prefer the dedicated [Image Generation
API](image_generation_api.md) or [Image Edit API](image_edit_api.md) for
task-specific clients; their request fields and response contracts are more
direct.
## Passing Generation Parameters
The `/v1/chat/completions` endpoint follows the OpenAI Chat API schema, which
does not natively include diffusion-specific fields like `num_inference_steps`
or `height`. How you pass these extra fields depends on your client.
### curl / Python `requests`
Wrap generation parameters inside an `"extra_body"` key in the JSON body:
## Basic Request
```bash
curl -s http://localhost:8091/v1/chat/completions \
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "A beautiful landscape painting"}
{"role": "user", "content": "Describe vLLM-Omni briefly."}
],
"extra_body": {
"num_inference_steps": 50,
"seed": 42
}
"modalities": ["text"]
}'
```
### OpenAI Python SDK
Use `stream: true` for Server-Sent Events. Input media syntax, supported output
`modalities`, and response choices depend on the model; see the
[model-specific online serving examples](../user_guide/examples/online_serving/qwen3_omni.md).
Use the `extra_body` **keyword argument**. The SDK automatically merges these
fields into the top-level request body:
!!! tip
Each server hosts one model. Query `GET /v1/models` for its served model
name when your client requires the `model` field.
```python
response = client.chat.completions.create(
model="Qwen/Qwen-Image",
messages=[{"role": "user", "content": "A beautiful landscape painting"}],
extra_body={
"num_inference_steps": 50,
"seed": 42,
},
)
## vLLM Extension Parameters
Send standard and vLLM-specific Chat parameters as top-level fields in a
direct HTTP request. When using the OpenAI Python SDK, pass vLLM-specific
fields through the SDK's `extra_body` keyword; the SDK merges them into the
top-level JSON sent to the server.
=== "curl"
```bash
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "Write a short haiku."}],
"top_k": 40
}'
```
=== "OpenAI Python SDK"
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.chat.completions.create(
model="your-served-model",
messages=[{"role": "user", "content": "Write a short haiku."}],
extra_body={"top_k": 40},
)
```
??? note "Diffusion compatibility"
Diffusion models exposed through Chat Completions can accept supported
fields such as `num_inference_steps`, `seed`, `height`, and `width`. Send
them at the top level with direct HTTP, or in the SDK's `extra_body`
keyword argument.
A literal nested `"extra_body"` JSON object is accepted for compatibility,
but it is not recommended for new direct HTTP clients. Do not provide the
same parameter in multiple locations; the server returns a `400` error for
duplicate diffusion parameters. Prefer the dedicated [Image Generation
API](image_generation_api.md) or [Image Edit API](image_edit_api.md) when
either endpoint matches the task.
## Batch Requests
`POST /v1/chat/completions/batch` accepts the same shared generation fields,
but `messages` is a list of conversations. The response contains one choice
per conversation in input order.
```bash
curl http://localhost:8091/v1/chat/completions/batch \
-H "Content-Type: application/json" \
-d '{
"messages": [
[{"role": "user", "content": "Summarize vLLM in one sentence."}],
[{"role": "user", "content": "Summarize vLLM-Omni in one sentence."}]
],
"max_tokens": 64
}'
```
!!! note "SDK `extra_body` vs. JSON `extra_body`"
These two `extra_body` usages look similar but work differently under the
hood. The SDK flattens the dict into the top-level request JSON, while the
curl/requests approach sends it as a nested `"extra_body"` key. Both are
handled correctly by the server.
!!! note "About the `ignored fields` warning"
You may see a log message like:
```
WARNING: The following fields were present in the request but ignored: {'height', 'width', ...}
```
This is **harmless**. It is emitted by vLLM's request validation layer
because these fields are not part of the standard OpenAI
`ChatCompletionRequest` schema. The fields are still stored internally
and correctly forwarded to the diffusion pipeline.
Batch Chat Completions does not support streaming, tools, beam search, or
`n > 1`.
## Model-Specific Examples
For complete examples with full request/response details, see the model-specific
guides:
For complete examples with model-specific inputs and outputs, see:
- [Qwen3-Omni](../user_guide/examples/online_serving/qwen3_omni.md)
- [Qwen2.5-Omni](../user_guide/examples/online_serving/qwen2_5_omni.md)
- [Text-to-Image (Qwen-Image)](../user_guide/examples/online_serving/text_to_image.md)
- [Image-to-Image (Qwen-Image-Edit, Qwen-Image-Layered)](../user_guide/examples/online_serving/image_to_image.md)
- [GLM-Image](../user_guide/examples/online_serving/glm_image.md)
docs/user_guide/examples/online_serving/text_to_audio.md
+1 -1
View File
@@ -15,7 +15,7 @@ This example demonstrates how to deploy Stable Audio models for online text-to-a
### Basic Start
```bash
vllm-omni serve stabilityai/stable-audio-open-1.0 \
vllm serve stabilityai/stable-audio-open-1.0 \
--host 0.0.0.0 \
--port 8091 \
--gpu-memory-utilization 0.9 \
examples/online_serving/stable_audio/README.md
+1 -1
View File
@@ -14,7 +14,7 @@ Generate audio from text prompts using Stable Audio models via an OpenAI-compati
### 1. Start the Server
```bash
vllm-omni serve stabilityai/stable-audio-open-1.0 \
vllm serve stabilityai/stable-audio-open-1.0 \
--host 0.0.0.0 \
--port 8091 \
--gpu-memory-utilization 0.9 \