From 5c6ed9b4a9143dfcbc8d874e1074c7cdf7f155ad Mon Sep 17 00:00:00 2001 From: Hongsheng Liu Date: Wed, 12 Aug 2026 22:56:38 +0800 Subject: [PATCH] docs: add API server endpoint guide Signed-off-by: Hongsheng Liu --- docs/.nav.yml | 3 +- docs/examples/README.md | 4 +- docs/getting_started/quickstart.md | 24 +-- docs/serving/README.md | 173 ++++++++++++++++++ docs/serving/audio_generate_api.md | 6 +- docs/serving/chat_completions_api.md | 140 ++++++++------ .../examples/online_serving/text_to_audio.md | 2 +- .../online_serving/stable_audio/README.md | 2 +- 8 files changed, 279 insertions(+), 75 deletions(-) create mode 100644 docs/serving/README.md diff --git a/docs/.nav.yml b/docs/.nav.yml index 730c43be6..10eb6f773 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -5,7 +5,8 @@ nav: - getting_started/quickstart.md - getting_started/installation/* - Serving: - - OpenAI-Compatible API: + - API Server: + - Overview: serving/README.md - Chat Completions API: serving/chat_completions_api.md - Image Generation: serving/image_generation_api.md - Image Edit: serving/image_edit_api.md diff --git a/docs/examples/README.md b/docs/examples/README.md index 4594d2bed..30e3fce7f 100644 --- a/docs/examples/README.md +++ b/docs/examples/README.md @@ -3,4 +3,6 @@ vLLM-Omni's examples are split into two categories: - If you are using vLLM-Omni from within Python code, see the *Offline Inference* section. -- If you are using vLLM-Omni from an HTTP application or client, see the *Online Serving* section. +- If you are using vLLM-Omni from an HTTP application or client, start with + the [API Server guide](../serving/README.md) to choose an endpoint, then use + the *Online Serving* section for model-specific examples. diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index f9673eac7..0d8323760 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -92,7 +92,7 @@ if __name__ == "__main__": For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md) -## Online Serving with OpenAI-Completions API +## Online Serving with the API Server Text-to-image generation quickstart with vLLM-Omni: @@ -101,20 +101,16 @@ vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 ``` ```bash -curl -s http://localhost:8091/v1/chat/completions \ +curl -s http://localhost:8091/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ - "messages": [ - {"role": "user", "content": "a cup of coffee on the table"} - ], - "extra_body": { - "height": 1024, - "width": 1024, - "num_inference_steps": 50, - "guidance_scale": 4.0, - "seed": 42 - } - }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > coffee.png + "prompt": "a cup of coffee on the table", + "size": "1024x1024", + "response_format": "b64_json", + "seed": 42 + }' | jq -r '.data[0].b64_json' | base64 -d > coffee.png ``` -For more details, please refer to [online serving](../user_guide/examples/online_serving/text_to_image.md). +See the [API Server guide](../serving/README.md) to choose an endpoint. For +model-specific details, refer to the [text-to-image online serving +example](../user_guide/examples/online_serving/text_to_image.md). diff --git a/docs/serving/README.md b/docs/serving/README.md new file mode 100644 index 000000000..baeefbe10 --- /dev/null +++ b/docs/serving/README.md @@ -0,0 +1,173 @@ +# API Server + +vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for +image, audio, video, and realtime workloads. Use this page to choose an +endpoint; follow the linked reference page for model-specific fields and +response details. + +Each server instance hosts one model. An endpoint is usable only when the +loaded model supports that task. + +This guide focuses on vLLM-Omni's primary task endpoints. Compatible models +may also expose standard routes inherited from vLLM, such as +`POST /v1/completions`; those routes remain model- and task-dependent. + +## Start and Verify the Server + +Start a model with the shared serving command: + +```bash +vllm serve --omni --port 8000 +``` + +Some models require additional flags or a deployment configuration. Use the +model-specific guide when one is provided. + +After the server starts, check its health and served model name: + +```bash +export VLLM_OMNI_BASE_URL=http://localhost:8000 + +curl "$VLLM_OMNI_BASE_URL/health" +curl "$VLLM_OMNI_BASE_URL/v1/models" | jq . +``` + +Use `http://localhost:8000/v1` as the `base_url` for an OpenAI SDK client. If +the server was started with `--api-key`, include an `Authorization` header with +`Bearer ` in requests. + +## Choose an Endpoint + +Prefer a task-specific endpoint when one matches your workload. Use Chat +Completions for conversational or heterogeneous omni pipelines, rather than as +the default wrapper for every generation task. + +| Task | Endpoint | Request | Response | Details | +|------|----------|---------|----------|---------| +| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) | +| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) | +| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) | +| Text-to-image | `POST /v1/images/generations` | JSON | Base64 JSON or an image file | [Image Generation](image_generation_api.md) | +| Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) | +| Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) | +| Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) | +| Live video understanding | `WS /v1/video/chat/stream` | WebSocket events | Streaming text/audio events | [Streaming Video Input](video_stream_api.md) | + +`POST /v1/videos/sync` is intended for tests and simple scripts. Use the +asynchronous job API for production requests so clients can poll, download, +and delete generated videos independently. + +## Minimal Requests + +The examples below assume a compatible model is already running. TTS voices, +media inputs, generation controls, and output capabilities vary by model. + +=== "Chat" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/chat/completions" \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "Describe vLLM-Omni briefly."}], + "modalities": ["text"] + }' + ``` + +=== "Speech" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/audio/speech" \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello from vLLM-Omni.", + "voice": "vivian" + }' \ + --output speech.wav + ``` + +=== "Audio" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/audio/generate" \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Ocean waves on a quiet beach", + "audio_length": 5.0 + }' \ + --output audio.wav + ``` + +=== "Image" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/images/generations" \ + -H "Content-Type: application/json" \ + -d '{ + "prompt": "A small robot reading beside a window", + "size": "1024x1024", + "response_format": "file" + }' \ + --output image.png + ``` + +=== "Image edit" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/images/edits" \ + -F "image=@input.png" \ + -F "prompt=Turn the background into a snowy mountain" \ + | jq -r '.data[0].b64_json' \ + | base64 --decode > edited.png + ``` + +=== "Video" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/videos" \ + -F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \ + -F "size=1280x720" \ + -F "seconds=5" + ``` + + The response contains a video job `id`. Poll and download it with: + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/videos/" + curl -L "$VLLM_OMNI_BASE_URL/v1/videos//content" --output video.mp4 + ``` + +## Related Endpoints + +| Purpose | Endpoints | Reference | +|---------|-----------|-----------| +| Discovery and readiness | `GET /health`, `GET /v1/models` | This page | +| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) | +| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) | +| TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) | +| Streaming text-to-speech | `WS /v1/audio/speech/stream` | [Streaming speech](speech_api.md#streaming-text-input-websocket) | +| Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) | +| Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) | + +??? info "Specialized and experimental WebSocket endpoints" + + These routes are enabled only for specific pipelines. Start with the + associated example instead of treating them as general-purpose APIs. + + | Endpoint | Use case | Example | + |----------|----------|---------| + | `WS /v1/realtime` | OpenAI-style realtime audio interaction | [Qwen3-Omni realtime client](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) | + | `WS /v1/realtime/video` | Stream generated video chunks | [Streaming video generation](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) | + | `WS /v1/duplex` or `WS /v1/realtime?duplex=1` | Experimental full-duplex speech interaction | [Full-duplex examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo) | + | `WS /v1/realtime/robot/openpi` | OpenPI-compatible robot policy inference | [DreamZero](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) | + +## Compatibility Notes + +- A registered path does not mean every model supports it. Unsupported tasks + return an OpenAI-style error or a WebSocket `error` event. +- Use JSON for Chat, Speech, Audio Generation, and Image Generation. Use + `multipart/form-data` when uploading images, videos, or other files. +- Generation can take minutes for large diffusion models. Configure client + timeouts accordingly; prefer asynchronous video jobs over long blocking + requests. +- Model-specific parameters are vLLM-Omni extensions. Check the endpoint + reference before assuming an OpenAI SDK exposes them directly. diff --git a/docs/serving/audio_generate_api.md b/docs/serving/audio_generate_api.md index 3109446c1..7fd7e0d74 100644 --- a/docs/serving/audio_generate_api.md +++ b/docs/serving/audio_generate_api.md @@ -4,14 +4,14 @@ vLLM-Omni provides an API for text-to-audio generation using diffusion-based mod Unlike the [Speech API](speech_api.md) which targets text-to-speech synthesis, the Audio Generate API is designed for general-purpose audio generation from text descriptions (sound effects, music, ambient soundscapes, etc.). -Each server instance runs a single model (specified at startup via `vllm-omni serve --omni`). +Each server instance runs a single model (specified at startup via `vllm serve --omni`). ## Quick Start ### Start the Server ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \ @@ -326,7 +326,7 @@ curl http://localhost:8091/health Enable debug logging: ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \ diff --git a/docs/serving/chat_completions_api.md b/docs/serving/chat_completions_api.md index d8f852f99..0807c815b 100644 --- a/docs/serving/chat_completions_api.md +++ b/docs/serving/chat_completions_api.md @@ -1,78 +1,110 @@ # Chat Completions API -vLLM-Omni supports generating and editing images via the `/v1/chat/completions` -endpoint using diffusion models. This page explains how to pass generation -parameters (such as `num_inference_steps`, `height`, `width`) to diffusion -models through this endpoint. +Use `/v1/chat/completions` for conversational and multimodal pipelines that +follow the OpenAI Chat API schema. Depending on the loaded model, a request can +contain text, image, audio, or video input and can return text, audio, images, +or other model-specific output. -!!! tip - For dedicated endpoints that accept generation parameters as top-level - fields, see [Image Generation API](image_generation_api.md) and - [Image Edit API](image_edit_api.md). +Some diffusion pipelines also support image generation and editing through +Chat Completions. Prefer the dedicated [Image Generation +API](image_generation_api.md) or [Image Edit API](image_edit_api.md) for +task-specific clients; their request fields and response contracts are more +direct. -## Passing Generation Parameters - -The `/v1/chat/completions` endpoint follows the OpenAI Chat API schema, which -does not natively include diffusion-specific fields like `num_inference_steps` -or `height`. How you pass these extra fields depends on your client. - -### curl / Python `requests` - -Wrap generation parameters inside an `"extra_body"` key in the JSON body: +## Basic Request ```bash -curl -s http://localhost:8091/v1/chat/completions \ +curl http://localhost:8091/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ - {"role": "user", "content": "A beautiful landscape painting"} + {"role": "user", "content": "Describe vLLM-Omni briefly."} ], - "extra_body": { - "num_inference_steps": 50, - "seed": 42 - } + "modalities": ["text"] }' ``` -### OpenAI Python SDK +Use `stream: true` for Server-Sent Events. Input media syntax, supported output +`modalities`, and response choices depend on the model; see the +[model-specific online serving examples](../user_guide/examples/online_serving/qwen3_omni.md). -Use the `extra_body` **keyword argument**. The SDK automatically merges these -fields into the top-level request body: +!!! tip + Each server hosts one model. Query `GET /v1/models` for its served model + name when your client requires the `model` field. -```python -response = client.chat.completions.create( - model="Qwen/Qwen-Image", - messages=[{"role": "user", "content": "A beautiful landscape painting"}], - extra_body={ - "num_inference_steps": 50, - "seed": 42, - }, -) +## vLLM Extension Parameters + +Send standard and vLLM-specific Chat parameters as top-level fields in a +direct HTTP request. When using the OpenAI Python SDK, pass vLLM-specific +fields through the SDK's `extra_body` keyword; the SDK merges them into the +top-level JSON sent to the server. + +=== "curl" + + ```bash + curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "Write a short haiku."}], + "top_k": 40 + }' + ``` + +=== "OpenAI Python SDK" + + ```python + from openai import OpenAI + + client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") + + response = client.chat.completions.create( + model="your-served-model", + messages=[{"role": "user", "content": "Write a short haiku."}], + extra_body={"top_k": 40}, + ) + ``` + +??? note "Diffusion compatibility" + + Diffusion models exposed through Chat Completions can accept supported + fields such as `num_inference_steps`, `seed`, `height`, and `width`. Send + them at the top level with direct HTTP, or in the SDK's `extra_body` + keyword argument. + + A literal nested `"extra_body"` JSON object is accepted for compatibility, + but it is not recommended for new direct HTTP clients. Do not provide the + same parameter in multiple locations; the server returns a `400` error for + duplicate diffusion parameters. Prefer the dedicated [Image Generation + API](image_generation_api.md) or [Image Edit API](image_edit_api.md) when + either endpoint matches the task. + +## Batch Requests + +`POST /v1/chat/completions/batch` accepts the same shared generation fields, +but `messages` is a list of conversations. The response contains one choice +per conversation in input order. + +```bash +curl http://localhost:8091/v1/chat/completions/batch \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [ + [{"role": "user", "content": "Summarize vLLM in one sentence."}], + [{"role": "user", "content": "Summarize vLLM-Omni in one sentence."}] + ], + "max_tokens": 64 + }' ``` -!!! note "SDK `extra_body` vs. JSON `extra_body`" - These two `extra_body` usages look similar but work differently under the - hood. The SDK flattens the dict into the top-level request JSON, while the - curl/requests approach sends it as a nested `"extra_body"` key. Both are - handled correctly by the server. - -!!! note "About the `ignored fields` warning" - You may see a log message like: - - ``` - WARNING: The following fields were present in the request but ignored: {'height', 'width', ...} - ``` - - This is **harmless**. It is emitted by vLLM's request validation layer - because these fields are not part of the standard OpenAI - `ChatCompletionRequest` schema. The fields are still stored internally - and correctly forwarded to the diffusion pipeline. +Batch Chat Completions does not support streaming, tools, beam search, or +`n > 1`. ## Model-Specific Examples -For complete examples with full request/response details, see the model-specific -guides: +For complete examples with model-specific inputs and outputs, see: +- [Qwen3-Omni](../user_guide/examples/online_serving/qwen3_omni.md) +- [Qwen2.5-Omni](../user_guide/examples/online_serving/qwen2_5_omni.md) - [Text-to-Image (Qwen-Image)](../user_guide/examples/online_serving/text_to_image.md) - [Image-to-Image (Qwen-Image-Edit, Qwen-Image-Layered)](../user_guide/examples/online_serving/image_to_image.md) - [GLM-Image](../user_guide/examples/online_serving/glm_image.md) diff --git a/docs/user_guide/examples/online_serving/text_to_audio.md b/docs/user_guide/examples/online_serving/text_to_audio.md index a4833aef6..1b6dbc463 100644 --- a/docs/user_guide/examples/online_serving/text_to_audio.md +++ b/docs/user_guide/examples/online_serving/text_to_audio.md @@ -15,7 +15,7 @@ This example demonstrates how to deploy Stable Audio models for online text-to-a ### Basic Start ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \ diff --git a/examples/online_serving/stable_audio/README.md b/examples/online_serving/stable_audio/README.md index 3aaf34e56..954ac4c85 100644 --- a/examples/online_serving/stable_audio/README.md +++ b/examples/online_serving/stable_audio/README.md @@ -14,7 +14,7 @@ Generate audio from text prompts using Stable Audio models via an OpenAI-compati ### 1. Start the Server ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \