diff --git a/docs/.nav.yml b/docs/.nav.yml index 1de85696a..94c4de213 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -5,14 +5,22 @@ nav: - getting_started/quickstart.md - getting_started/installation/* - Serving: - - OpenAI-Compatible API: - - Chat Completions API: serving/chat_completions_api.md - - Image Generation: serving/image_generation_api.md - - Image Edit: serving/image_edit_api.md - - Text to Speech: serving/speech_api.md - - Videos: serving/videos_api.md - - Audio Generation: serving/audio_generate_api.md - - Streaming Video Input: serving/video_stream_api.md + - API Server: + - Overview: serving/README.md + - Core APIs: + - Chat Completions: serving/chat_completions_api.md + - Text to Speech: serving/speech_api.md + - Audio Generation: serving/audio_generate_api.md + - Image Generation: serving/image_generation_api.md + - Image Edit: serving/image_edit_api.md + - Videos: serving/videos_api.md + - Streaming and Realtime: + - Realtime Audio: serving/realtime_api.md + - Full Duplex: serving/full_duplex_api.md + - Streaming Video Input: serving/video_stream_api.md + - Streaming Video Output: serving/streaming_video_output_api.md + - OpenPI Robot Policy: serving/openpi_api.md + - Standalone Experimental Servers: serving/standalone_servers.md - Examples: - examples/README.md - Offline Inference: diff --git a/docs/examples/README.md b/docs/examples/README.md index 4594d2bed..30e3fce7f 100644 --- a/docs/examples/README.md +++ b/docs/examples/README.md @@ -3,4 +3,6 @@ vLLM-Omni's examples are split into two categories: - If you are using vLLM-Omni from within Python code, see the *Offline Inference* section. -- If you are using vLLM-Omni from an HTTP application or client, see the *Online Serving* section. +- If you are using vLLM-Omni from an HTTP application or client, start with + the [API Server guide](../serving/README.md) to choose an endpoint, then use + the *Online Serving* section for model-specific examples. diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index 439c7aedc..036c2da17 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -91,7 +91,7 @@ if __name__ == "__main__": For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md) -## Online Serving with OpenAI-Completions API +## Online Serving with the API Server Text-to-image generation quickstart with vLLM-Omni: @@ -100,20 +100,16 @@ vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091 ``` ```bash -curl -s http://localhost:8091/v1/chat/completions \ +curl -s http://localhost:8091/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ - "messages": [ - {"role": "user", "content": "a cup of coffee on the table"} - ], - "extra_body": { - "height": 1024, - "width": 1024, - "num_inference_steps": 50, - "guidance_scale": 4.0, - "seed": 42 - } - }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > coffee.png + "prompt": "a cup of coffee on the table", + "size": "1024x1024", + "response_format": "b64_json", + "seed": 42 + }' | jq -r '.data[0].b64_json' | base64 -d > coffee.png ``` -For more details, please refer to [online serving](../user_guide/examples/online_serving/text_to_image.md). +See the [API Server guide](../serving/README.md) to choose an endpoint. For +model-specific details, refer to the [text-to-image online serving +example](../user_guide/examples/online_serving/text_to_image.md). diff --git a/docs/serving/README.md b/docs/serving/README.md new file mode 100644 index 000000000..6acfa534a --- /dev/null +++ b/docs/serving/README.md @@ -0,0 +1,187 @@ +# API Server + +vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for +image, audio, video, realtime, and robot-policy workloads. Use this page to +choose an endpoint; follow the linked reference page for model-specific fields, +wire protocols, and response details. + +Each server instance hosts one model. An endpoint is usable only when the +loaded model supports that task. + +This guide focuses on vLLM-Omni's primary task endpoints. Compatible models +may also expose standard routes inherited from vLLM, such as +`POST /v1/completions`; those routes remain model- and task-dependent. + +## Start and Verify the Server + +Start a model with the shared serving command: + +```bash +vllm serve --omni --port 8091 +``` + +Some models require additional flags or a deployment configuration. Use the +model-specific guide when one is provided. + +After the server starts, check its health and served model name: + +```bash +export VLLM_OMNI_BASE_URL=http://localhost:8091 + +curl "$VLLM_OMNI_BASE_URL/health" +curl "$VLLM_OMNI_BASE_URL/v1/models" | jq . +``` + +Use `http://localhost:8091/v1` as the `base_url` for an OpenAI SDK client. If +the server was started with `--api-key`, include an `Authorization` header with +`Bearer ` in requests. + +## Choose a Core HTTP Endpoint + +Prefer a task-specific endpoint when one matches your workload. Use Chat +Completions for conversational or heterogeneous omni pipelines, rather than as +the default wrapper for every generation task. + +| Task | Endpoint | Request | Response | Details | +|------|----------|---------|----------|---------| +| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) | +| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) | +| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) | +| Text-to-image | `POST /v1/images/generations` | JSON | Base64 JSON or an image file | [Image Generation](image_generation_api.md) | +| Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) | +| Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) | +| Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) | + +`POST /v1/videos/sync` is intended for tests and simple scripts. Use the +asynchronous job API for production requests so clients can poll, download, +and delete generated videos independently. + +## Minimal Requests + +The examples below assume a compatible model is already running. TTS voices, +media inputs, generation controls, and output capabilities vary by model. + +=== "Chat" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/chat/completions" \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "Describe vLLM-Omni briefly."}], + "modalities": ["text"] + }' + ``` + +=== "Speech" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/audio/speech" \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello from vLLM-Omni.", + "voice": "vivian" + }' \ + --output speech.wav + ``` + +=== "Audio" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/audio/generate" \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Ocean waves on a quiet beach", + "audio_length": 5.0 + }' \ + --output audio.wav + ``` + +=== "Image" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/images/generations" \ + -H "Content-Type: application/json" \ + -d '{ + "prompt": "A small robot reading beside a window", + "size": "1024x1024", + "response_format": "file" + }' \ + --output image.png + ``` + +=== "Image edit" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/images/edits" \ + -F "image=@input.png" \ + -F "prompt=Turn the background into a snowy mountain" \ + | jq -r '.data[0].b64_json' \ + | base64 --decode > edited.png + ``` + +=== "Video" + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/videos" \ + -F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \ + -F "size=1280x720" \ + -F "seconds=5" + ``` + + The response contains a video job `id`. Poll and download it with: + + ```bash + curl "$VLLM_OMNI_BASE_URL/v1/videos/" + curl -L "$VLLM_OMNI_BASE_URL/v1/videos//content" --output video.mp4 + ``` + +## Choose a Streaming or Realtime Endpoint + +These WebSocket APIs have different event schemas and cannot be used +interchangeably. + +| Workload | Endpoint | Interaction model | Details | +|----------|----------|-------------------|---------| +| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) | +| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) | +| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) | +| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) | +| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) | +| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) | + +All six routes are model- or configuration-dependent. In particular, +`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the +deployment explicitly enables duplex sessions. Clients should also verify the +duplex capability payload because the query-parameter form falls back to the +ordinary realtime handler when duplex is unavailable. + +## Related Endpoints + +| Purpose | Endpoints | Reference | +|---------|-----------|-----------| +| Discovery and readiness | `GET /health`, `GET /v1/models` | This page | +| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) | +| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) | +| TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) | +| Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) | +| Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) | + +## Standalone Experimental Servers + +PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL +provides a stateful interaction orchestrator in front of another model server. +These are separate processes with their own routes; they are not additional +paths on every `vllm serve --omni` instance. See [Standalone Experimental +Servers](standalone_servers.md) before deploying either one. + +## Compatibility Notes + +- A registered path does not mean every model supports it. Unsupported tasks + return an OpenAI-style error or a WebSocket `error` event. +- Use JSON for Chat, Speech, Audio Generation, and Image Generation. Use + `multipart/form-data` when uploading images, videos, or other files. +- Generation can take minutes for large diffusion models. Configure client + timeouts accordingly; prefer asynchronous video jobs over long blocking + requests. +- Model-specific parameters are vLLM-Omni extensions. Check the endpoint + reference before assuming an OpenAI SDK exposes them directly. diff --git a/docs/serving/audio_generate_api.md b/docs/serving/audio_generate_api.md index 3109446c1..7fd7e0d74 100644 --- a/docs/serving/audio_generate_api.md +++ b/docs/serving/audio_generate_api.md @@ -4,14 +4,14 @@ vLLM-Omni provides an API for text-to-audio generation using diffusion-based mod Unlike the [Speech API](speech_api.md) which targets text-to-speech synthesis, the Audio Generate API is designed for general-purpose audio generation from text descriptions (sound effects, music, ambient soundscapes, etc.). -Each server instance runs a single model (specified at startup via `vllm-omni serve --omni`). +Each server instance runs a single model (specified at startup via `vllm serve --omni`). ## Quick Start ### Start the Server ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \ @@ -326,7 +326,7 @@ curl http://localhost:8091/health Enable debug logging: ```bash -vllm-omni serve stabilityai/stable-audio-open-1.0 \ +vllm serve stabilityai/stable-audio-open-1.0 \ --host 0.0.0.0 \ --port 8091 \ --gpu-memory-utilization 0.9 \ diff --git a/docs/serving/chat_completions_api.md b/docs/serving/chat_completions_api.md index d8f852f99..0807c815b 100644 --- a/docs/serving/chat_completions_api.md +++ b/docs/serving/chat_completions_api.md @@ -1,78 +1,110 @@ # Chat Completions API -vLLM-Omni supports generating and editing images via the `/v1/chat/completions` -endpoint using diffusion models. This page explains how to pass generation -parameters (such as `num_inference_steps`, `height`, `width`) to diffusion -models through this endpoint. +Use `/v1/chat/completions` for conversational and multimodal pipelines that +follow the OpenAI Chat API schema. Depending on the loaded model, a request can +contain text, image, audio, or video input and can return text, audio, images, +or other model-specific output. -!!! tip - For dedicated endpoints that accept generation parameters as top-level - fields, see [Image Generation API](image_generation_api.md) and - [Image Edit API](image_edit_api.md). +Some diffusion pipelines also support image generation and editing through +Chat Completions. Prefer the dedicated [Image Generation +API](image_generation_api.md) or [Image Edit API](image_edit_api.md) for +task-specific clients; their request fields and response contracts are more +direct. -## Passing Generation Parameters - -The `/v1/chat/completions` endpoint follows the OpenAI Chat API schema, which -does not natively include diffusion-specific fields like `num_inference_steps` -or `height`. How you pass these extra fields depends on your client. - -### curl / Python `requests` - -Wrap generation parameters inside an `"extra_body"` key in the JSON body: +## Basic Request ```bash -curl -s http://localhost:8091/v1/chat/completions \ +curl http://localhost:8091/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ - {"role": "user", "content": "A beautiful landscape painting"} + {"role": "user", "content": "Describe vLLM-Omni briefly."} ], - "extra_body": { - "num_inference_steps": 50, - "seed": 42 - } + "modalities": ["text"] }' ``` -### OpenAI Python SDK +Use `stream: true` for Server-Sent Events. Input media syntax, supported output +`modalities`, and response choices depend on the model; see the +[model-specific online serving examples](../user_guide/examples/online_serving/qwen3_omni.md). -Use the `extra_body` **keyword argument**. The SDK automatically merges these -fields into the top-level request body: +!!! tip + Each server hosts one model. Query `GET /v1/models` for its served model + name when your client requires the `model` field. -```python -response = client.chat.completions.create( - model="Qwen/Qwen-Image", - messages=[{"role": "user", "content": "A beautiful landscape painting"}], - extra_body={ - "num_inference_steps": 50, - "seed": 42, - }, -) +## vLLM Extension Parameters + +Send standard and vLLM-specific Chat parameters as top-level fields in a +direct HTTP request. When using the OpenAI Python SDK, pass vLLM-specific +fields through the SDK's `extra_body` keyword; the SDK merges them into the +top-level JSON sent to the server. + +=== "curl" + + ```bash + curl http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [{"role": "user", "content": "Write a short haiku."}], + "top_k": 40 + }' + ``` + +=== "OpenAI Python SDK" + + ```python + from openai import OpenAI + + client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") + + response = client.chat.completions.create( + model="your-served-model", + messages=[{"role": "user", "content": "Write a short haiku."}], + extra_body={"top_k": 40}, + ) + ``` + +??? note "Diffusion compatibility" + + Diffusion models exposed through Chat Completions can accept supported + fields such as `num_inference_steps`, `seed`, `height`, and `width`. Send + them at the top level with direct HTTP, or in the SDK's `extra_body` + keyword argument. + + A literal nested `"extra_body"` JSON object is accepted for compatibility, + but it is not recommended for new direct HTTP clients. Do not provide the + same parameter in multiple locations; the server returns a `400` error for + duplicate diffusion parameters. Prefer the dedicated [Image Generation + API](image_generation_api.md) or [Image Edit API](image_edit_api.md) when + either endpoint matches the task. + +## Batch Requests + +`POST /v1/chat/completions/batch` accepts the same shared generation fields, +but `messages` is a list of conversations. The response contains one choice +per conversation in input order. + +```bash +curl http://localhost:8091/v1/chat/completions/batch \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [ + [{"role": "user", "content": "Summarize vLLM in one sentence."}], + [{"role": "user", "content": "Summarize vLLM-Omni in one sentence."}] + ], + "max_tokens": 64 + }' ``` -!!! note "SDK `extra_body` vs. JSON `extra_body`" - These two `extra_body` usages look similar but work differently under the - hood. The SDK flattens the dict into the top-level request JSON, while the - curl/requests approach sends it as a nested `"extra_body"` key. Both are - handled correctly by the server. - -!!! note "About the `ignored fields` warning" - You may see a log message like: - - ``` - WARNING: The following fields were present in the request but ignored: {'height', 'width', ...} - ``` - - This is **harmless**. It is emitted by vLLM's request validation layer - because these fields are not part of the standard OpenAI - `ChatCompletionRequest` schema. The fields are still stored internally - and correctly forwarded to the diffusion pipeline. +Batch Chat Completions does not support streaming, tools, beam search, or +`n > 1`. ## Model-Specific Examples -For complete examples with full request/response details, see the model-specific -guides: +For complete examples with model-specific inputs and outputs, see: +- [Qwen3-Omni](../user_guide/examples/online_serving/qwen3_omni.md) +- [Qwen2.5-Omni](../user_guide/examples/online_serving/qwen2_5_omni.md) - [Text-to-Image (Qwen-Image)](../user_guide/examples/online_serving/text_to_image.md) - [Image-to-Image (Qwen-Image-Edit, Qwen-Image-Layered)](../user_guide/examples/online_serving/image_to_image.md) - [GLM-Image](../user_guide/examples/online_serving/glm_image.md) diff --git a/docs/serving/full_duplex_api.md b/docs/serving/full_duplex_api.md new file mode 100644 index 000000000..c7c2ac44c --- /dev/null +++ b/docs/serving/full_duplex_api.md @@ -0,0 +1,116 @@ +# Full-Duplex WebSocket API + +vLLM-Omni provides an experimental full-duplex runtime for models that can +continue receiving speech while producing speech. It adds persistent session +state, model-specific turn policy, overlap handling, playback acknowledgement, +and optional session resume. + +Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md). + +## Choose an Endpoint + +| Endpoint | Protocol | Recommended use | +|----------|----------|-----------------| +| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients | +| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing | + +Both endpoints use the same duplex engine and require the same model-side +adapter. Prefer the Realtime projection unless the native lifecycle events are +specifically required. + +## Enable Full Duplex + +The route becomes usable only when the deployment configuration explicitly +sets: + +```yaml +session_mode: duplex +``` + +The selected model pipeline must also provide a duplex serving adapter. +Model-native deployments configure the corresponding engine runtime extension +and control plane as part of their registered pipeline. + +!!! warning + + `WS /v1/duplex` fails with `Duplex API is not available` when duplex is not + enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary + turn-based realtime handler when no duplex handler exists. Confirm that + `session.created.session.capabilities` is present before treating the + connection as full duplex. + +The current unified-runtime integrations are: + +- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`; +- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex. + +JoyVL is a separate HTTP interaction orchestrator and does not use these +WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md). + +## MiniCPM-o Quick Start + +Start the duplex deployment: + +```bash +vllm serve openbmb/MiniCPM-o-4_5 --omni \ + --deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \ + --trust-remote-code \ + --port 8091 +``` + +Stream a mono, PCM16, 16 kHz WAV file with the provided client: + +```bash +python examples/online_serving/minicpmo/realtime_duplex_demo.py \ + --url 'ws://localhost:8091/v1/realtime?duplex=1' \ + --model openbmb/MiniCPM-o-4_5 \ + --input-wav input_16k_mono.wav \ + --ref-audio reference_voice.wav \ + --output-dir /tmp/minicpmo-duplex +``` + +## Realtime Event Lifecycle + +A typical `/v1/realtime?duplex=1` session follows this lifecycle: + +1. Send `session.update` with the model, modalities, audio formats, and session + options. +2. Wait for `session.created`; inspect `session.capabilities` instead of + assuming every model supports the same controls. +3. Send `input_audio_buffer.append` events while microphone audio arrives. +4. Send `input_audio_buffer.commit` at a user-turn boundary when required by + the model policy. +5. Consume `response.created`, transcript deltas, `response.audio.delta`, and + `response.done` or `response.listen` events. +6. Send `playback.ack` after audio has been played when the session advertises + playback acknowledgement support. +7. Send `session.close` and wait for `session.closed`. + +Unlike the turn-based realtime endpoint, input may continue while a response +is active. The server can emit `overlap.decision` to describe whether input was +deferred, treated as a short acknowledgement, or used to interrupt output. + +## Capabilities and Model Differences + +The `session.created` payload includes capability fields such as +`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`, +`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the +runtime contract. + +For example, PersonaPlex supports native overlapping speech but currently +advertises `supports_barge_in=false`; destructive output interruption and +model-state rewind have not been validated for that integration. Capacity and +session-resume behavior also depend on the selected deployment configuration. + +## Native Protocol + +`WS /v1/duplex` exposes lower-level lifecycle names including +`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`, +and `session.close`. It returns native session, input, response, overlap, and +error events. This protocol is experimental and may evolve with the runtime; +applications should use the provided Realtime client where possible. + +See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo), +[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex), +and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md) +for model-specific validation and architecture details. diff --git a/docs/serving/openpi_api.md b/docs/serving/openpi_api.md new file mode 100644 index 000000000..27f092d87 --- /dev/null +++ b/docs/serving/openpi_api.md @@ -0,0 +1,109 @@ +# OpenPI Robot Policy WebSocket API + +Use `WS /v1/realtime/robot/openpi` for low-latency robot-policy inference with +OpenPI-compatible clients. A persistent WebSocket carries observations to the +loaded policy model and returns action arrays. + +Despite the `/realtime` prefix, this endpoint does not use the OpenAI Realtime +event schema. It uses binary MessagePack frames with NumPy extensions. + +## Availability + +The endpoint is available only when the loaded diffusion policy configuration +contains `policy_server_config`. The server sends that model-specific +configuration to the client as the first binary frame. + +For example, DreamZero declares: + +```yaml +policy_server_config: + image_resolution: [180, 320] + n_external_cameras: 2 + needs_wrist_camera: true + needs_stereo_camera: false + needs_session_id: true + action_space: joint_position +``` + +Without this configuration, the WebSocket returns `Robot policy not available` +and closes. + +## DreamZero Quick Start + +Start the bundled DreamZero deployment: + +```bash +vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8091 \ + --served-model-name dreamzero-droid \ + --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \ + --enforce-eager --disable-log-stats +``` + +Install the optional client dependencies, download the sample camera inputs, +and run the client: + +```bash +pip install openpi-client websockets opencv-python + +hf download YangshenDeng/vllm-omni-dreamzero-assets \ + --repo-type dataset \ + --local-dir outputs/dreamzero/assets + +python examples/online_serving/dreamzero/openpi_client.py \ + --host 127.0.0.1 \ + --port 8091 \ + --video-dir outputs/dreamzero/assets +``` + +## Protocol + +The connection is request-response after an initial server handshake: + +```text +connect + <- msgpack(policy_server_config) + +infer + -> msgpack({"endpoint": "infer", "session_id": "...", ...observation}) + <- msgpack(ndarray | dict[str, ndarray]) + +reset + -> msgpack({"endpoint": "reset"}) + <- msgpack({"status": "reset successful"}) +``` + +If `endpoint` is omitted, the server treats the message as `infer`. Each +inference message produces one action response after the engine request +completes; action tokens or intermediate tensors are not streamed. + +NumPy arrays use the marker format implemented by `openpi-client`. The server +also accepts the legacy vLLM NumPy marker representation on input. JSON text +frames are not observation messages. + +## Sessions and Reset + +- The API layer's current-session and first-call counters are scoped to each + WebSocket connection. +- `session_id` identifies model-side state across observations on that + connection. If omitted, it defaults to `default`. +- The first inference for a session is sent to the model with `reset=true`. +- Changing `session_id`, or sending the `reset` command, causes the next + inference to start with `reset=true`. +- The policy pipeline owns observation transforms and persistent model state, + normally keyed by `session_id`; the API layer forwards the raw observation + dictionary. + +## Limits and Errors + +- Maximum inbound payload size is 64 MiB. +- The server closes an idle connection after 30 seconds. +- Invalid binary input returns a MessagePack + `{"type":"error","message":"Invalid request payload"}` response. +- Inference failures return a generic `Internal inference error` without + exposing an internal traceback. +- Unsupported NumPy object, structured, and complex dtypes are rejected. + +Observation keys, camera layout, state tensors, and action shapes are defined +by the loaded policy rather than by this transport. Read the handshake before +constructing observations. See the [DreamZero example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) +for a complete OpenPI client and DROID simulation loop. diff --git a/docs/serving/realtime_api.md b/docs/serving/realtime_api.md new file mode 100644 index 000000000..58fe8b7c3 --- /dev/null +++ b/docs/serving/realtime_api.md @@ -0,0 +1,82 @@ +# Realtime Audio WebSocket API + +Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and +receive incremental transcription and synthesized response audio. This is a +turn-based streaming API: it starts one generation over an audio stream and +finishes after the client closes that input stream. + +For a model that listens while it is already speaking, use the experimental +[Full-Duplex API](full_duplex_api.md) instead. + +## Quick Start + +Start a compatible model. The currently documented example uses Qwen3-Omni: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 +``` + +Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file: + +```bash +python examples/online_serving/qwen3_omni/openai_realtime_client.py \ + --url ws://localhost:8091/v1/realtime \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --input-wav input_16k_mono.wav \ + --output-wav response.wav \ + --output-text response.txt +``` + +The client dependency is `websockets`. + +## Protocol + +Messages in both directions are JSON text frames. Audio carried inside an +event is base64-encoded raw PCM16. + +| Direction | Event | Purpose | +|-----------|-------|---------| +| Server to client | `session.created` | Confirms that the WebSocket connection is ready | +| Client to server | `session.update` | Selects and validates the served model | +| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream | +| Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio | +| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input | +| Server to client | `transcription.delta` | Carries incremental response text | +| Server to client | `transcription.done` | Carries final text and token usage | +| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio | +| Server to client | `response.audio.done` | Marks the end of response audio | +| Server to client | `error` | Reports an invalid event, model, or audio payload | + +A minimal client sends events in this order: + +```json +{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"} +{"type":"input_audio_buffer.commit","final":false} +{"type":"input_audio_buffer.append","audio":""} +{"type":"input_audio_buffer.commit","final":true} +``` + +The initial non-final commit intentionally starts generation before all audio +has arrived. Continue sending `input_audio_buffer.append` events while the +engine is consuming the stream. + +## Audio Handling + +- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example. +- `response.audio.delta.audio` contains base64-encoded PCM16 bytes. +- Read `sample_rate_hz` from each audio event instead of assuming an output + rate. Qwen3-Omni output is typically 24 kHz. +- Concatenate audio deltas in receive order to construct the output waveform. +- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so + the server can terminate the streaming request. + +## Availability and Limitations + +The path is registered on the unified API server, but it is usable only when +the loaded pipeline implements realtime audio input and produces compatible +audio output. Unsupported deployments return an `error` event. The endpoint +does not provide duplex session resume, playback acknowledgement, overlap +policy, or barge-in controls. + +See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) +for concurrency options, chunk pacing, and per-delta audio debugging. diff --git a/docs/serving/standalone_servers.md b/docs/serving/standalone_servers.md new file mode 100644 index 000000000..66e0c1e08 --- /dev/null +++ b/docs/serving/standalone_servers.md @@ -0,0 +1,109 @@ +# Standalone Experimental Servers + +Most vLLM-Omni endpoints run on the unified server started with +`vllm serve --omni`. The experimental full-duplex package also contains +two separate user-facing server processes. Their routes are not added to the +unified API server. + +| Server | Transport | Role | +|--------|-----------|------| +| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative | +| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend | + +## PersonaPlex Compatibility Server + +Use this server when a client requires the official Moshi/PersonaPlex wire +protocol or the bundled browser UI. For the engine-owned unified duplex path, +use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead. + +Start it with: + +```bash +HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \ + vllm_omni.experimental.fullduplex.personaplex.serving.server \ + --port 8091 +``` + +| Route | Purpose | +|-------|---------| +| `GET /` | Official PersonaPlex browser client, when its assets are available | +| `GET /health` | Readiness check | +| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio | +| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol | + +### `/api/chat` Protocol + +The first byte of every binary frame is a message tag: + +| Direction | Tag | Payload | +|-----------|-----|---------| +| Server to client | `0x00` | Ready handshake | +| Client to server | `0x01` | Opus microphone audio at 24 kHz | +| Server to client | `0x01` | Opus response audio | +| Server to client | `0x02` | UTF-8 inner-monologue text | + +The optional `text_prompt` and `voice_prompt` query parameters select a persona +and voice. The default server admits one conversation; use `--batch-size N` to +configure multiple lockstep slots. Connections beyond capacity close with +WebSocket code 1013. + +### `/v1/audio/duplex` Protocol + +Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame, +then exchange binary float32 PCM frames. The server returns generated float32 +audio as binary frames and text as `{"type":"text","text":"..."}`. Send +`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish. + +See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex) +for the browser and headless clients. + +## JoyVL Interaction Server + +JoyVL is an orchestration layer, not a model-serving engine. It calls a +separate OpenAI-compatible backend and adds per-session frame history, memory, +persona policy, and optional delegation. + +Start the model backend and orchestrator separately: + +```bash +vllm serve jdopensource/JoyAI-VL-Interaction-Preview \ + --served-model-name JoyAI-VL-Interaction-Preview \ + --port 8092 \ + --max-model-len 131072 \ + --enable-prefix-caching \ + --limit-mm-per-prompt '{"image":256,"video":1}' + +python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \ + --port 8091 \ + --main-backend-url http://127.0.0.1:8092/v1 \ + --main-model JoyAI-VL-Interaction-Preview +``` + +| Method and route | Purpose | +|------------------|---------| +| `GET /health` | Readiness check | +| `GET /v1/models` | Reports the configured interaction model | +| `POST /v1/chat/completions` | Processes one frame or interaction tick | +| `POST /reset` | Resets a session | +| `POST /v1/streaming/reset` | Alias for `/reset` | +| `POST /v1/streaming/persona` | Changes the session persona | + +Send one multimodal Chat Completions request per video frame, normally around +one frame per second, and identify the session with `x-session-id`. Responses +include an `interaction` block whose action is `silence`, `response`, or +`delegate`. + +```bash +curl http://127.0.0.1:8091/v1/chat/completions \ + -H 'content-type: application/json' \ + -H 'x-session-id: demo' \ + -d '{ + "messages": [{"role": "user", "content": [ + {"type": "text", "text": "Alert me if a fire appears"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}} + ]}] + }' +``` + +See the [JoyVL recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/JD/JoyAI-VL-Interaction.md) +for memory, personas, delegation backends, and the browser UI. diff --git a/docs/serving/streaming_video_output_api.md b/docs/serving/streaming_video_output_api.md new file mode 100644 index 000000000..626a26e97 --- /dev/null +++ b/docs/serving/streaming_video_output_api.md @@ -0,0 +1,92 @@ +# Streaming Video Output WebSocket API + +Use `WS /v1/realtime/video` to receive fragmented MP4 chunks while a compatible +diffusion video model is generating. This endpoint streams generated video +output; it is unrelated to [`WS /v1/video/chat/stream`](video_stream_api.md), +which streams live video input into a conversational model. + +## Quick Start + +Start a model that supports step execution and enable diffusion streaming +output: + +```bash +vllm serve BestWishYsh/Helios-Distilled --omni \ + --diffusion-streaming-output \ + --port 8091 +``` + +Install the client dependency and run the provided client: + +```bash +pip install av websockets + +python examples/online_serving/streaming_video_generation/streaming_video_client.py \ + --host 127.0.0.1 \ + --port 8091 \ + --model BestWishYsh/Helios-Distilled \ + --prompt "A serene lakeside sunrise with mist over the water" \ + --output output.mp4 +``` + +## Protocol + +Control events are JSON text frames. Each `video.chunk_metadata` event is +immediately followed by one binary WebSocket frame containing fragmented MP4 +bytes. + +| Direction | Event or frame | Purpose | +|-----------|----------------|---------| +| Client to server | `session.start` | Starts generation with a prompt and video parameters | +| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration | +| Server to client | `video.chunk_metadata` | Describes the next binary media frame | +| Server to client | Binary frame | Carries one fragmented MP4 chunk | +| Client to server | `session.interaction` | Queues an optional prompt update during generation | +| Server to client | `session.interaction.queued` | Confirms that an interaction was queued | +| Client to server | `session.ping` | Refreshes the stall clock | +| Server to client | `session.pong` | Acknowledges a ping | +| Client to server | `session.stop` | Cancels the active generation | +| Server to client | `session.done` | Reports completion or cancellation and the chunk count | +| Server to client | `error` | Reports validation, generation, or timeout failures | + +Start a minimal session with: + +```json +{ + "type": "session.start", + "model": "BestWishYsh/Helios-Distilled", + "prompt": "A serene lakeside sunrise", + "format": "m4s", + "width": 640, + "height": 384, + "fps": 16, + "num_frames": 99 +} +``` + +Only `m4s` output is currently accepted. Concatenate the binary frames in +receive order, then remux the fragmented stream if the target player requires +a progressive MP4 file. + +## Mid-Generation Interaction + +Models that implement interaction updates can change the active prompt: + +```json +{ + "type": "session.interaction", + "interaction": { + "event_id": "sunlight-update", + "event": {"prompt": "Sunlight breaks through the mist"}, + "transition_chunks": 2 + } +} +``` + +This feature is pipeline-dependent. An enabled endpoint does not imply that +the loaded model supports midway prompt updates. + +The default start timeout is 10 seconds and the server reports a stall after +about 60 seconds without generation progress or a `session.ping`. See the +[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) +for prompt-update scheduling and browser playback. diff --git a/examples/online_serving/minicpmo/README.md b/examples/online_serving/minicpmo/README.md index 7cd60bda9..96c021a44 100644 --- a/examples/online_serving/minicpmo/README.md +++ b/examples/online_serving/minicpmo/README.md @@ -49,7 +49,7 @@ scenario below for live barge-in validation on the target GPU. Default: ```bash -vllm-omni serve openbmb/MiniCPM-o-4_5 \ +vllm serve openbmb/MiniCPM-o-4_5 \ --omni \ --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \ --trust-remote-code \ diff --git a/examples/online_serving/streaming_video_generation/README.md b/examples/online_serving/streaming_video_generation/README.md index 7caa74906..f2d465797 100644 --- a/examples/online_serving/streaming_video_generation/README.md +++ b/examples/online_serving/streaming_video_generation/README.md @@ -37,7 +37,7 @@ During generation the client normally sends only `session.start` and then receiv ## Install Client Dependency ```bash -pip install websockets +pip install av websockets # For the Gradio demo: pip install 'vllm-omni[demo]' websockets ``` diff --git a/examples/online_serving/streaming_video_generation/streaming_video_client.py b/examples/online_serving/streaming_video_generation/streaming_video_client.py index e8dcd8cdd..d1a9052ed 100644 --- a/examples/online_serving/streaming_video_generation/streaming_video_client.py +++ b/examples/online_serving/streaming_video_generation/streaming_video_client.py @@ -8,7 +8,7 @@ the received bytes to disk when the session finishes or is interrupted. Image/reference input is intentionally not supported in this example yet. Requirements: - pip install websockets + pip install av websockets """ import argparse