Merge remote-tracking branch 'origin/main' into codex-api-server-user-guide

Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>

# Conflicts:
#	docs/user_guide/examples/online_serving/text_to_audio.md
#	examples/online_serving/stable_audio/README.md

docs/.nav.yml

@@ -5,14 +5,22 @@ nav:
     - getting_started/quickstart.md
     - getting_started/installation/*
   - Serving:
-    - OpenAI-Compatible API:
-      - Chat Completions API: serving/chat_completions_api.md
-      - Image Generation: serving/image_generation_api.md
-      - Image Edit: serving/image_edit_api.md
-      - Text to Speech: serving/speech_api.md
-      - Videos: serving/videos_api.md
-      - Audio Generation: serving/audio_generate_api.md
-      - Streaming Video Input: serving/video_stream_api.md
+    - API Server:
+      - Overview: serving/README.md
+      - Core APIs:
+        - Chat Completions: serving/chat_completions_api.md
+        - Text to Speech: serving/speech_api.md
+        - Audio Generation: serving/audio_generate_api.md
+        - Image Generation: serving/image_generation_api.md
+        - Image Edit: serving/image_edit_api.md
+        - Videos: serving/videos_api.md
+      - Streaming and Realtime:
+        - Realtime Audio: serving/realtime_api.md
+        - Full Duplex: serving/full_duplex_api.md
+        - Streaming Video Input: serving/video_stream_api.md
+        - Streaming Video Output: serving/streaming_video_output_api.md
+        - OpenPI Robot Policy: serving/openpi_api.md
+      - Standalone Experimental Servers: serving/standalone_servers.md
   - Examples:
     - examples/README.md
     - Offline Inference:

docs/examples/README.md

@@ -3,4 +3,6 @@
 vLLM-Omni's examples are split into two categories:
 
 - If you are using vLLM-Omni from within Python code, see the *Offline Inference* section.
-- If you are using vLLM-Omni from an HTTP application or client, see the *Online Serving* section.
+- If you are using vLLM-Omni from an HTTP application or client, start with
+  the [API Server guide](../serving/README.md) to choose an endpoint, then use
+  the *Online Serving* section for model-specific examples.

docs/getting_started/quickstart.md

@@ -91,7 +91,7 @@ if __name__ == "__main__":
 
 For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md)
 
-## Online Serving with OpenAI-Completions API
+## Online Serving with the API Server
 
 Text-to-image generation quickstart with vLLM-Omni:
 
@@ -100,20 +100,16 @@ vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091
 ```
 
 ```bash
-curl -s http://localhost:8091/v1/chat/completions \
+curl -s http://localhost:8091/v1/images/generations \
   -H "Content-Type: application/json" \
   -d '{
-    "messages": [
-      {"role": "user", "content": "a cup of coffee on the table"}
-    ],
-    "extra_body": {
-      "height": 1024,
-      "width": 1024,
-      "num_inference_steps": 50,
-      "guidance_scale": 4.0,
-      "seed": 42
-    }
-  }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > coffee.png
+    "prompt": "a cup of coffee on the table",
+    "size": "1024x1024",
+    "response_format": "b64_json",
+    "seed": 42
+  }' | jq -r '.data[0].b64_json' | base64 -d > coffee.png
 ```
 
-For more details, please refer to [online serving](../user_guide/examples/online_serving/text_to_image.md).
+See the [API Server guide](../serving/README.md) to choose an endpoint. For
+model-specific details, refer to the [text-to-image online serving
+example](../user_guide/examples/online_serving/text_to_image.md).

docs/serving/README.md

@@ -0,0 +1,187 @@
+# API Server
+
+vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for
+image, audio, video, realtime, and robot-policy workloads. Use this page to
+choose an endpoint; follow the linked reference page for model-specific fields,
+wire protocols, and response details.
+
+Each server instance hosts one model. An endpoint is usable only when the
+loaded model supports that task.
+
+This guide focuses on vLLM-Omni's primary task endpoints. Compatible models
+may also expose standard routes inherited from vLLM, such as
+`POST /v1/completions`; those routes remain model- and task-dependent.
+
+## Start and Verify the Server
+
+Start a model with the shared serving command:
+
+```bash
+vllm serve <model> --omni --port 8091
+```
+
+Some models require additional flags or a deployment configuration. Use the
+model-specific guide when one is provided.
+
+After the server starts, check its health and served model name:
+
+```bash
+export VLLM_OMNI_BASE_URL=http://localhost:8091
+
+curl "$VLLM_OMNI_BASE_URL/health"
+curl "$VLLM_OMNI_BASE_URL/v1/models" | jq .
+```
+
+Use `http://localhost:8091/v1` as the `base_url` for an OpenAI SDK client. If
+the server was started with `--api-key`, include an `Authorization` header with
+`Bearer <api-key>` in requests.
+
+## Choose a Core HTTP Endpoint
+
+Prefer a task-specific endpoint when one matches your workload. Use Chat
+Completions for conversational or heterogeneous omni pipelines, rather than as
+the default wrapper for every generation task.
+
+| Task | Endpoint | Request | Response | Details |
+|------|----------|---------|----------|---------|
+| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) |
+| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) |
+| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) |
+| Text-to-image | `POST /v1/images/generations` | JSON | Base64 JSON or an image file | [Image Generation](image_generation_api.md) |
+| Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) |
+| Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) |
+| Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) |
+
+`POST /v1/videos/sync` is intended for tests and simple scripts. Use the
+asynchronous job API for production requests so clients can poll, download,
+and delete generated videos independently.
+
+## Minimal Requests
+
+The examples below assume a compatible model is already running. TTS voices,
+media inputs, generation controls, and output capabilities vary by model.
+
+=== "Chat"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/chat/completions" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "messages": [{"role": "user", "content": "Describe vLLM-Omni briefly."}],
+        "modalities": ["text"]
+      }'
+    ```
+
+=== "Speech"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/audio/speech" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "input": "Hello from vLLM-Omni.",
+        "voice": "vivian"
+      }' \
+      --output speech.wav
+    ```
+
+=== "Audio"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/audio/generate" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "input": "Ocean waves on a quiet beach",
+        "audio_length": 5.0
+      }' \
+      --output audio.wav
+    ```
+
+=== "Image"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/images/generations" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "prompt": "A small robot reading beside a window",
+        "size": "1024x1024",
+        "response_format": "file"
+      }' \
+      --output image.png
+    ```
+
+=== "Image edit"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/images/edits" \
+      -F "image=@input.png" \
+      -F "prompt=Turn the background into a snowy mountain" \
+      | jq -r '.data[0].b64_json' \
+      | base64 --decode > edited.png
+    ```
+
+=== "Video"
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/videos" \
+      -F "prompt=A cinematic tracking shot of a mountain lake at sunrise" \
+      -F "size=1280x720" \
+      -F "seconds=5"
+    ```
+
+    The response contains a video job `id`. Poll and download it with:
+
+    ```bash
+    curl "$VLLM_OMNI_BASE_URL/v1/videos/<video_id>"
+    curl -L "$VLLM_OMNI_BASE_URL/v1/videos/<video_id>/content" --output video.mp4
+    ```
+
+## Choose a Streaming or Realtime Endpoint
+
+These WebSocket APIs have different event schemas and cannot be used
+interchangeably.
+
+| Workload | Endpoint | Interaction model | Details |
+|----------|----------|-------------------|---------|
+| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
+| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
+| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
+| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
+| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
+| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |
+
+All six routes are model- or configuration-dependent. In particular,
+`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the
+deployment explicitly enables duplex sessions. Clients should also verify the
+duplex capability payload because the query-parameter form falls back to the
+ordinary realtime handler when duplex is unavailable.
+
+## Related Endpoints
+
+| Purpose | Endpoints | Reference |
+|---------|-----------|-----------|
+| Discovery and readiness | `GET /health`, `GET /v1/models` | This page |
+| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
+| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
+| TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) |
+| Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) |
+| Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) |
+
+## Standalone Experimental Servers
+
+PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL
+provides a stateful interaction orchestrator in front of another model server.
+These are separate processes with their own routes; they are not additional
+paths on every `vllm serve --omni` instance. See [Standalone Experimental
+Servers](standalone_servers.md) before deploying either one.
+
+## Compatibility Notes
+
+- A registered path does not mean every model supports it. Unsupported tasks
+  return an OpenAI-style error or a WebSocket `error` event.
+- Use JSON for Chat, Speech, Audio Generation, and Image Generation. Use
+  `multipart/form-data` when uploading images, videos, or other files.
+- Generation can take minutes for large diffusion models. Configure client
+  timeouts accordingly; prefer asynchronous video jobs over long blocking
+  requests.
+- Model-specific parameters are vLLM-Omni extensions. Check the endpoint
+  reference before assuming an OpenAI SDK exposes them directly.

docs/serving/audio_generate_api.md

@@ -4,14 +4,14 @@ vLLM-Omni provides an API for text-to-audio generation using diffusion-based mod
 
 Unlike the [Speech API](speech_api.md) which targets text-to-speech synthesis, the Audio Generate API is designed for general-purpose audio generation from text descriptions (sound effects, music, ambient soundscapes, etc.).
 
-Each server instance runs a single model (specified at startup via `vllm-omni serve <model> --omni`).
+Each server instance runs a single model (specified at startup via `vllm serve <model> --omni`).
 
 ## Quick Start
 
 ### Start the Server
 
 ```bash
-vllm-omni serve stabilityai/stable-audio-open-1.0 \
+vllm serve stabilityai/stable-audio-open-1.0 \
     --host 0.0.0.0 \
     --port 8091 \
     --gpu-memory-utilization 0.9 \
@@ -326,7 +326,7 @@ curl http://localhost:8091/health
 Enable debug logging:
 
 ```bash
-vllm-omni serve stabilityai/stable-audio-open-1.0 \
+vllm serve stabilityai/stable-audio-open-1.0 \
     --host 0.0.0.0 \
     --port 8091 \
     --gpu-memory-utilization 0.9 \

docs/serving/chat_completions_api.md

@@ -1,78 +1,110 @@
 # Chat Completions API
 
-vLLM-Omni supports generating and editing images via the `/v1/chat/completions`
-endpoint using diffusion models. This page explains how to pass generation
-parameters (such as `num_inference_steps`, `height`, `width`) to diffusion
-models through this endpoint.
+Use `/v1/chat/completions` for conversational and multimodal pipelines that
+follow the OpenAI Chat API schema. Depending on the loaded model, a request can
+contain text, image, audio, or video input and can return text, audio, images,
+or other model-specific output.
 
-!!! tip
-    For dedicated endpoints that accept generation parameters as top-level
-    fields, see [Image Generation API](image_generation_api.md) and
-    [Image Edit API](image_edit_api.md).
+Some diffusion pipelines also support image generation and editing through
+Chat Completions. Prefer the dedicated [Image Generation
+API](image_generation_api.md) or [Image Edit API](image_edit_api.md) for
+task-specific clients; their request fields and response contracts are more
+direct.
 
-## Passing Generation Parameters
-
-The `/v1/chat/completions` endpoint follows the OpenAI Chat API schema, which
-does not natively include diffusion-specific fields like `num_inference_steps`
-or `height`. How you pass these extra fields depends on your client.
-
-### curl / Python `requests`
-
-Wrap generation parameters inside an `"extra_body"` key in the JSON body:
+## Basic Request
 
 ```bash
-curl -s http://localhost:8091/v1/chat/completions \
+curl http://localhost:8091/v1/chat/completions \
   -H "Content-Type: application/json" \
   -d '{
     "messages": [
-      {"role": "user", "content": "A beautiful landscape painting"}
+      {"role": "user", "content": "Describe vLLM-Omni briefly."}
     ],
-    "extra_body": {
-      "num_inference_steps": 50,
-      "seed": 42
-    }
+    "modalities": ["text"]
   }'
 ```
 
-### OpenAI Python SDK
+Use `stream: true` for Server-Sent Events. Input media syntax, supported output
+`modalities`, and response choices depend on the model; see the
+[model-specific online serving examples](../user_guide/examples/online_serving/qwen3_omni.md).
 
-Use the `extra_body` **keyword argument**. The SDK automatically merges these
-fields into the top-level request body:
+!!! tip
+    Each server hosts one model. Query `GET /v1/models` for its served model
+    name when your client requires the `model` field.
 
-```python
-response = client.chat.completions.create(
-    model="Qwen/Qwen-Image",
-    messages=[{"role": "user", "content": "A beautiful landscape painting"}],
-    extra_body={
-        "num_inference_steps": 50,
-        "seed": 42,
-    },
-)
+## vLLM Extension Parameters
+
+Send standard and vLLM-specific Chat parameters as top-level fields in a
+direct HTTP request. When using the OpenAI Python SDK, pass vLLM-specific
+fields through the SDK's `extra_body` keyword; the SDK merges them into the
+top-level JSON sent to the server.
+
+=== "curl"
+
+    ```bash
+    curl http://localhost:8091/v1/chat/completions \
+      -H "Content-Type: application/json" \
+      -d '{
+        "messages": [{"role": "user", "content": "Write a short haiku."}],
+        "top_k": 40
+      }'
+    ```
+
+=== "OpenAI Python SDK"
+
+    ```python
+    from openai import OpenAI
+
+    client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
+
+    response = client.chat.completions.create(
+        model="your-served-model",
+        messages=[{"role": "user", "content": "Write a short haiku."}],
+        extra_body={"top_k": 40},
+    )
+    ```
+
+??? note "Diffusion compatibility"
+
+    Diffusion models exposed through Chat Completions can accept supported
+    fields such as `num_inference_steps`, `seed`, `height`, and `width`. Send
+    them at the top level with direct HTTP, or in the SDK's `extra_body`
+    keyword argument.
+
+    A literal nested `"extra_body"` JSON object is accepted for compatibility,
+    but it is not recommended for new direct HTTP clients. Do not provide the
+    same parameter in multiple locations; the server returns a `400` error for
+    duplicate diffusion parameters. Prefer the dedicated [Image Generation
+    API](image_generation_api.md) or [Image Edit API](image_edit_api.md) when
+    either endpoint matches the task.
+
+## Batch Requests
+
+`POST /v1/chat/completions/batch` accepts the same shared generation fields,
+but `messages` is a list of conversations. The response contains one choice
+per conversation in input order.
+
+```bash
+curl http://localhost:8091/v1/chat/completions/batch \
+  -H "Content-Type: application/json" \
+  -d '{
+    "messages": [
+      [{"role": "user", "content": "Summarize vLLM in one sentence."}],
+      [{"role": "user", "content": "Summarize vLLM-Omni in one sentence."}]
+    ],
+    "max_tokens": 64
+  }'
 ```
 
-!!! note "SDK `extra_body` vs. JSON `extra_body`"
-    These two `extra_body` usages look similar but work differently under the
-    hood. The SDK flattens the dict into the top-level request JSON, while the
-    curl/requests approach sends it as a nested `"extra_body"` key. Both are
-    handled correctly by the server.
-
-!!! note "About the `ignored fields` warning"
-    You may see a log message like:
-
-    ```
-    WARNING: The following fields were present in the request but ignored: {'height', 'width', ...}
-    ```
-
-    This is **harmless**. It is emitted by vLLM's request validation layer
-    because these fields are not part of the standard OpenAI
-    `ChatCompletionRequest` schema. The fields are still stored internally
-    and correctly forwarded to the diffusion pipeline.
+Batch Chat Completions does not support streaming, tools, beam search, or
+`n > 1`.
 
 ## Model-Specific Examples
 
-For complete examples with full request/response details, see the model-specific
-guides:
+For complete examples with model-specific inputs and outputs, see:
 
+- [Qwen3-Omni](../user_guide/examples/online_serving/qwen3_omni.md)
+- [Qwen2.5-Omni](../user_guide/examples/online_serving/qwen2_5_omni.md)
 - [Text-to-Image (Qwen-Image)](../user_guide/examples/online_serving/text_to_image.md)
 - [Image-to-Image (Qwen-Image-Edit, Qwen-Image-Layered)](../user_guide/examples/online_serving/image_to_image.md)
 - [GLM-Image](../user_guide/examples/online_serving/glm_image.md)

docs/serving/full_duplex_api.md

@@ -0,0 +1,116 @@
+# Full-Duplex WebSocket API
+
+vLLM-Omni provides an experimental full-duplex runtime for models that can
+continue receiving speech while producing speech. It adds persistent session
+state, model-specific turn policy, overlap handling, playback acknowledgement,
+and optional session resume.
+
+Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
+
+## Choose an Endpoint
+
+| Endpoint | Protocol | Recommended use |
+|----------|----------|-----------------|
+| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
+| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
+
+Both endpoints use the same duplex engine and require the same model-side
+adapter. Prefer the Realtime projection unless the native lifecycle events are
+specifically required.
+
+## Enable Full Duplex
+
+The route becomes usable only when the deployment configuration explicitly
+sets:
+
+```yaml
+session_mode: duplex
+```
+
+The selected model pipeline must also provide a duplex serving adapter.
+Model-native deployments configure the corresponding engine runtime extension
+and control plane as part of their registered pipeline.
+
+!!! warning
+
+    `WS /v1/duplex` fails with `Duplex API is not available` when duplex is not
+    enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary
+    turn-based realtime handler when no duplex handler exists. Confirm that
+    `session.created.session.capabilities` is present before treating the
+    connection as full duplex.
+
+The current unified-runtime integrations are:
+
+- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`;
+- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex.
+
+JoyVL is a separate HTTP interaction orchestrator and does not use these
+WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
+
+## MiniCPM-o Quick Start
+
+Start the duplex deployment:
+
+```bash
+vllm serve openbmb/MiniCPM-o-4_5 --omni \
+  --deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
+  --trust-remote-code \
+  --port 8091
+```
+
+Stream a mono, PCM16, 16 kHz WAV file with the provided client:
+
+```bash
+python examples/online_serving/minicpmo/realtime_duplex_demo.py \
+  --url 'ws://localhost:8091/v1/realtime?duplex=1' \
+  --model openbmb/MiniCPM-o-4_5 \
+  --input-wav input_16k_mono.wav \
+  --ref-audio reference_voice.wav \
+  --output-dir /tmp/minicpmo-duplex
+```
+
+## Realtime Event Lifecycle
+
+A typical `/v1/realtime?duplex=1` session follows this lifecycle:
+
+1. Send `session.update` with the model, modalities, audio formats, and session
+   options.
+2. Wait for `session.created`; inspect `session.capabilities` instead of
+   assuming every model supports the same controls.
+3. Send `input_audio_buffer.append` events while microphone audio arrives.
+4. Send `input_audio_buffer.commit` at a user-turn boundary when required by
+   the model policy.
+5. Consume `response.created`, transcript deltas, `response.audio.delta`, and
+   `response.done` or `response.listen` events.
+6. Send `playback.ack` after audio has been played when the session advertises
+   playback acknowledgement support.
+7. Send `session.close` and wait for `session.closed`.
+
+Unlike the turn-based realtime endpoint, input may continue while a response
+is active. The server can emit `overlap.decision` to describe whether input was
+deferred, treated as a short acknowledgement, or used to interrupt output.
+
+## Capabilities and Model Differences
+
+The `session.created` payload includes capability fields such as
+`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`,
+`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the
+runtime contract.
+
+For example, PersonaPlex supports native overlapping speech but currently
+advertises `supports_barge_in=false`; destructive output interruption and
+model-state rewind have not been validated for that integration. Capacity and
+session-resume behavior also depend on the selected deployment configuration.
+
+## Native Protocol
+
+`WS /v1/duplex` exposes lower-level lifecycle names including
+`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`,
+and `session.close`. It returns native session, input, response, overlap, and
+error events. This protocol is experimental and may evolve with the runtime;
+applications should use the provided Realtime client where possible.
+
+See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
+[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
+and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md)
+for model-specific validation and architecture details.

docs/serving/openpi_api.md

@@ -0,0 +1,109 @@
+# OpenPI Robot Policy WebSocket API
+
+Use `WS /v1/realtime/robot/openpi` for low-latency robot-policy inference with
+OpenPI-compatible clients. A persistent WebSocket carries observations to the
+loaded policy model and returns action arrays.
+
+Despite the `/realtime` prefix, this endpoint does not use the OpenAI Realtime
+event schema. It uses binary MessagePack frames with NumPy extensions.
+
+## Availability
+
+The endpoint is available only when the loaded diffusion policy configuration
+contains `policy_server_config`. The server sends that model-specific
+configuration to the client as the first binary frame.
+
+For example, DreamZero declares:
+
+```yaml
+policy_server_config:
+  image_resolution: [180, 320]
+  n_external_cameras: 2
+  needs_wrist_camera: true
+  needs_stereo_camera: false
+  needs_session_id: true
+  action_space: joint_position
+```
+
+Without this configuration, the WebSocket returns `Robot policy not available`
+and closes.
+
+## DreamZero Quick Start
+
+Start the bundled DreamZero deployment:
+
+```bash
+vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8091 \
+  --served-model-name dreamzero-droid \
+  --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
+  --enforce-eager --disable-log-stats
+```
+
+Install the optional client dependencies, download the sample camera inputs,
+and run the client:
+
+```bash
+pip install openpi-client websockets opencv-python
+
+hf download YangshenDeng/vllm-omni-dreamzero-assets \
+  --repo-type dataset \
+  --local-dir outputs/dreamzero/assets
+
+python examples/online_serving/dreamzero/openpi_client.py \
+  --host 127.0.0.1 \
+  --port 8091 \
+  --video-dir outputs/dreamzero/assets
+```
+
+## Protocol
+
+The connection is request-response after an initial server handshake:
+
+```text
+connect
+  <- msgpack(policy_server_config)
+
+infer
+  -> msgpack({"endpoint": "infer", "session_id": "...", ...observation})
+  <- msgpack(ndarray | dict[str, ndarray])
+
+reset
+  -> msgpack({"endpoint": "reset"})
+  <- msgpack({"status": "reset successful"})
+```
+
+If `endpoint` is omitted, the server treats the message as `infer`. Each
+inference message produces one action response after the engine request
+completes; action tokens or intermediate tensors are not streamed.
+
+NumPy arrays use the marker format implemented by `openpi-client`. The server
+also accepts the legacy vLLM NumPy marker representation on input. JSON text
+frames are not observation messages.
+
+## Sessions and Reset
+
+- The API layer's current-session and first-call counters are scoped to each
+  WebSocket connection.
+- `session_id` identifies model-side state across observations on that
+  connection. If omitted, it defaults to `default`.
+- The first inference for a session is sent to the model with `reset=true`.
+- Changing `session_id`, or sending the `reset` command, causes the next
+  inference to start with `reset=true`.
+- The policy pipeline owns observation transforms and persistent model state,
+  normally keyed by `session_id`; the API layer forwards the raw observation
+  dictionary.
+
+## Limits and Errors
+
+- Maximum inbound payload size is 64 MiB.
+- The server closes an idle connection after 30 seconds.
+- Invalid binary input returns a MessagePack
+  `{"type":"error","message":"Invalid request payload"}` response.
+- Inference failures return a generic `Internal inference error` without
+  exposing an internal traceback.
+- Unsupported NumPy object, structured, and complex dtypes are rejected.
+
+Observation keys, camera layout, state tensors, and action shapes are defined
+by the loaded policy rather than by this transport. Read the handshake before
+constructing observations. See the [DreamZero example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero)
+for a complete OpenPI client and DROID simulation loop.

docs/serving/realtime_api.md

@@ -0,0 +1,82 @@
+# Realtime Audio WebSocket API
+
+Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and
+receive incremental transcription and synthesized response audio. This is a
+turn-based streaming API: it starts one generation over an audio stream and
+finishes after the client closes that input stream.
+
+For a model that listens while it is already speaking, use the experimental
+[Full-Duplex API](full_duplex_api.md) instead.
+
+## Quick Start
+
+Start a compatible model. The currently documented example uses Qwen3-Omni:
+
+```bash
+vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
+```
+
+Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:
+
+```bash
+python examples/online_serving/qwen3_omni/openai_realtime_client.py \
+  --url ws://localhost:8091/v1/realtime \
+  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
+  --input-wav input_16k_mono.wav \
+  --output-wav response.wav \
+  --output-text response.txt
+```
+
+The client dependency is `websockets`.
+
+## Protocol
+
+Messages in both directions are JSON text frames. Audio carried inside an
+event is base64-encoded raw PCM16.
+
+| Direction | Event | Purpose |
+|-----------|-------|---------|
+| Server to client | `session.created` | Confirms that the WebSocket connection is ready |
+| Client to server | `session.update` | Selects and validates the served model |
+| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
+| Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio |
+| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
+| Server to client | `transcription.delta` | Carries incremental response text |
+| Server to client | `transcription.done` | Carries final text and token usage |
+| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio |
+| Server to client | `response.audio.done` | Marks the end of response audio |
+| Server to client | `error` | Reports an invalid event, model, or audio payload |
+
+A minimal client sends events in this order:
+
+```json
+{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+{"type":"input_audio_buffer.commit","final":false}
+{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
+{"type":"input_audio_buffer.commit","final":true}
+```
+
+The initial non-final commit intentionally starts generation before all audio
+has arrived. Continue sending `input_audio_buffer.append` events while the
+engine is consuming the stream.
+
+## Audio Handling
+
+- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
+- `response.audio.delta.audio` contains base64-encoded PCM16 bytes.
+- Read `sample_rate_hz` from each audio event instead of assuming an output
+  rate. Qwen3-Omni output is typically 24 kHz.
+- Concatenate audio deltas in receive order to construct the output waveform.
+- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so
+  the server can terminate the streaming request.
+
+## Availability and Limitations
+
+The path is registered on the unified API server, but it is usable only when
+the loaded pipeline implements realtime audio input and produces compatible
+audio output. Unsupported deployments return an `error` event. The endpoint
+does not provide duplex session resume, playback acknowledgement, overlap
+policy, or barge-in controls.
+
+See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni)
+for concurrency options, chunk pacing, and per-delta audio debugging.

docs/serving/standalone_servers.md

@@ -0,0 +1,109 @@
+# Standalone Experimental Servers
+
+Most vLLM-Omni endpoints run on the unified server started with
+`vllm serve <model> --omni`. The experimental full-duplex package also contains
+two separate user-facing server processes. Their routes are not added to the
+unified API server.
+
+| Server | Transport | Role |
+|--------|-----------|------|
+| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative |
+| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend |
+
+## PersonaPlex Compatibility Server
+
+Use this server when a client requires the official Moshi/PersonaPlex wire
+protocol or the bundled browser UI. For the engine-owned unified duplex path,
+use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead.
+
+Start it with:
+
+```bash
+HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
+  vllm_omni.experimental.fullduplex.personaplex.serving.server \
+  --port 8091
+```
+
+| Route | Purpose |
+|-------|---------|
+| `GET /` | Official PersonaPlex browser client, when its assets are available |
+| `GET /health` | Readiness check |
+| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio |
+| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol |
+
+### `/api/chat` Protocol
+
+The first byte of every binary frame is a message tag:
+
+| Direction | Tag | Payload |
+|-----------|-----|---------|
+| Server to client | `0x00` | Ready handshake |
+| Client to server | `0x01` | Opus microphone audio at 24 kHz |
+| Server to client | `0x01` | Opus response audio |
+| Server to client | `0x02` | UTF-8 inner-monologue text |
+
+The optional `text_prompt` and `voice_prompt` query parameters select a persona
+and voice. The default server admits one conversation; use `--batch-size N` to
+configure multiple lockstep slots. Connections beyond capacity close with
+WebSocket code 1013.
+
+### `/v1/audio/duplex` Protocol
+
+Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame,
+then exchange binary float32 PCM frames. The server returns generated float32
+audio as binary frames and text as `{"type":"text","text":"..."}`. Send
+`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish.
+
+See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
+for the browser and headless clients.
+
+## JoyVL Interaction Server
+
+JoyVL is an orchestration layer, not a model-serving engine. It calls a
+separate OpenAI-compatible backend and adds per-session frame history, memory,
+persona policy, and optional delegation.
+
+Start the model backend and orchestrator separately:
+
+```bash
+vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
+  --served-model-name JoyAI-VL-Interaction-Preview \
+  --port 8092 \
+  --max-model-len 131072 \
+  --enable-prefix-caching \
+  --limit-mm-per-prompt '{"image":256,"video":1}'
+
+python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
+  --port 8091 \
+  --main-backend-url http://127.0.0.1:8092/v1 \
+  --main-model JoyAI-VL-Interaction-Preview
+```
+
+| Method and route | Purpose |
+|------------------|---------|
+| `GET /health` | Readiness check |
+| `GET /v1/models` | Reports the configured interaction model |
+| `POST /v1/chat/completions` | Processes one frame or interaction tick |
+| `POST /reset` | Resets a session |
+| `POST /v1/streaming/reset` | Alias for `/reset` |
+| `POST /v1/streaming/persona` | Changes the session persona |
+
+Send one multimodal Chat Completions request per video frame, normally around
+one frame per second, and identify the session with `x-session-id`. Responses
+include an `interaction` block whose action is `silence`, `response`, or
+`delegate`.
+
+```bash
+curl http://127.0.0.1:8091/v1/chat/completions \
+  -H 'content-type: application/json' \
+  -H 'x-session-id: demo' \
+  -d '{
+    "messages": [{"role": "user", "content": [
+      {"type": "text", "text": "Alert me if a fire appears"},
+      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
+    ]}]
+  }'
+```
+
+See the [JoyVL recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/JD/JoyAI-VL-Interaction.md)
+for memory, personas, delegation backends, and the browser UI.

docs/serving/streaming_video_output_api.md

@@ -0,0 +1,92 @@
+# Streaming Video Output WebSocket API
+
+Use `WS /v1/realtime/video` to receive fragmented MP4 chunks while a compatible
+diffusion video model is generating. This endpoint streams generated video
+output; it is unrelated to [`WS /v1/video/chat/stream`](video_stream_api.md),
+which streams live video input into a conversational model.
+
+## Quick Start
+
+Start a model that supports step execution and enable diffusion streaming
+output:
+
+```bash
+vllm serve BestWishYsh/Helios-Distilled --omni \
+  --diffusion-streaming-output \
+  --port 8091
+```
+
+Install the client dependency and run the provided client:
+
+```bash
+pip install av websockets
+
+python examples/online_serving/streaming_video_generation/streaming_video_client.py \
+  --host 127.0.0.1 \
+  --port 8091 \
+  --model BestWishYsh/Helios-Distilled \
+  --prompt "A serene lakeside sunrise with mist over the water" \
+  --output output.mp4
+```
+
+## Protocol
+
+Control events are JSON text frames. Each `video.chunk_metadata` event is
+immediately followed by one binary WebSocket frame containing fragmented MP4
+bytes.
+
+| Direction | Event or frame | Purpose |
+|-----------|----------------|---------|
+| Client to server | `session.start` | Starts generation with a prompt and video parameters |
+| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration |
+| Server to client | `video.chunk_metadata` | Describes the next binary media frame |
+| Server to client | Binary frame | Carries one fragmented MP4 chunk |
+| Client to server | `session.interaction` | Queues an optional prompt update during generation |
+| Server to client | `session.interaction.queued` | Confirms that an interaction was queued |
+| Client to server | `session.ping` | Refreshes the stall clock |
+| Server to client | `session.pong` | Acknowledges a ping |
+| Client to server | `session.stop` | Cancels the active generation |
+| Server to client | `session.done` | Reports completion or cancellation and the chunk count |
+| Server to client | `error` | Reports validation, generation, or timeout failures |
+
+Start a minimal session with:
+
+```json
+{
+  "type": "session.start",
+  "model": "BestWishYsh/Helios-Distilled",
+  "prompt": "A serene lakeside sunrise",
+  "format": "m4s",
+  "width": 640,
+  "height": 384,
+  "fps": 16,
+  "num_frames": 99
+}
+```
+
+Only `m4s` output is currently accepted. Concatenate the binary frames in
+receive order, then remux the fragmented stream if the target player requires
+a progressive MP4 file.
+
+## Mid-Generation Interaction
+
+Models that implement interaction updates can change the active prompt:
+
+```json
+{
+  "type": "session.interaction",
+  "interaction": {
+    "event_id": "sunlight-update",
+    "event": {"prompt": "Sunlight breaks through the mist"},
+    "transition_chunks": 2
+  }
+}
+```
+
+This feature is pipeline-dependent. An enabled endpoint does not imply that
+the loaded model supports midway prompt updates.
+
+The default start timeout is 10 seconds and the server reports a stall after
+about 60 seconds without generation progress or a `session.ping`. See the
+[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation)
+for prompt-update scheduling and browser playback.

examples/online_serving/minicpmo/README.md

@@ -49,7 +49,7 @@ scenario below for live barge-in validation on the target GPU.
 Default:
 
 ```bash
-vllm-omni serve openbmb/MiniCPM-o-4_5 \
+vllm serve openbmb/MiniCPM-o-4_5 \
     --omni \
     --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
     --trust-remote-code \

examples/online_serving/streaming_video_generation/README.md

@@ -37,7 +37,7 @@ During generation the client normally sends only `session.start` and then receiv
 ## Install Client Dependency
 
 ```bash
-pip install websockets
+pip install av websockets
 # For the Gradio demo:
 pip install 'vllm-omni[demo]' websockets
 ```

examples/online_serving/streaming_video_generation/streaming_video_client.py

@@ -8,7 +8,7 @@ the received bytes to disk when the session finishes or is interrupted.
 Image/reference input is intentionally not supported in this example yet.
 
 Requirements:
-    pip install websockets
+    pip install av websockets
 """
 
 import argparse