diff --git a/docs/.nav.yml b/docs/.nav.yml index 10eb6f773..f349e6fe6 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -7,13 +7,20 @@ nav: - Serving: - API Server: - Overview: serving/README.md - - Chat Completions API: serving/chat_completions_api.md - - Image Generation: serving/image_generation_api.md - - Image Edit: serving/image_edit_api.md - - Text to Speech: serving/speech_api.md - - Videos: serving/videos_api.md - - Audio Generation: serving/audio_generate_api.md - - Streaming Video Input: serving/video_stream_api.md + - Core APIs: + - Chat Completions: serving/chat_completions_api.md + - Text to Speech: serving/speech_api.md + - Audio Generation: serving/audio_generate_api.md + - Image Generation: serving/image_generation_api.md + - Image Edit: serving/image_edit_api.md + - Videos: serving/videos_api.md + - Streaming and Realtime: + - Realtime Audio: serving/realtime_api.md + - Full Duplex: serving/full_duplex_api.md + - Streaming Video Input: serving/video_stream_api.md + - Streaming Video Output: serving/streaming_video_output_api.md + - OpenPI Robot Policy: serving/openpi_api.md + - Standalone Experimental Servers: serving/standalone_servers.md - Examples: - examples/README.md - Offline Inference: diff --git a/docs/serving/README.md b/docs/serving/README.md index baeefbe10..512bbf18d 100644 --- a/docs/serving/README.md +++ b/docs/serving/README.md @@ -1,9 +1,9 @@ # API Server vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for -image, audio, video, and realtime workloads. Use this page to choose an -endpoint; follow the linked reference page for model-specific fields and -response details. +image, audio, video, realtime, and robot-policy workloads. Use this page to +choose an endpoint; follow the linked reference page for model-specific fields, +wire protocols, and response details. Each server instance hosts one model. An endpoint is usable only when the loaded model supports that task. @@ -36,7 +36,7 @@ Use `http://localhost:8000/v1` as the `base_url` for an OpenAI SDK client. If the server was started with `--api-key`, include an `Authorization` header with `Bearer ` in requests. -## Choose an Endpoint +## Choose a Core HTTP Endpoint Prefer a task-specific endpoint when one matches your workload. Use Chat Completions for conversational or heterogeneous omni pipelines, rather than as @@ -51,7 +51,6 @@ the default wrapper for every generation task. | Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) | | Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) | | Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) | -| Live video understanding | `WS /v1/video/chat/stream` | WebSocket events | Streaming text/audio events | [Streaming Video Input](video_stream_api.md) | `POST /v1/videos/sync` is intended for tests and simple scripts. Use the asynchronous job API for production requests so clients can poll, download, @@ -136,6 +135,26 @@ media inputs, generation controls, and output capabilities vary by model. curl -L "$VLLM_OMNI_BASE_URL/v1/videos//content" --output video.mp4 ``` +## Choose a Streaming or Realtime Endpoint + +These WebSocket APIs have different event schemas and cannot be used +interchangeably. + +| Workload | Endpoint | Interaction model | Details | +|----------|----------|-------------------|---------| +| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) | +| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) | +| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) | +| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) | +| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) | +| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) | + +All six routes are model- or configuration-dependent. In particular, +`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the +deployment explicitly enables duplex sessions. Clients should also verify the +duplex capability payload because the query-parameter form falls back to the +ordinary realtime handler when duplex is unavailable. + ## Related Endpoints | Purpose | Endpoints | Reference | @@ -144,21 +163,16 @@ media inputs, generation controls, and output capabilities vary by model. | Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) | | Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) | | TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) | -| Streaming text-to-speech | `WS /v1/audio/speech/stream` | [Streaming speech](speech_api.md#streaming-text-input-websocket) | | Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) | | Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) | -??? info "Specialized and experimental WebSocket endpoints" +## Standalone Experimental Servers - These routes are enabled only for specific pipelines. Start with the - associated example instead of treating them as general-purpose APIs. - - | Endpoint | Use case | Example | - |----------|----------|---------| - | `WS /v1/realtime` | OpenAI-style realtime audio interaction | [Qwen3-Omni realtime client](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) | - | `WS /v1/realtime/video` | Stream generated video chunks | [Streaming video generation](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) | - | `WS /v1/duplex` or `WS /v1/realtime?duplex=1` | Experimental full-duplex speech interaction | [Full-duplex examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo) | - | `WS /v1/realtime/robot/openpi` | OpenPI-compatible robot policy inference | [DreamZero](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) | +PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL +provides a stateful interaction orchestrator in front of another model server. +These are separate processes with their own routes; they are not additional +paths on every `vllm serve --omni` instance. See [Standalone Experimental +Servers](standalone_servers.md) before deploying either one. ## Compatibility Notes diff --git a/docs/serving/full_duplex_api.md b/docs/serving/full_duplex_api.md new file mode 100644 index 000000000..8487bbb94 --- /dev/null +++ b/docs/serving/full_duplex_api.md @@ -0,0 +1,116 @@ +# Full-Duplex WebSocket API + +vLLM-Omni provides an experimental full-duplex runtime for models that can +continue receiving speech while producing speech. It adds persistent session +state, model-specific turn policy, overlap handling, playback acknowledgement, +and optional session resume. + +Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md). + +## Choose an Endpoint + +| Endpoint | Protocol | Recommended use | +|----------|----------|-----------------| +| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients | +| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing | + +Both endpoints use the same duplex engine and require the same model-side +adapter. Prefer the Realtime projection unless the native lifecycle events are +specifically required. + +## Enable Full Duplex + +The route becomes usable only when the deployment configuration explicitly +sets: + +```yaml +session_mode: duplex +``` + +The selected model pipeline must also provide a duplex serving adapter. +Model-native deployments configure the corresponding engine runtime extension +and control plane as part of their registered pipeline. + +!!! warning + + `WS /v1/duplex` fails with `Duplex API is not available` when duplex is not + enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary + turn-based realtime handler when no duplex handler exists. Confirm that + `session.created.session.capabilities` is present before treating the + connection as full duplex. + +The current unified-runtime integrations are: + +- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`; +- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex. + +JoyVL is a separate HTTP interaction orchestrator and does not use these +WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md). + +## MiniCPM-o Quick Start + +Start the duplex deployment: + +```bash +vllm serve openbmb/MiniCPM-o-4_5 --omni \ + --deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \ + --trust-remote-code \ + --port 8099 +``` + +Stream a mono, PCM16, 16 kHz WAV file with the provided client: + +```bash +python examples/online_serving/minicpmo/realtime_duplex_demo.py \ + --url 'ws://localhost:8099/v1/realtime?duplex=1' \ + --model openbmb/MiniCPM-o-4_5 \ + --input-wav input_16k_mono.wav \ + --ref-audio reference_voice.wav \ + --output-dir /tmp/minicpmo-duplex +``` + +## Realtime Event Lifecycle + +A typical `/v1/realtime?duplex=1` session follows this lifecycle: + +1. Send `session.update` with the model, modalities, audio formats, and session + options. +2. Wait for `session.created`; inspect `session.capabilities` instead of + assuming every model supports the same controls. +3. Send `input_audio_buffer.append` events while microphone audio arrives. +4. Send `input_audio_buffer.commit` at a user-turn boundary when required by + the model policy. +5. Consume `response.created`, transcript deltas, `response.audio.delta`, and + `response.done` or `response.listen` events. +6. Send `playback.ack` after audio has been played when the session advertises + playback acknowledgement support. +7. Send `session.close` and wait for `session.closed`. + +Unlike the turn-based realtime endpoint, input may continue while a response +is active. The server can emit `overlap.decision` to describe whether input was +deferred, treated as a short acknowledgement, or used to interrupt output. + +## Capabilities and Model Differences + +The `session.created` payload includes capability fields such as +`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`, +`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the +runtime contract. + +For example, PersonaPlex supports native overlapping speech but currently +advertises `supports_barge_in=false`; destructive output interruption and +model-state rewind have not been validated for that integration. Capacity and +session-resume behavior also depend on the selected deployment configuration. + +## Native Protocol + +`WS /v1/duplex` exposes lower-level lifecycle names including +`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`, +and `session.close`. It returns native session, input, response, overlap, and +error events. This protocol is experimental and may evolve with the runtime; +applications should use the provided Realtime client where possible. + +See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo), +[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex), +and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md) +for model-specific validation and architecture details. diff --git a/docs/serving/openpi_api.md b/docs/serving/openpi_api.md new file mode 100644 index 000000000..222df58dc --- /dev/null +++ b/docs/serving/openpi_api.md @@ -0,0 +1,109 @@ +# OpenPI Robot Policy WebSocket API + +Use `WS /v1/realtime/robot/openpi` for low-latency robot-policy inference with +OpenPI-compatible clients. A persistent WebSocket carries observations to the +loaded policy model and returns action arrays. + +Despite the `/realtime` prefix, this endpoint does not use the OpenAI Realtime +event schema. It uses binary MessagePack frames with NumPy extensions. + +## Availability + +The endpoint is available only when the loaded diffusion policy configuration +contains `policy_server_config`. The server sends that model-specific +configuration to the client as the first binary frame. + +For example, DreamZero declares: + +```yaml +policy_server_config: + image_resolution: [180, 320] + n_external_cameras: 2 + needs_wrist_camera: true + needs_stereo_camera: false + needs_session_id: true + action_space: joint_position +``` + +Without this configuration, the WebSocket returns `Robot policy not available` +and closes. + +## DreamZero Quick Start + +Start the bundled DreamZero deployment: + +```bash +vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 \ + --served-model-name dreamzero-droid \ + --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \ + --enforce-eager --disable-log-stats +``` + +Install the optional client dependencies, download the sample camera inputs, +and run the client: + +```bash +pip install openpi-client websockets opencv-python + +hf download YangshenDeng/vllm-omni-dreamzero-assets \ + --repo-type dataset \ + --local-dir outputs/dreamzero/assets + +python examples/online_serving/dreamzero/openpi_client.py \ + --host 127.0.0.1 \ + --port 8000 \ + --video-dir outputs/dreamzero/assets +``` + +## Protocol + +The connection is request-response after an initial server handshake: + +```text +connect + <- msgpack(policy_server_config) + +infer + -> msgpack({"endpoint": "infer", "session_id": "...", ...observation}) + <- msgpack(ndarray | dict[str, ndarray]) + +reset + -> msgpack({"endpoint": "reset"}) + <- msgpack({"status": "reset successful"}) +``` + +If `endpoint` is omitted, the server treats the message as `infer`. Each +inference message produces one action response after the engine request +completes; action tokens or intermediate tensors are not streamed. + +NumPy arrays use the marker format implemented by `openpi-client`. The server +also accepts the legacy vLLM NumPy marker representation on input. JSON text +frames are not observation messages. + +## Sessions and Reset + +- The API layer's current-session and first-call counters are scoped to each + WebSocket connection. +- `session_id` identifies model-side state across observations on that + connection. If omitted, it defaults to `default`. +- The first inference for a session is sent to the model with `reset=true`. +- Changing `session_id`, or sending the `reset` command, causes the next + inference to start with `reset=true`. +- The policy pipeline owns observation transforms and persistent model state, + normally keyed by `session_id`; the API layer forwards the raw observation + dictionary. + +## Limits and Errors + +- Maximum inbound payload size is 64 MiB. +- The server closes an idle connection after 30 seconds. +- Invalid binary input returns a MessagePack + `{"type":"error","message":"Invalid request payload"}` response. +- Inference failures return a generic `Internal inference error` without + exposing an internal traceback. +- Unsupported NumPy object, structured, and complex dtypes are rejected. + +Observation keys, camera layout, state tensors, and action shapes are defined +by the loaded policy rather than by this transport. Read the handshake before +constructing observations. See the [DreamZero example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) +for a complete OpenPI client and DROID simulation loop. diff --git a/docs/serving/realtime_api.md b/docs/serving/realtime_api.md new file mode 100644 index 000000000..58fe8b7c3 --- /dev/null +++ b/docs/serving/realtime_api.md @@ -0,0 +1,82 @@ +# Realtime Audio WebSocket API + +Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and +receive incremental transcription and synthesized response audio. This is a +turn-based streaming API: it starts one generation over an audio stream and +finishes after the client closes that input stream. + +For a model that listens while it is already speaking, use the experimental +[Full-Duplex API](full_duplex_api.md) instead. + +## Quick Start + +Start a compatible model. The currently documented example uses Qwen3-Omni: + +```bash +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 +``` + +Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file: + +```bash +python examples/online_serving/qwen3_omni/openai_realtime_client.py \ + --url ws://localhost:8091/v1/realtime \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --input-wav input_16k_mono.wav \ + --output-wav response.wav \ + --output-text response.txt +``` + +The client dependency is `websockets`. + +## Protocol + +Messages in both directions are JSON text frames. Audio carried inside an +event is base64-encoded raw PCM16. + +| Direction | Event | Purpose | +|-----------|-------|---------| +| Server to client | `session.created` | Confirms that the WebSocket connection is ready | +| Client to server | `session.update` | Selects and validates the served model | +| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream | +| Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio | +| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input | +| Server to client | `transcription.delta` | Carries incremental response text | +| Server to client | `transcription.done` | Carries final text and token usage | +| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio | +| Server to client | `response.audio.done` | Marks the end of response audio | +| Server to client | `error` | Reports an invalid event, model, or audio payload | + +A minimal client sends events in this order: + +```json +{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"} +{"type":"input_audio_buffer.commit","final":false} +{"type":"input_audio_buffer.append","audio":""} +{"type":"input_audio_buffer.commit","final":true} +``` + +The initial non-final commit intentionally starts generation before all audio +has arrived. Continue sending `input_audio_buffer.append` events while the +engine is consuming the stream. + +## Audio Handling + +- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example. +- `response.audio.delta.audio` contains base64-encoded PCM16 bytes. +- Read `sample_rate_hz` from each audio event instead of assuming an output + rate. Qwen3-Omni output is typically 24 kHz. +- Concatenate audio deltas in receive order to construct the output waveform. +- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so + the server can terminate the streaming request. + +## Availability and Limitations + +The path is registered on the unified API server, but it is usable only when +the loaded pipeline implements realtime audio input and produces compatible +audio output. Unsupported deployments return an `error` event. The endpoint +does not provide duplex session resume, playback acknowledgement, overlap +policy, or barge-in controls. + +See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) +for concurrency options, chunk pacing, and per-delta audio debugging. diff --git a/docs/serving/standalone_servers.md b/docs/serving/standalone_servers.md new file mode 100644 index 000000000..460936ad0 --- /dev/null +++ b/docs/serving/standalone_servers.md @@ -0,0 +1,109 @@ +# Standalone Experimental Servers + +Most vLLM-Omni endpoints run on the unified server started with +`vllm serve --omni`. The experimental full-duplex package also contains +two separate user-facing server processes. Their routes are not added to the +unified API server. + +| Server | Transport | Role | +|--------|-----------|------| +| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative | +| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend | + +## PersonaPlex Compatibility Server + +Use this server when a client requires the official Moshi/PersonaPlex wire +protocol or the bundled browser UI. For the engine-owned unified duplex path, +use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead. + +Start it with: + +```bash +HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \ + vllm_omni.experimental.fullduplex.personaplex.serving.server \ + --port 8124 +``` + +| Route | Purpose | +|-------|---------| +| `GET /` | Official PersonaPlex browser client, when its assets are available | +| `GET /health` | Readiness check | +| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio | +| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol | + +### `/api/chat` Protocol + +The first byte of every binary frame is a message tag: + +| Direction | Tag | Payload | +|-----------|-----|---------| +| Server to client | `0x00` | Ready handshake | +| Client to server | `0x01` | Opus microphone audio at 24 kHz | +| Server to client | `0x01` | Opus response audio | +| Server to client | `0x02` | UTF-8 inner-monologue text | + +The optional `text_prompt` and `voice_prompt` query parameters select a persona +and voice. The default server admits one conversation; use `--batch-size N` to +configure multiple lockstep slots. Connections beyond capacity close with +WebSocket code 1013. + +### `/v1/audio/duplex` Protocol + +Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame, +then exchange binary float32 PCM frames. The server returns generated float32 +audio as binary frames and text as `{"type":"text","text":"..."}`. Send +`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish. + +See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex) +for the browser and headless clients. + +## JoyVL Interaction Server + +JoyVL is an orchestration layer, not a model-serving engine. It calls a +separate OpenAI-compatible backend and adds per-session frame history, memory, +persona policy, and optional delegation. + +Start the model backend and orchestrator separately: + +```bash +vllm serve jdopensource/JoyAI-VL-Interaction-Preview \ + --served-model-name JoyAI-VL-Interaction-Preview \ + --port 8061 \ + --max-model-len 131072 \ + --enable-prefix-caching \ + --limit-mm-per-prompt '{"image":256,"video":1}' + +python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \ + --port 8070 \ + --main-backend-url http://127.0.0.1:8061/v1 \ + --main-model JoyAI-VL-Interaction-Preview +``` + +| Method and route | Purpose | +|------------------|---------| +| `GET /health` | Readiness check | +| `GET /v1/models` | Reports the configured interaction model | +| `POST /v1/chat/completions` | Processes one frame or interaction tick | +| `POST /reset` | Resets a session | +| `POST /v1/streaming/reset` | Alias for `/reset` | +| `POST /v1/streaming/persona` | Changes the session persona | + +Send one multimodal Chat Completions request per video frame, normally around +one frame per second, and identify the session with `x-session-id`. Responses +include an `interaction` block whose action is `silence`, `response`, or +`delegate`. + +```bash +curl http://127.0.0.1:8070/v1/chat/completions \ + -H 'content-type: application/json' \ + -H 'x-session-id: demo' \ + -d '{ + "messages": [{"role": "user", "content": [ + {"type": "text", "text": "Alert me if a fire appears"}, + {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}} + ]}] + }' +``` + +See the [JoyVL recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/JD/JoyAI-VL-Interaction.md) +for memory, personas, delegation backends, and the browser UI. diff --git a/docs/serving/streaming_video_output_api.md b/docs/serving/streaming_video_output_api.md new file mode 100644 index 000000000..21e78b6a2 --- /dev/null +++ b/docs/serving/streaming_video_output_api.md @@ -0,0 +1,92 @@ +# Streaming Video Output WebSocket API + +Use `WS /v1/realtime/video` to receive fragmented MP4 chunks while a compatible +diffusion video model is generating. This endpoint streams generated video +output; it is unrelated to [`WS /v1/video/chat/stream`](video_stream_api.md), +which streams live video input into a conversational model. + +## Quick Start + +Start a model that supports step execution and enable diffusion streaming +output: + +```bash +vllm serve BestWishYsh/Helios-Distilled --omni \ + --diffusion-streaming-output \ + --port 8000 +``` + +Install the client dependency and run the provided client: + +```bash +pip install websockets + +python examples/online_serving/streaming_video_generation/streaming_video_client.py \ + --host 127.0.0.1 \ + --port 8000 \ + --model BestWishYsh/Helios-Distilled \ + --prompt "A serene lakeside sunrise with mist over the water" \ + --output output.mp4 +``` + +## Protocol + +Control events are JSON text frames. Each `video.chunk_metadata` event is +immediately followed by one binary WebSocket frame containing fragmented MP4 +bytes. + +| Direction | Event or frame | Purpose | +|-----------|----------------|---------| +| Client to server | `session.start` | Starts generation with a prompt and video parameters | +| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration | +| Server to client | `video.chunk_metadata` | Describes the next binary media frame | +| Server to client | Binary frame | Carries one fragmented MP4 chunk | +| Client to server | `session.interaction` | Queues an optional prompt update during generation | +| Server to client | `session.interaction.queued` | Confirms that an interaction was queued | +| Client to server | `session.ping` | Refreshes the stall clock | +| Server to client | `session.pong` | Acknowledges a ping | +| Client to server | `session.stop` | Cancels the active generation | +| Server to client | `session.done` | Reports completion or cancellation and the chunk count | +| Server to client | `error` | Reports validation, generation, or timeout failures | + +Start a minimal session with: + +```json +{ + "type": "session.start", + "model": "BestWishYsh/Helios-Distilled", + "prompt": "A serene lakeside sunrise", + "format": "m4s", + "width": 640, + "height": 384, + "fps": 16, + "num_frames": 99 +} +``` + +Only `m4s` output is currently accepted. Concatenate the binary frames in +receive order, then remux the fragmented stream if the target player requires +a progressive MP4 file. + +## Mid-Generation Interaction + +Models that implement interaction updates can change the active prompt: + +```json +{ + "type": "session.interaction", + "interaction": { + "event_id": "sunlight-update", + "event": {"prompt": "Sunlight breaks through the mist"}, + "transition_chunks": 2 + } +} +``` + +This feature is pipeline-dependent. An enabled endpoint does not imply that +the loaded model supports midway prompt updates. + +The default start timeout is 10 seconds and the server reports a stall after +about 60 seconds without generation progress or a `session.ping`. See the +[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) +for prompt-update scheduling and browser playback.