anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: document realtime and specialized API servers

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-13 21:46:24 +08:00
1 parent 5c6ed9b4a9
commit 8327f26216
7 files changed
+552 -23
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+14 -7
View File
@@ -7,13 +7,20 @@ nav:
- Serving:
- API Server:
- Overview: serving/README.md
- Chat Completions API: serving/chat_completions_api.md
- Image Generation: serving/image_generation_api.md
- Image Edit: serving/image_edit_api.md
- Text to Speech: serving/speech_api.md
- Videos: serving/videos_api.md
- Audio Generation: serving/audio_generate_api.md
- Streaming Video Input: serving/video_stream_api.md
- Core APIs:
- Chat Completions: serving/chat_completions_api.md
- Text to Speech: serving/speech_api.md
- Audio Generation: serving/audio_generate_api.md
- Image Generation: serving/image_generation_api.md
- Image Edit: serving/image_edit_api.md
- Videos: serving/videos_api.md
- Streaming and Realtime:
- Realtime Audio: serving/realtime_api.md
- Full Duplex: serving/full_duplex_api.md
- Streaming Video Input: serving/video_stream_api.md
- Streaming Video Output: serving/streaming_video_output_api.md
- OpenPI Robot Policy: serving/openpi_api.md
- Standalone Experimental Servers: serving/standalone_servers.md
- Examples:
- examples/README.md
- Offline Inference:
docs/serving/README.md
+30 -16
View File
@@ -1,9 +1,9 @@
# API Server
vLLM-Omni exposes OpenAI-compatible HTTP APIs plus vLLM-Omni extensions for
image, audio, video, and realtime workloads. Use this page to choose an
endpoint; follow the linked reference page for model-specific fields and
response details.
image, audio, video, realtime, and robot-policy workloads. Use this page to
choose an endpoint; follow the linked reference page for model-specific fields,
wire protocols, and response details.
Each server instance hosts one model. An endpoint is usable only when the
loaded model supports that task.
@@ -36,7 +36,7 @@ Use `http://localhost:8000/v1` as the `base_url` for an OpenAI SDK client. If
the server was started with `--api-key`, include an `Authorization` header with
`Bearer <api-key>` in requests.
## Choose an Endpoint
## Choose a Core HTTP Endpoint
Prefer a task-specific endpoint when one matches your workload. Use Chat
Completions for conversational or heterogeneous omni pipelines, rather than as
@@ -51,7 +51,6 @@ the default wrapper for every generation task.
| Image editing | `POST /v1/images/edits` | Multipart form | Base64 JSON or SSE | [Image Edit](image_edit_api.md) |
| Video generation (recommended) | `POST /v1/videos` | Multipart form | Asynchronous job JSON | [Videos](videos_api.md) |
| Video generation (blocking) | `POST /v1/videos/sync` | Multipart form | Video bytes | [Videos](videos_api.md#synchronous-response) |
| Live video understanding | `WS /v1/video/chat/stream` | WebSocket events | Streaming text/audio events | [Streaming Video Input](video_stream_api.md) |
`POST /v1/videos/sync` is intended for tests and simple scripts. Use the
asynchronous job API for production requests so clients can poll, download,
@@ -136,6 +135,26 @@ media inputs, generation controls, and output capabilities vary by model.
curl -L "$VLLM_OMNI_BASE_URL/v1/videos/<video_id>/content" --output video.mp4
```
## Choose a Streaming or Realtime Endpoint
These WebSocket APIs have different event schemas and cannot be used
interchangeably.
| Workload | Endpoint | Interaction model | Details |
|----------|----------|-------------------|---------|
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |
All six routes are model- or configuration-dependent. In particular,
`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the
deployment explicitly enables duplex sessions. Clients should also verify the
duplex capability payload because the query-parameter form falls back to the
ordinary realtime handler when duplex is unavailable.
## Related Endpoints
| Purpose | Endpoints | Reference |
@@ -144,21 +163,16 @@ media inputs, generation controls, and output capabilities vary by model.
| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
| TTS voice management | `GET/POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}` | [Voices](speech_api.md#voices-endpoint) |
| Streaming text-to-speech | `WS /v1/audio/speech/stream` | [Streaming speech](speech_api.md#streaming-text-input-websocket) |
| Video job lifecycle | `GET /v1/videos`, `GET/DELETE /v1/videos/{video_id}`, `GET /v1/videos/{video_id}/content` | [Video endpoints](videos_api.md#endpoints) |
| Release and restore stage memory | `POST /v1/omni/sleep`, `POST /v1/omni/wakeup` | [Sleep Mode](../features/sleep_mode.md) |
??? info "Specialized and experimental WebSocket endpoints"
## Standalone Experimental Servers
These routes are enabled only for specific pipelines. Start with the
associated example instead of treating them as general-purpose APIs.
| Endpoint | Use case | Example |
|----------|----------|---------|
| `WS /v1/realtime` | OpenAI-style realtime audio interaction | [Qwen3-Omni realtime client](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni) |
| `WS /v1/realtime/video` | Stream generated video chunks | [Streaming video generation](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation) |
| `WS /v1/duplex` or `WS /v1/realtime?duplex=1` | Experimental full-duplex speech interaction | [Full-duplex examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo) |
| `WS /v1/realtime/robot/openpi` | OpenPI-compatible robot policy inference | [DreamZero](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero) |
PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL
provides a stateful interaction orchestrator in front of another model server.
These are separate processes with their own routes; they are not additional
paths on every `vllm serve --omni` instance. See [Standalone Experimental
Servers](standalone_servers.md) before deploying either one.
## Compatibility Notes
docs/serving/full_duplex_api.md
+116
View File
@@ -0,0 +1,116 @@
# Full-Duplex WebSocket API
vLLM-Omni provides an experimental full-duplex runtime for models that can
continue receiving speech while producing speech. It adds persistent session
state, model-specific turn policy, overlap handling, playback acknowledgement,
and optional session resume.
Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
## Choose an Endpoint
| Endpoint | Protocol | Recommended use |
|----------|----------|-----------------|
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
Both endpoints use the same duplex engine and require the same model-side
adapter. Prefer the Realtime projection unless the native lifecycle events are
specifically required.
## Enable Full Duplex
The route becomes usable only when the deployment configuration explicitly
sets:
```yaml
session_mode: duplex
```
The selected model pipeline must also provide a duplex serving adapter.
Model-native deployments configure the corresponding engine runtime extension
and control plane as part of their registered pipeline.
!!! warning
`WS /v1/duplex` fails with `Duplex API is not available` when duplex is not
enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary
turn-based realtime handler when no duplex handler exists. Confirm that
`session.created.session.capabilities` is present before treating the
connection as full duplex.
The current unified-runtime integrations are:
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`;
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex.
JoyVL is a separate HTTP interaction orchestrator and does not use these
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
## MiniCPM-o Quick Start
Start the duplex deployment:
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
--trust-remote-code \
--port 8099
```
Stream a mono, PCM16, 16 kHz WAV file with the provided client:
```bash
python examples/online_serving/minicpmo/realtime_duplex_demo.py \
--url 'ws://localhost:8099/v1/realtime?duplex=1' \
--model openbmb/MiniCPM-o-4_5 \
--input-wav input_16k_mono.wav \
--ref-audio reference_voice.wav \
--output-dir /tmp/minicpmo-duplex
```
## Realtime Event Lifecycle
A typical `/v1/realtime?duplex=1` session follows this lifecycle:
1. Send `session.update` with the model, modalities, audio formats, and session
options.
2. Wait for `session.created`; inspect `session.capabilities` instead of
assuming every model supports the same controls.
3. Send `input_audio_buffer.append` events while microphone audio arrives.
4. Send `input_audio_buffer.commit` at a user-turn boundary when required by
the model policy.
5. Consume `response.created`, transcript deltas, `response.audio.delta`, and
`response.done` or `response.listen` events.
6. Send `playback.ack` after audio has been played when the session advertises
playback acknowledgement support.
7. Send `session.close` and wait for `session.closed`.
Unlike the turn-based realtime endpoint, input may continue while a response
is active. The server can emit `overlap.decision` to describe whether input was
deferred, treated as a short acknowledgement, or used to interrupt output.
## Capabilities and Model Differences
The `session.created` payload includes capability fields such as
`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`,
`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the
runtime contract.
For example, PersonaPlex supports native overlapping speech but currently
advertises `supports_barge_in=false`; destructive output interruption and
model-state rewind have not been validated for that integration. Capacity and
session-resume behavior also depend on the selected deployment configuration.
## Native Protocol
`WS /v1/duplex` exposes lower-level lifecycle names including
`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`,
and `session.close`. It returns native session, input, response, overlap, and
error events. This protocol is experimental and may evolve with the runtime;
applications should use the provided Realtime client where possible.
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md)
for model-specific validation and architecture details.
docs/serving/openpi_api.md
+109
View File
@@ -0,0 +1,109 @@
# OpenPI Robot Policy WebSocket API
Use `WS /v1/realtime/robot/openpi` for low-latency robot-policy inference with
OpenPI-compatible clients. A persistent WebSocket carries observations to the
loaded policy model and returns action arrays.
Despite the `/realtime` prefix, this endpoint does not use the OpenAI Realtime
event schema. It uses binary MessagePack frames with NumPy extensions.
## Availability
The endpoint is available only when the loaded diffusion policy configuration
contains `policy_server_config`. The server sends that model-specific
configuration to the client as the first binary frame.
For example, DreamZero declares:
```yaml
policy_server_config:
image_resolution: [180, 320]
n_external_cameras: 2
needs_wrist_camera: true
needs_stereo_camera: false
needs_session_id: true
action_space: joint_position
```
Without this configuration, the WebSocket returns `Robot policy not available`
and closes.
## DreamZero Quick Start
Start the bundled DreamZero deployment:
```bash
vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 \
--served-model-name dreamzero-droid \
--deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml \
--enforce-eager --disable-log-stats
```
Install the optional client dependencies, download the sample camera inputs,
and run the client:
```bash
pip install openpi-client websockets opencv-python
hf download YangshenDeng/vllm-omni-dreamzero-assets \
--repo-type dataset \
--local-dir outputs/dreamzero/assets
python examples/online_serving/dreamzero/openpi_client.py \
--host 127.0.0.1 \
--port 8000 \
--video-dir outputs/dreamzero/assets
```
## Protocol
The connection is request-response after an initial server handshake:
```text
connect
<- msgpack(policy_server_config)
infer
-> msgpack({"endpoint": "infer", "session_id": "...", ...observation})
<- msgpack(ndarray | dict[str, ndarray])
reset
-> msgpack({"endpoint": "reset"})
<- msgpack({"status": "reset successful"})
```
If `endpoint` is omitted, the server treats the message as `infer`. Each
inference message produces one action response after the engine request
completes; action tokens or intermediate tensors are not streamed.
NumPy arrays use the marker format implemented by `openpi-client`. The server
also accepts the legacy vLLM NumPy marker representation on input. JSON text
frames are not observation messages.
## Sessions and Reset
- The API layer's current-session and first-call counters are scoped to each
WebSocket connection.
- `session_id` identifies model-side state across observations on that
connection. If omitted, it defaults to `default`.
- The first inference for a session is sent to the model with `reset=true`.
- Changing `session_id`, or sending the `reset` command, causes the next
inference to start with `reset=true`.
- The policy pipeline owns observation transforms and persistent model state,
normally keyed by `session_id`; the API layer forwards the raw observation
dictionary.
## Limits and Errors
- Maximum inbound payload size is 64 MiB.
- The server closes an idle connection after 30 seconds.
- Invalid binary input returns a MessagePack
`{"type":"error","message":"Invalid request payload"}` response.
- Inference failures return a generic `Internal inference error` without
exposing an internal traceback.
- Unsupported NumPy object, structured, and complex dtypes are rejected.
Observation keys, camera layout, state tensors, and action shapes are defined
by the loaded policy rather than by this transport. Read the handshake before
constructing observations. See the [DreamZero example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/dreamzero)
for a complete OpenPI client and DROID simulation loop.
docs/serving/realtime_api.md
+82
View File
@@ -0,0 +1,82 @@
# Realtime Audio WebSocket API
Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and
receive incremental transcription and synthesized response audio. This is a
turn-based streaming API: it starts one generation over an audio stream and
finishes after the client closes that input stream.
For a model that listens while it is already speaking, use the experimental
[Full-Duplex API](full_duplex_api.md) instead.
## Quick Start
Start a compatible model. The currently documented example uses Qwen3-Omni:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:
```bash
python examples/online_serving/qwen3_omni/openai_realtime_client.py \
--url ws://localhost:8091/v1/realtime \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--input-wav input_16k_mono.wav \
--output-wav response.wav \
--output-text response.txt
```
The client dependency is `websockets`.
## Protocol
Messages in both directions are JSON text frames. Audio carried inside an
event is base64-encoded raw PCM16.
| Direction | Event | Purpose |
|-----------|-------|---------|
| Server to client | `session.created` | Confirms that the WebSocket connection is ready |
| Client to server | `session.update` | Selects and validates the served model |
| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
| Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio |
| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
| Server to client | `transcription.delta` | Carries incremental response text |
| Server to client | `transcription.done` | Carries final text and token usage |
| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio |
| Server to client | `response.audio.done` | Marks the end of response audio |
| Server to client | `error` | Reports an invalid event, model, or audio payload |
A minimal client sends events in this order:
```json
{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
{"type":"input_audio_buffer.commit","final":false}
{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
{"type":"input_audio_buffer.commit","final":true}
```
The initial non-final commit intentionally starts generation before all audio
has arrived. Continue sending `input_audio_buffer.append` events while the
engine is consuming the stream.
## Audio Handling
- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
- `response.audio.delta.audio` contains base64-encoded PCM16 bytes.
- Read `sample_rate_hz` from each audio event instead of assuming an output
rate. Qwen3-Omni output is typically 24 kHz.
- Concatenate audio deltas in receive order to construct the output waveform.
- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so
the server can terminate the streaming request.
## Availability and Limitations
The path is registered on the unified API server, but it is usable only when
the loaded pipeline implements realtime audio input and produces compatible
audio output. Unsupported deployments return an `error` event. The endpoint
does not provide duplex session resume, playback acknowledgement, overlap
policy, or barge-in controls.
See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni)
for concurrency options, chunk pacing, and per-delta audio debugging.
docs/serving/standalone_servers.md
+109
View File
@@ -0,0 +1,109 @@
# Standalone Experimental Servers
Most vLLM-Omni endpoints run on the unified server started with
`vllm serve <model> --omni`. The experimental full-duplex package also contains
two separate user-facing server processes. Their routes are not added to the
unified API server.
| Server | Transport | Role |
|--------|-----------|------|
| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative |
| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend |
## PersonaPlex Compatibility Server
Use this server when a client requires the official Moshi/PersonaPlex wire
protocol or the bundled browser UI. For the engine-owned unified duplex path,
use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead.
Start it with:
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
vllm_omni.experimental.fullduplex.personaplex.serving.server \
--port 8124
```
| Route | Purpose |
|-------|---------|
| `GET /` | Official PersonaPlex browser client, when its assets are available |
| `GET /health` | Readiness check |
| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio |
| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol |
### `/api/chat` Protocol
The first byte of every binary frame is a message tag:
| Direction | Tag | Payload |
|-----------|-----|---------|
| Server to client | `0x00` | Ready handshake |
| Client to server | `0x01` | Opus microphone audio at 24 kHz |
| Server to client | `0x01` | Opus response audio |
| Server to client | `0x02` | UTF-8 inner-monologue text |
The optional `text_prompt` and `voice_prompt` query parameters select a persona
and voice. The default server admits one conversation; use `--batch-size N` to
configure multiple lockstep slots. Connections beyond capacity close with
WebSocket code 1013.
### `/v1/audio/duplex` Protocol
Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame,
then exchange binary float32 PCM frames. The server returns generated float32
audio as binary frames and text as `{"type":"text","text":"..."}`. Send
`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish.
See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
for the browser and headless clients.
## JoyVL Interaction Server
JoyVL is an orchestration layer, not a model-serving engine. It calls a
separate OpenAI-compatible backend and adds per-session frame history, memory,
persona policy, and optional delegation.
Start the model backend and orchestrator separately:
```bash
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
--served-model-name JoyAI-VL-Interaction-Preview \
--port 8061 \
--max-model-len 131072 \
--enable-prefix-caching \
--limit-mm-per-prompt '{"image":256,"video":1}'
python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
--port 8070 \
--main-backend-url http://127.0.0.1:8061/v1 \
--main-model JoyAI-VL-Interaction-Preview
```
| Method and route | Purpose |
|------------------|---------|
| `GET /health` | Readiness check |
| `GET /v1/models` | Reports the configured interaction model |
| `POST /v1/chat/completions` | Processes one frame or interaction tick |
| `POST /reset` | Resets a session |
| `POST /v1/streaming/reset` | Alias for `/reset` |
| `POST /v1/streaming/persona` | Changes the session persona |
Send one multimodal Chat Completions request per video frame, normally around
one frame per second, and identify the session with `x-session-id`. Responses
include an `interaction` block whose action is `silence`, `response`, or
`delegate`.
```bash
curl http://127.0.0.1:8070/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Alert me if a fire appears"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
]}]
}'
```
See the [JoyVL recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/JD/JoyAI-VL-Interaction.md)
for memory, personas, delegation backends, and the browser UI.
docs/serving/streaming_video_output_api.md
+92
View File
@@ -0,0 +1,92 @@
# Streaming Video Output WebSocket API
Use `WS /v1/realtime/video` to receive fragmented MP4 chunks while a compatible
diffusion video model is generating. This endpoint streams generated video
output; it is unrelated to [`WS /v1/video/chat/stream`](video_stream_api.md),
which streams live video input into a conversational model.
## Quick Start
Start a model that supports step execution and enable diffusion streaming
output:
```bash
vllm serve BestWishYsh/Helios-Distilled --omni \
--diffusion-streaming-output \
--port 8000
```
Install the client dependency and run the provided client:
```bash
pip install websockets
python examples/online_serving/streaming_video_generation/streaming_video_client.py \
--host 127.0.0.1 \
--port 8000 \
--model BestWishYsh/Helios-Distilled \
--prompt "A serene lakeside sunrise with mist over the water" \
--output output.mp4
```
## Protocol
Control events are JSON text frames. Each `video.chunk_metadata` event is
immediately followed by one binary WebSocket frame containing fragmented MP4
bytes.
| Direction | Event or frame | Purpose |
|-----------|----------------|---------|
| Client to server | `session.start` | Starts generation with a prompt and video parameters |
| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration |
| Server to client | `video.chunk_metadata` | Describes the next binary media frame |
| Server to client | Binary frame | Carries one fragmented MP4 chunk |
| Client to server | `session.interaction` | Queues an optional prompt update during generation |
| Server to client | `session.interaction.queued` | Confirms that an interaction was queued |
| Client to server | `session.ping` | Refreshes the stall clock |
| Server to client | `session.pong` | Acknowledges a ping |
| Client to server | `session.stop` | Cancels the active generation |
| Server to client | `session.done` | Reports completion or cancellation and the chunk count |
| Server to client | `error` | Reports validation, generation, or timeout failures |
Start a minimal session with:
```json
{
"type": "session.start",
"model": "BestWishYsh/Helios-Distilled",
"prompt": "A serene lakeside sunrise",
"format": "m4s",
"width": 640,
"height": 384,
"fps": 16,
"num_frames": 99
}
```
Only `m4s` output is currently accepted. Concatenate the binary frames in
receive order, then remux the fragmented stream if the target player requires
a progressive MP4 file.
## Mid-Generation Interaction
Models that implement interaction updates can change the active prompt:
```json
{
"type": "session.interaction",
"interaction": {
"event_id": "sunlight-update",
"event": {"prompt": "Sunlight breaks through the mist"},
"transition_chunks": 2
}
}
```
This feature is pipeline-dependent. An enabled endpoint does not imply that
the loaded model supports midway prompt updates.
The default start timeout is 10 seconds and the server reports a stall after
about 60 seconds without generation progress or a `session.ping`. See the
[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation)
for prompt-update scheduling and browser playback.