diff --git a/realtime_video_prompt_interaction_protocol.md b/realtime_video_prompt_interaction_protocol.md deleted file mode 100755 index 534d76a51..000000000 --- a/realtime_video_prompt_interaction_protocol.md +++ /dev/null @@ -1,138 +0,0 @@ -# Realtime Video Prompt Interaction Protocol - -This document specifies the prompt-update subset of the realtime video -interaction protocol exposed by: - -```text -WebSocket /v1/realtime/video -``` - -A WebSocket connection contains exactly one video-generation request. Clients -therefore do not send a request or session ID with an interaction. The server -assigns a `request_id` in `video.start` and includes it in subsequent -server-to-client messages. - -## Interaction Request - -After `video.start`, a client may submit a prompt update: - -```json -{ - "type": "session.interaction", - "interaction": { - "event_id": "ui-update-42", - "event": { - "prompt": "A snowy street illuminated by warm lanterns" - }, - "transition_chunks": 3 - } -} -``` - -`interaction` has the following fields: - -- `event_id` (optional string): session-unique idempotency key. The - server preserves a client-provided value or generates a UUID when omitted. -- `event.prompt` (required string): non-empty target prompt. - over which the prompt reaches its target. -- `transition_chunks` (optional non-negative integer): model-generation chunks - over which the prompt reaches its target. If omitted, the model's default transition is used. - -Note: Although `["interaction"]["event"]` only contains `prompt`, still consider using `OmniTextPrompt` for this part (with total=false) so that other fields will be supported in the future. Currently, `vllm_omni/diffusion/worker/diffusion_model_runner.py` already takes care of enforcing subset format, routing to meaningful input handing routines, and raising `NotImplementedError`. Only need to adapt this design to the input data structure above. - -### Transition Semantics - -Prompt transitions are target updates and begin at the next eligible model -generation boundary after the interaction is processed. - -- `transition_chunks = 0` is a hard update: the first affected generation - chunk uses the target prompt. -- `transition_chunks = N`, where `N > 0`, reaches the target in the `N`th - affected generation chunk. For example, a three-chunk linear transition uses - progress values `1/3`, `2/3`, and `1`. - -### Idempotency and Replacement - -Within a session: - -- repeating an `event_id` with the same payload is a retry and must not - apply the update twice; -- repeating an `event_id` with a different payload is rejected with - `event_id_conflict`; -- a newer prompt target supersedes any older pending prompt target; -- when a new target interrupts an active transition, its transition starts - from the prompt state resolved at its effective boundary. - -## Queue Acknowledgement - -The server quickly acknowledges admission to the interaction queue: - -```json -{ - "type": "session.interaction.queued", - "request_id": "video_stream-...", - "event_id": "ui-update-42" -} -``` - -`queued` confirms syntax validation, idempotency handling, and queue admission. -It does not claim that the model has scheduled or applied the update. A later -failure is reported as a correlated error: - -```json -{ - "type": "error", - "request_id": "video_stream-...", - "event_id": "ui-update-42", - "code": "interaction_failed", - "message": "The active model does not support prompt updates" -} -``` - -## Video Chunk Metadata - -Each emitted binary payload is preceded by a JSON metadata message. The -metadata message and its binary payload form one ordered pair and must not be -interleaved with another outbound message. - -```json -{ - "type": "video.chunk_metadata", - "request_id": "video_stream-...", - "kind": "media", - "transport_chunk_index": 7, - "generation_chunk_index": 6, - "num_frames": 9, - "byte_length": 184320, - "started_event_ids": ["ui-update-42"], - "active_event_ids": ["ui-update-42"], - "completed_event_ids": [] -} -``` - -- `transport_chunk_index` identifies the binary WebSocket payload. -- `generation_chunk_index` identifies the model-generation chunk represented - by the payload. -- `started_event_ids` lists interactions that first affect this - generation chunk. -- `active_event_ids` lists interactions contributing to this generation - chunk, including an ongoing transition. -- `completed_event_ids` lists interactions that reach their target in - this generation chunk. - -Encoder trailer bytes are emitted as: - -```json -{ - "type": "video.chunk_metadata", - "request_id": "video_stream-...", - "kind": "trailer", - "transport_chunk_index": 8, - "generation_chunk_index": null, - "num_frames": 0, - "byte_length": 824, - "started_event_ids": [], - "active_event_ids": [], - "completed_event_ids": [] -} -```