Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
This commit is contained in:
Zeyu Huang | 黃澤宇 authored and GitHub committed
1 parent
39663d5287
commit
b8e7b21188
1 file changed
-138
@@ -1,138 +0,0 @@
|
||||
# Realtime Video Prompt Interaction Protocol
|
||||
|
||||
This document specifies the prompt-update subset of the realtime video
|
||||
interaction protocol exposed by:
|
||||
|
||||
```text
|
||||
WebSocket /v1/realtime/video
|
||||
```
|
||||
|
||||
A WebSocket connection contains exactly one video-generation request. Clients
|
||||
therefore do not send a request or session ID with an interaction. The server
|
||||
assigns a `request_id` in `video.start` and includes it in subsequent
|
||||
server-to-client messages.
|
||||
|
||||
## Interaction Request
|
||||
|
||||
After `video.start`, a client may submit a prompt update:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "session.interaction",
|
||||
"interaction": {
|
||||
"event_id": "ui-update-42",
|
||||
"event": {
|
||||
"prompt": "A snowy street illuminated by warm lanterns"
|
||||
},
|
||||
"transition_chunks": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`interaction` has the following fields:
|
||||
|
||||
- `event_id` (optional string): session-unique idempotency key. The
|
||||
server preserves a client-provided value or generates a UUID when omitted.
|
||||
- `event.prompt` (required string): non-empty target prompt.
|
||||
over which the prompt reaches its target.
|
||||
- `transition_chunks` (optional non-negative integer): model-generation chunks
|
||||
over which the prompt reaches its target. If omitted, the model's default transition is used.
|
||||
|
||||
Note: Although `["interaction"]["event"]` only contains `prompt`, still consider using `OmniTextPrompt` for this part (with total=false) so that other fields will be supported in the future. Currently, `vllm_omni/diffusion/worker/diffusion_model_runner.py` already takes care of enforcing subset format, routing to meaningful input handing routines, and raising `NotImplementedError`. Only need to adapt this design to the input data structure above.
|
||||
|
||||
### Transition Semantics
|
||||
|
||||
Prompt transitions are target updates and begin at the next eligible model
|
||||
generation boundary after the interaction is processed.
|
||||
|
||||
- `transition_chunks = 0` is a hard update: the first affected generation
|
||||
chunk uses the target prompt.
|
||||
- `transition_chunks = N`, where `N > 0`, reaches the target in the `N`th
|
||||
affected generation chunk. For example, a three-chunk linear transition uses
|
||||
progress values `1/3`, `2/3`, and `1`.
|
||||
|
||||
### Idempotency and Replacement
|
||||
|
||||
Within a session:
|
||||
|
||||
- repeating an `event_id` with the same payload is a retry and must not
|
||||
apply the update twice;
|
||||
- repeating an `event_id` with a different payload is rejected with
|
||||
`event_id_conflict`;
|
||||
- a newer prompt target supersedes any older pending prompt target;
|
||||
- when a new target interrupts an active transition, its transition starts
|
||||
from the prompt state resolved at its effective boundary.
|
||||
|
||||
## Queue Acknowledgement
|
||||
|
||||
The server quickly acknowledges admission to the interaction queue:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "session.interaction.queued",
|
||||
"request_id": "video_stream-...",
|
||||
"event_id": "ui-update-42"
|
||||
}
|
||||
```
|
||||
|
||||
`queued` confirms syntax validation, idempotency handling, and queue admission.
|
||||
It does not claim that the model has scheduled or applied the update. A later
|
||||
failure is reported as a correlated error:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"request_id": "video_stream-...",
|
||||
"event_id": "ui-update-42",
|
||||
"code": "interaction_failed",
|
||||
"message": "The active model does not support prompt updates"
|
||||
}
|
||||
```
|
||||
|
||||
## Video Chunk Metadata
|
||||
|
||||
Each emitted binary payload is preceded by a JSON metadata message. The
|
||||
metadata message and its binary payload form one ordered pair and must not be
|
||||
interleaved with another outbound message.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "video.chunk_metadata",
|
||||
"request_id": "video_stream-...",
|
||||
"kind": "media",
|
||||
"transport_chunk_index": 7,
|
||||
"generation_chunk_index": 6,
|
||||
"num_frames": 9,
|
||||
"byte_length": 184320,
|
||||
"started_event_ids": ["ui-update-42"],
|
||||
"active_event_ids": ["ui-update-42"],
|
||||
"completed_event_ids": []
|
||||
}
|
||||
```
|
||||
|
||||
- `transport_chunk_index` identifies the binary WebSocket payload.
|
||||
- `generation_chunk_index` identifies the model-generation chunk represented
|
||||
by the payload.
|
||||
- `started_event_ids` lists interactions that first affect this
|
||||
generation chunk.
|
||||
- `active_event_ids` lists interactions contributing to this generation
|
||||
chunk, including an ongoing transition.
|
||||
- `completed_event_ids` lists interactions that reach their target in
|
||||
this generation chunk.
|
||||
|
||||
Encoder trailer bytes are emitted as:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "video.chunk_metadata",
|
||||
"request_id": "video_stream-...",
|
||||
"kind": "trailer",
|
||||
"transport_chunk_index": 8,
|
||||
"generation_chunk_index": null,
|
||||
"num_frames": 0,
|
||||
"byte_length": 824,
|
||||
"started_event_ids": [],
|
||||
"active_event_ids": [],
|
||||
"completed_event_ids": []
|
||||
}
|
||||
```
|
||||
Reference in new issue
Block a user