[Core][Diffusion] Add a typed pre-D2H video media contract (#6615)
Signed-off-by: NancyFyong <NancyFyong@users.noreply.github.com> Signed-off-by: Samit <285365963@qq.com> Co-authored-by: NancyFyong <NancyFyong@users.noreply.github.com> Co-authored-by: Samit <285365963@qq.com>
This commit is contained in:
31 files changed
+3054
-60
No files matched your search
@@ -0,0 +1,72 @@
|
||||
# Device-side video postprocessing
|
||||
|
||||
vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point
|
||||
`[B, C, T, H, W]` tensors to contiguous `uint8` `[B, T, H, W, C]` frames before
|
||||
the worker-to-engine device-to-host transfer. This reduces the transported video
|
||||
payload by four times when the original IPC path widens bfloat16 to float32.
|
||||
|
||||
The optimization is disabled by default. Enable it with a structured diffusion
|
||||
configuration:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
engine = Omni(
|
||||
model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
|
||||
video_output_transport={"enable_device_postprocess": True},
|
||||
)
|
||||
```
|
||||
|
||||
For the server CLI, pass the same object as JSON:
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
|
||||
--video-output-transport '{"enable_device_postprocess": true}'
|
||||
```
|
||||
|
||||
For a deploy configuration, place it on the diffusion stage:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
video_output_transport:
|
||||
enable_device_postprocess: true
|
||||
```
|
||||
|
||||
## Runtime contract
|
||||
|
||||
WAN2.2 emits decoded video through the typed `DiffusionOutput.media` field. The
|
||||
model runner validates and splits the batch into request-local tensors before
|
||||
preparing each video for transport. The worker then performs the D2H copy and
|
||||
shared-memory IPC using the prepared representation. The engine uses the generic
|
||||
media finalizer instead of a model-specific postprocessor.
|
||||
|
||||
The runtime leaves the video in normalized floating-point form when:
|
||||
|
||||
- `enable_device_postprocess` is false;
|
||||
- the requested output type is not `np`; or
|
||||
- frame interpolation still requires the floating-point tensor.
|
||||
|
||||
These are policy fallbacks, not malformed-contract errors. If the device cannot
|
||||
allocate the temporary conversion buffer, the runner logs a warning and prepares
|
||||
the request-local float representation instead. Invalid tensor layout, encoding,
|
||||
value range, lifecycle state, and non-memory runtime failures still fail before
|
||||
IPC.
|
||||
|
||||
## Precision
|
||||
|
||||
Device preparation converts to float32 before denormalization and quantization.
|
||||
For bfloat16 WAN output this is more precise than the historical host path,
|
||||
which denormalizes in bfloat16. Relative to that path, values may differ by at
|
||||
most one level out of 255. Float32 inputs remain byte-identical after
|
||||
quantization.
|
||||
|
||||
## Scope
|
||||
|
||||
WAN2.2 is the reference migration for the first contract version. Other video
|
||||
pipelines continue using their existing output paths until they can declare the
|
||||
same typed media contract without bypassing model-specific float consumers such
|
||||
as safety checks or audio/video packaging.
|
||||
|
||||
See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
|
||||
contract, lifecycle, batching rules, and migration plan.
|
||||
Reference in new issue
Block a user