[Core][Diffusion] Add a typed pre-D2H video media contract (#6615)
Signed-off-by: NancyFyong <NancyFyong@users.noreply.github.com> Signed-off-by: Samit <285365963@qq.com> Co-authored-by: NancyFyong <NancyFyong@users.noreply.github.com> Co-authored-by: Samit <285365963@qq.com>
This commit is contained in:
31 files changed
+3054
-60
No files matched your search
@@ -71,6 +71,7 @@ nav:
|
||||
- Overview and Compatibility:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- Device-side Video Postprocessing: user_guide/diffusion/device_side_video_postprocess.md
|
||||
- CPU Offloading:
|
||||
- Overview: user_guide/diffusion/cpu_offload.md
|
||||
- Model-Level: user_guide/diffusion/offloader/module_offload.md
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import ast
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
@@ -124,6 +125,7 @@ def extract_omni_serve_subparser_init():
|
||||
"make_arg_parser": lambda parser: parser, # no-op for doc
|
||||
"_ensure_vllm_platform": lambda: None, # no-op for doc
|
||||
"_nonneg_finite_float": float,
|
||||
"_json_object": json.loads,
|
||||
"_parse_stage_overrides": __import__("json").loads,
|
||||
"VLLM_SUBCMD_PARSER_EPILOG": "",
|
||||
"logger": logger,
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Device-side video postprocessing
|
||||
|
||||
vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point
|
||||
`[B, C, T, H, W]` tensors to contiguous `uint8` `[B, T, H, W, C]` frames before
|
||||
the worker-to-engine device-to-host transfer. This reduces the transported video
|
||||
payload by four times when the original IPC path widens bfloat16 to float32.
|
||||
|
||||
The optimization is disabled by default. Enable it with a structured diffusion
|
||||
configuration:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
engine = Omni(
|
||||
model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
|
||||
video_output_transport={"enable_device_postprocess": True},
|
||||
)
|
||||
```
|
||||
|
||||
For the server CLI, pass the same object as JSON:
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
|
||||
--video-output-transport '{"enable_device_postprocess": true}'
|
||||
```
|
||||
|
||||
For a deploy configuration, place it on the diffusion stage:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
video_output_transport:
|
||||
enable_device_postprocess: true
|
||||
```
|
||||
|
||||
## Runtime contract
|
||||
|
||||
WAN2.2 emits decoded video through the typed `DiffusionOutput.media` field. The
|
||||
model runner validates and splits the batch into request-local tensors before
|
||||
preparing each video for transport. The worker then performs the D2H copy and
|
||||
shared-memory IPC using the prepared representation. The engine uses the generic
|
||||
media finalizer instead of a model-specific postprocessor.
|
||||
|
||||
The runtime leaves the video in normalized floating-point form when:
|
||||
|
||||
- `enable_device_postprocess` is false;
|
||||
- the requested output type is not `np`; or
|
||||
- frame interpolation still requires the floating-point tensor.
|
||||
|
||||
These are policy fallbacks, not malformed-contract errors. If the device cannot
|
||||
allocate the temporary conversion buffer, the runner logs a warning and prepares
|
||||
the request-local float representation instead. Invalid tensor layout, encoding,
|
||||
value range, lifecycle state, and non-memory runtime failures still fail before
|
||||
IPC.
|
||||
|
||||
## Precision
|
||||
|
||||
Device preparation converts to float32 before denormalization and quantization.
|
||||
For bfloat16 WAN output this is more precise than the historical host path,
|
||||
which denormalizes in bfloat16. Relative to that path, values may differ by at
|
||||
most one level out of 255. Float32 inputs remain byte-identical after
|
||||
quantization.
|
||||
|
||||
## Scope
|
||||
|
||||
WAN2.2 is the reference migration for the first contract version. Other video
|
||||
pipelines continue using their existing output paths until they can declare the
|
||||
same typed media contract without bypassing model-specific float consumers such
|
||||
as safety checks or audio/video packaging.
|
||||
|
||||
See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
|
||||
contract, lifecycle, batching rules, and migration plan.
|
||||
Reference in new issue
Block a user