Kandinsky6 (#8537)
Signed-off-by: leffff <levnovitskiy@gmail.com> Signed-off-by: Artem Borisov <divotionchannel@gmail.com> Co-authored-by: leffff <levnovitskiy@gmail.com> Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
This commit is contained in:
38 files changed
+8574
-8
No files matched your search
@@ -149,6 +149,10 @@ x-lingbot-video: &lingbot_video
|
||||
x-lingbot-world: &lingbot_world
|
||||
- vllm_omni/diffusion/models/lingbot_world/
|
||||
|
||||
x-kandinsky6: &kandinsky6
|
||||
- vllm_omni/diffusion/models/kandinsky6/
|
||||
- vllm_omni/model_extras/kandinsky6.py
|
||||
|
||||
|
||||
source_file_dependencies:
|
||||
# Hardware presets and shared test helpers. Not a job key — do not attach this
|
||||
@@ -491,6 +495,11 @@ source_file_dependencies:
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json
|
||||
|
||||
diffusion_kandinsky6_function:
|
||||
- *kandinsky6
|
||||
- tests/e2e/offline_inference/test_kandinsky6.py
|
||||
- tests/e2e/online_serving/test_kandinsky6.py
|
||||
|
||||
|
||||
# Invalid-param jobs: model business code only, plus the reliability scripts.
|
||||
reliability_invalid_param_h100:
|
||||
|
||||
@@ -189,6 +189,15 @@ steps:
|
||||
'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Kandinsky6 Test"
|
||||
source_file_dependencies: diffusion_kandinsky6_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
pytest -s -v tests/e2e/offline_inference/test_kandinsky6.py tests/e2e/online_serving/test_kandinsky6.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
commands:
|
||||
|
||||
@@ -106,6 +106,7 @@ th {
|
||||
| `SenseNovaU1Pipeline` | SenseNova-U1-A3B (MoE) | `sensenova/SenseNova-U1-A3B-MoT` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.md#a3b-moe-variant) |
|
||||
| `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — |
|
||||
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
|
||||
| `Kandinsky6TI2VAPipeline` | Kandinsky 6 TI2VA | `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Kandinsky/Kandinsky6-TI2VA.md) |
|
||||
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
|
||||
@@ -196,6 +196,7 @@ The following tables show which models support each feature:
|
||||
| **SANA-Video-2B T2V I2V** | ❌ | ❌ | ✅ (`--usp`, frame-sharded) | ✅ | ✅ (TP=2 only) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
|
||||
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **Kandinsky 6 TI2VA** | ✅* (uncalibrated) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ (FP8) | ✅ (1 request) |
|
||||
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Image-To-Video
|
||||
|
||||
This shared example generates videos from images with VACE, Wan2.2, LTX-2,
|
||||
HunyuanVideo-1.5, SANA-Video, Cosmos3, MAGI-2, and other compatible pipelines.
|
||||
HunyuanVideo-1.5, SANA-Video, Cosmos3, MAGI-2, Kandinsky 6, and other compatible pipelines.
|
||||
|
||||
- `image_to_video.py`: command-line script for single video generation with advanced options.
|
||||
|
||||
@@ -30,6 +30,7 @@ This folder provides a unified CLI script for image-to-video generation using vL
|
||||
| `sand-ai/MAGI-2-preview` | 512 x 896 | 125 | 100 | Model-fixed | Native four-GPU TP/SP; resident SP4 default; DLO available |
|
||||
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | 480 x 832 | 81 | 50 | 6.0 | Native `SanaImageToVideoPipeline`; Wan VAE |
|
||||
| `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | 704 x 1280 | 81 | 50 | 6.0 | Native `SanaImageToVideoPipeline`; LTX-2 Video VAE |
|
||||
| `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | 480 x 864 | 125 | 50 | 5.0 | ~75 GiB reserved (H100, CPU offload); gated Hub repo |
|
||||
|
||||
MAGI-2 native Preview setup, four-GPU topology and DLO choices, I2VA commands,
|
||||
and eight-GPU validation status are documented in the
|
||||
|
||||
@@ -14,6 +14,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
|
||||
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
|
||||
| `nvidia/Cosmos3-Nano` | 720x1280 | 189 | 35 | 6.0 | ~46 GiB (peak, 720p) |
|
||||
| `BestWishYsh/Helios-Base` / `Helios-Mid` / `Helios-Distilled` | 384x640 | 99 | 50 | 5.0 / 5.0 / 1.0 | — |
|
||||
| `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | 480x864 | 125 | 50 | 5.0 | ~75 GiB reserved (H100, CPU offload, SDPA) |
|
||||
| `sand-ai/MAGI-2-preview` | 512x896 | 125 | 100 | Model-fixed | Native four-GPU TP/SP; resident SP4 default; DLO available |
|
||||
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | 480x832 | 81 | 50 | 6.0 | BF16 DiT + FP32 Wan VAE wrapper |
|
||||
| `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | 704x1280 | 81 | 50 | 6.0 | BF16 DiT + LTX-2 Video VAE wrapper |
|
||||
@@ -249,6 +250,24 @@ python text_to_video.py \
|
||||
> conditioning tensors that cannot be passed through the JSON `--extra-body`
|
||||
> flag; they are out of scope for this text-to-video example.
|
||||
|
||||
### Kandinsky 6 (TI2VA)
|
||||
|
||||
The Hub checkpoint is gated; export `HF_TOKEN` if download returns 401.
|
||||
`--enable-cpu-offload` is required on an 80 GB H100. Audio is on by default.
|
||||
|
||||
```bash
|
||||
python text_to_video.py \
|
||||
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
|
||||
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
|
||||
--enable-cpu-offload \
|
||||
--output kandinsky6_t2va.mp4
|
||||
```
|
||||
|
||||
Video only: add `--extra-body '{"sample_audio": false}'`. Image-to-video uses
|
||||
the shared [`image_to_video.py`](../image_to_video/README.md) example with
|
||||
`--image`. Serving numbers and flags are in the
|
||||
[`Kandinsky 6 TI2VA recipe`](../../../recipes/Kandinsky/Kandinsky6-TI2VA.md).
|
||||
|
||||
## Key Arguments
|
||||
|
||||
### Common
|
||||
|
||||
@@ -235,7 +235,7 @@ def parse_extra_body(value: str) -> dict[str, Any]:
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate a video from a text prompt. "
|
||||
"Supports Wan2.2, HunyuanVideo-1.5, Helios, LingBot-Video, MAGI-2, "
|
||||
"Supports Wan2.2, HunyuanVideo-1.5, Helios, LingBot-Video, MAGI-2, Kandinsky 6, "
|
||||
"and other text-to-video models."
|
||||
)
|
||||
parser.add_argument(
|
||||
|
||||
@@ -232,6 +232,11 @@ ignore = [
|
||||
# F-correctness rules still apply.
|
||||
"vllm_omni/model_executor/models/nemotron_voicechat/nemo_vendored/**" = ["E501", "E721", "E741", "F401", "F841", "I001", "N803", "N814", "UP006", "UP007", "UP008", "UP011", "UP032", "UP035", "W291", "W293"]
|
||||
"vllm_omni/experimental/fullduplex/joyvl/decision/prompts.py" = ["E501"] # System/summarizer prompts are long prose lines
|
||||
# Kandinsky 6 keeps native/BigVGAN names (T/H/W, `l`) and NVIDIA vocoder
|
||||
# docstring length so the port stays diffable against the source modules.
|
||||
"vllm_omni/diffusion/models/kandinsky6/kandinsky6_transformer.py" = ["N803", "E741"]
|
||||
"vllm_omni/diffusion/models/kandinsky6/pipeline_kandinsky6.py" = ["N803"]
|
||||
"vllm_omni/diffusion/models/kandinsky6/modeling_kandinsky6_audio.py" = ["E501", "E741", "N803", "UP008"]
|
||||
|
||||
[tool.ruff.lint.flake8-tidy-imports.banned-api]
|
||||
"librosa".msg = "The librosa module is banned, use vllm.multimodal helpers instead"
|
||||
@@ -364,6 +369,8 @@ extend-ignore-identifiers-re = [
|
||||
# (output feature dim), not "doubt".
|
||||
".*dout.*",
|
||||
".*DOUT.*",
|
||||
# Kandinsky nabla threshold argument (`thr`), not the word "the".
|
||||
"^thr$",
|
||||
]
|
||||
extend-ignore-re = [
|
||||
"\\.\\.\\.n,d->\\.\\.\\.nd",
|
||||
@@ -404,3 +411,8 @@ IY = "IY"
|
||||
# Speech DAC codec identifiers (convnet_type, transconvnet_type) and is not a
|
||||
# typo of "convent".
|
||||
convnet = "convnet"
|
||||
# Upstream Kandinsky system prompt. "promt" and "scren" are part of the
|
||||
# conditioning text and must not be corrected.
|
||||
thr = "thr"
|
||||
promt = "promt"
|
||||
scren = "scren"
|
||||
@@ -0,0 +1,247 @@
|
||||
# Kandinsky 6 TI2VA — H100 (v1, single GPU, CPU offload)
|
||||
|
||||
> Text/image-to-video-and-audio serving with Kandinsky 6 Pro
|
||||
|
||||
## Summary
|
||||
|
||||
- Vendor: Kandinsky Lab ([`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers))
|
||||
- Model: Kandinsky 6 Pro (TI2VA — text/image-to-video-and-audio)
|
||||
- Task: Joint text/image-to-video-and-audio generation
|
||||
- Mode: Offline (`Omni(...)`) and online serving with the OpenAI-compatible
|
||||
`/v1/videos` API
|
||||
- Hardware: 1x NVIDIA H100 80GB HBM3 with component-level CPU offload
|
||||
- Maintainer: Community
|
||||
|
||||
## When to use this recipe
|
||||
|
||||
Use this recipe to serve Kandinsky 6's TI2VA pipeline — a single DiT that
|
||||
jointly denoises video and audio latents, conditioned on a text prompt and,
|
||||
optionally, a reference image (image-to-video). Tensor parallel, CFG
|
||||
parallel, Ulysses, ring, pipeline parallel, HSDP, layerwise offload,
|
||||
MagCache, TeaCache, Cache-DiT, FP8, and tiled VAE patch parallel are wired.
|
||||
NaviCache and a registered NABLA backend are not. TeaCache coefficients are
|
||||
uncalibrated. See "Known limitations".
|
||||
|
||||
## Supported model contract
|
||||
|
||||
| Task | Entrypoint | Input | Output |
|
||||
|---|---|---|---|
|
||||
| Text-to-video-and-audio | `vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni` / `Omni(model="kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers")` | text prompt | synchronized MP4 (H.264 video + AAC 44.1 kHz mono audio) |
|
||||
| Image-to-video-and-audio | same | text prompt + reference image | synchronized MP4, first frame conditioned on the reference image |
|
||||
|
||||
- Audio generation is on by default (`sample_audio=True`); pass
|
||||
`extra_args.sample_audio=false` to generate video only.
|
||||
- Batch size is 1 request per generation call (upstream limitation of the
|
||||
ported denoise loop; concurrent requests are still served by vLLM-Omni's
|
||||
own request scheduler, just not batched together within one DiT forward
|
||||
pass yet).
|
||||
- Serving defaults follow the Pro production geometry registered in
|
||||
`vllm_omni/model_extras/kandinsky6.py`: **864x480, 125 frames @ 24 fps
|
||||
(5.2 s), 50 steps, CFG 5.0**. Any `4k+1` frame count and 16-divisible
|
||||
resolution can be requested explicitly via `height`/`width`/`num_frames`/
|
||||
`num_inference_steps` — the model is *not* fixed-duration.
|
||||
- Audio is always 44.1 kHz; its length is derived from the video duration.
|
||||
- Negative prompt defaults to the pipeline's built-in K6 negative prompt
|
||||
when omitted.
|
||||
|
||||
## References
|
||||
|
||||
- Hub checkpoint: [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
|
||||
- Joint video+audio native-model reference: [`recipes/MiniMaxAI/MiniMax-H3.md`](../MiniMaxAI/MiniMax-H3.md)
|
||||
- Offline example: [`examples/offline_inference/text_to_video/text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py)
|
||||
- Image-to-video example: [`examples/offline_inference/image_to_video/image_to_video.py`](../../examples/offline_inference/image_to_video/image_to_video.py)
|
||||
|
||||
## Hardware
|
||||
|
||||
- Accelerator model and per-device memory: NVIDIA H100 80GB HBM3
|
||||
(driver 570.133.20)
|
||||
- Number of devices: 1 (multi-GPU tensor parallelism is implemented in the
|
||||
transformer's linear layers but not yet validated end-to-end)
|
||||
- Device interconnect: N/A (single-device path)
|
||||
- Host memory: 1.4 TB installed; the bf16 DiT (~56 GB) plus the text
|
||||
encoders are staged on the host under `--enable-cpu-offload`, so budget
|
||||
at least ~90 GB of free host RAM.
|
||||
- Qualification scope: single-GPU T2VA at the Pro default geometry and at a
|
||||
reduced smoke geometry, offline and via `vllm serve`. Numbers below were
|
||||
measured once on the shared box described above; treat them as
|
||||
indicative, not as a benchmark.
|
||||
|
||||
## Software environment
|
||||
|
||||
- OS: Ubuntu 22.04.5 LTS
|
||||
- Python: 3.12.13
|
||||
- Driver / runtime: CUDA 12.9 (torch 2.13.0+cu129)
|
||||
- vLLM version: 0.29.0
|
||||
- vLLM-Omni version or commit: this repository, `vllm_omni/diffusion/models/kandinsky6/`
|
||||
- diffusers 0.40.0, transformers 5.14.1
|
||||
- Attention: FlashAttention-3 on this box (`flash_attn_interface`). The Pro
|
||||
timings below use it. Without FA3, `attention_engine: "auto"` falls back
|
||||
to PyTorch SDPA; an earlier SDPA Pro run on the same GPU was 26.9 s/it
|
||||
and 1377 s.
|
||||
|
||||
## Weights
|
||||
|
||||
`Kandinsky6TI2VAPipeline._load_components` loads the public Diffusers layout
|
||||
from [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
|
||||
(`model_index.json`, `transformer/`, `vae/`, `text_encoder/`, `tokenizer/`,
|
||||
`text_encoder_2/`, `tokenizer_2/`, `audio_vae/`, `scheduler/`). Pass that
|
||||
repo id (or a local snapshot of it) to `vllm serve` / `Omni(model=...)`.
|
||||
The Hub repo is gated; export `HF_TOKEN` if download returns 401.
|
||||
|
||||
A raw Kandinsky 6 weight tree is not required. If you already have a local
|
||||
Diffusers-layout snapshot, point `--model` at that directory instead of the
|
||||
Hub id.
|
||||
|
||||
Each weight folder is loaded on its own (`transformer/`, `vae/`, `text_encoder/`, `text_encoder_2/`, `audio_vae/`). The DiT is built with initialization skipped, then the framework loader fills it from `transformer/`.
|
||||
|
||||
## Command
|
||||
|
||||
```bash
|
||||
vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni \
|
||||
--host 127.0.0.1 --port 8091 \
|
||||
--num-gpus 1 --enable-cpu-offload
|
||||
```
|
||||
|
||||
`--enable-cpu-offload` enables model-level offload with mutual exclusion
|
||||
between the transformer and the two text encoders (they never co-reside on
|
||||
the GPU). Without it the bf16 DiT (~56 GB) plus Qwen2.5-VL-7B (~16 GB) plus
|
||||
the VAEs and activations do not fit an 80 GB device at the Pro geometry.
|
||||
|
||||
Offline (writes an MP4 with the audio track muxed in):
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
|
||||
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
|
||||
--seed 42 --enable-cpu-offload --output kandinsky6_t2va.mp4
|
||||
# Video only: add --extra-body '{"sample_audio": false}'
|
||||
```
|
||||
|
||||
Image-to-video-and-audio uses the shared image-to-video example:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||||
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
|
||||
--image first_frame.png \
|
||||
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
|
||||
--seed 42 --enable-cpu-offload --output kandinsky6_i2va.mp4
|
||||
```
|
||||
|
||||
The generic scripts pick up the Pro defaults from
|
||||
`vllm_omni/model_extras/kandinsky6.py`.
|
||||
|
||||
## Verification
|
||||
|
||||
`/v1/videos` takes multipart form fields (not a JSON body). Submit, poll,
|
||||
then download:
|
||||
|
||||
```bash
|
||||
VID=$(curl -s -X POST http://127.0.0.1:8091/v1/videos \
|
||||
-F prompt="A golden retriever runs along a sunny beach, waves crashing" \
|
||||
-F size=864x480 -F num_frames=125 -F num_inference_steps=50 -F seed=42 \
|
||||
| python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")
|
||||
|
||||
# Poll until "status": "completed"
|
||||
curl -s http://127.0.0.1:8091/v1/videos/$VID
|
||||
|
||||
curl -s -o kandinsky6.mp4 http://127.0.0.1:8091/v1/videos/$VID/content
|
||||
# Expect an MP4 with an H.264 video stream (125 frames, 24 fps, 864x480)
|
||||
# and an AAC audio stream (44100 Hz, mono) of matching duration.
|
||||
```
|
||||
|
||||
Smoke-size request for quick validation (~30 s end-to-end on one H100):
|
||||
`-F size=512x320 -F num_frames=25 -F num_inference_steps=10`.
|
||||
|
||||
## Measurements
|
||||
|
||||
Single H100 80GB, seed 42, prompt
|
||||
"A golden retriever runs along a sunny beach, waves crashing, cinematic footage",
|
||||
negative prompt left at the pipeline default, MagCache off, prompt expansion
|
||||
off. Pro geometry: 864x480, 125 frames, 50 steps, CFG 5.0, audio on.
|
||||
|
||||
vLLM-Omni: `text_to_video.py --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --enable-cpu-offload --model-class-name Kandinsky6TI2VAPipeline`,
|
||||
FlashAttention-3.
|
||||
|
||||
| Wall / generation | Step time | Peak GPU | After load |
|
||||
|---|---|---|---|
|
||||
| 751.7 s generation (12.5 min); engine init 41 s | 14.4 s/it average | 78.39 GiB reserved (78.0 GiB nvidia-smi) | 17.74 GiB |
|
||||
|
||||
An earlier SDPA vLLM-Omni Pro run on the same GPU was 26.9 s/it and 1377 s,
|
||||
peak reserved 74.97 GiB. A 512x320 / 25-frame / 10-step SDPA smoke was 1.4 s/it
|
||||
and 29.5 s at the same reserved peak.
|
||||
|
||||
The MP4 is H.264 864x480, 125 frames at 24 fps (5.22 s), plus AAC 44.1 kHz
|
||||
mono (230400 samples): a golden retriever on a sunny beach with breaking waves.
|
||||
|
||||
vLLM-Omni's VAE decode logged two failed CUDA allocations (7.15 GiB, then
|
||||
4.27 GiB) and still wrote a complete file.
|
||||
|
||||
Audio is peak-normalized to full scale by default; pass
|
||||
`extra_args.audio_normalization="clip"` to keep the raw decoded amplitude
|
||||
instead.
|
||||
|
||||
## Notes
|
||||
|
||||
- Key flags: `extra_args.sample_audio` (bool, default true),
|
||||
`extra_args.audio_normalization` (`"normalize"` default | `"clip"`),
|
||||
`extra_args.visual_cond_scheme` (defaults to `pretrain` for text-only,
|
||||
`tail_cond_first_frame` when an image is supplied).
|
||||
- The post-process payload is a flat dict (`video`, `audio` as float32 in
|
||||
`[-1, 1]`, `audio_sample_rate=44100`, `fps`) so the serving muxer picks
|
||||
up the correct sample rate.
|
||||
- Acceleration that is wired:
|
||||
- Tensor parallel: the DiT loader narrows full checkpoint tensors onto
|
||||
each rank's `ColumnParallelLinear` / `RowParallelLinear` shard.
|
||||
- CFG parallel (`--cfg-parallel-size 2`): rank 0 runs the conditional
|
||||
DiT, rank 1 the unconditional one, then both all-gather video and
|
||||
audio velocities.
|
||||
- HSDP: `_hsdp_shard_conditions` covers the indexed transformer blocks.
|
||||
Do not combine with tensor parallel.
|
||||
- Sequence parallel: visual tokens (and their RoPE) are sharded;
|
||||
Ulysses all-to-all runs inside visual self-attention, Ring uses the
|
||||
PyTorch ring kernel. Text stays replicated. Audio queries gather the
|
||||
full visual sequence.
|
||||
- Pipeline parallel: `visual_transformer_blocks` are split with
|
||||
`PPMissingLayer`. Embeddings stay on the first stage, output heads on
|
||||
the last. Activations and the final velocity are sent between stages.
|
||||
- VAE patch parallel: Hunyuan decode splits the latent width when
|
||||
`vae_patch_parallel_size > 1` and tiling is on.
|
||||
- MagCache / TeaCache / Cache-DiT are registered. MagCache reuses the
|
||||
Pro `mag_ratios`. TeaCache coefficients are **uncalibrated** (copied
|
||||
from Qwen-Image) so skips are not quality-neutral. Cache-DiT targets
|
||||
`visual_transformer_blocks` only and falls back to the same step cache
|
||||
if the fused `(video, audio)` block return is rejected.
|
||||
- FP8: online quantization runs in the DiT loader via
|
||||
`process_weights_after_loading` after the sharded assign.
|
||||
- `attention_engine: auto` uses FlashAttention-3 when
|
||||
`flash_attn_interface` imports.
|
||||
- Known limitations:
|
||||
- NABLA sparse attention is not registered in the framework attention
|
||||
backend. NaviCache is not wired.
|
||||
- Prompt expansion, NF4 Qwen, and the super-resolution cascade are not
|
||||
part of this port.
|
||||
- Distributed layerwise offload still uses one process unless another
|
||||
parallel axis sets `world_size > 1` (for example tensor parallel).
|
||||
- `_load_components` loads the Diffusers checkpoint directly rather than the
|
||||
streamed-prefetch loader.
|
||||
|
||||
## Supported features
|
||||
|
||||
| Feature | Status | Guide |
|
||||
|---|---|---|
|
||||
| Text-to-video-and-audio | Supported (validated on H100) | — |
|
||||
| Image-to-video-and-audio | Supported (not yet validated on hardware) | — |
|
||||
| CPU offload (`--enable-cpu-offload`) | Supported (required on 80 GB) | — |
|
||||
| Layerwise offload (`--enable-layerwise-offload`) | Supported (about 36 GiB peak on the smoke geometry) | — |
|
||||
| Tensor parallelism | Wired (sharded DiT load) | — |
|
||||
| CFG parallelism | Wired (`--cfg-parallel-size 2`) | — |
|
||||
| Ulysses / Ring sequence parallel | Wired (visual tokens only) | — |
|
||||
| Pipeline parallelism | Wired (visual block split) | — |
|
||||
| HSDP | Wired (not with tensor parallel) | — |
|
||||
| VAE patch parallel | Wired (width-split decode) | — |
|
||||
| MagCache | Wired (Pro ratios, step reuse) | — |
|
||||
| TeaCache | Wired, coefficients uncalibrated | — |
|
||||
| Cache-DiT | Wired on `visual_transformer_blocks` | — |
|
||||
| FP8 | Wired (online quant; smoke peak about 46 GiB vs 75 GiB bf16) | [`FP8 quantization`](../../docs/user_guide/quantization/fp8.md) |
|
||||
| NaviCache | Not supported | — |
|
||||
| LoRA | Not supported | — |
|
||||
@@ -64,6 +64,7 @@ recipes/
|
||||
| [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) |
|
||||
| [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
|
||||
| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB / 1x Intel Arc Pro B70 32GB |
|
||||
| [`Kandinsky/Kandinsky6-TI2VA.md`](./Kandinsky/Kandinsky6-TI2VA.md) | Text/image-to-video-and-audio, offline and `/v1/videos` | 1x H100 80GB with CPU offload; 2x H100 for parallel modes |
|
||||
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU / 1x Intel Arc Pro B70 32GB |
|
||||
| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified / 1x Intel Arc Pro B70 32GB |
|
||||
| [`m-a-p/SheetSage2-H200.md`](./m-a-p/SheetSage2-H200.md) | Official audio-to-score preprocessing and YuE2 request export | 1x H200 141GB |
|
||||
|
||||
+1
@@ -59,6 +59,7 @@ def test_custom_cache_dit_enablers_are_registered_explicitly():
|
||||
"Cosmos3OmniDiffusersPipeline": cd_model_specific.enable_cache_for_cosmos3,
|
||||
"Cosmos3OmniPipeline": cd_model_specific.enable_cache_for_cosmos3,
|
||||
"Krea2Pipeline": cd_model_specific.enable_cache_for_krea2,
|
||||
"Kandinsky6TI2VAPipeline": cd_model_specific.enable_cache_for_kandinsky6,
|
||||
"Magi2Pipeline": cd_model_specific.enable_cache_for_magi2,
|
||||
"MammothModa2DiTPipeline": cd_model_specific.enable_cache_for_mammothmoda2,
|
||||
}
|
||||
|
||||
Whitespace-only changes.
@@ -0,0 +1,89 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""CPU checks that Kandinsky 6 acceleration hooks are actually wired."""
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
from vllm_omni.diffusion.models.kandinsky6.cache_accel import K6_PRO_MAG_RATIOS, K6StepCache
|
||||
from vllm_omni.diffusion.models.kandinsky6.kandinsky6_transformer import Kandinsky6Transformer3DModel
|
||||
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import _shard_loaded_weight
|
||||
from vllm_omni.diffusion.registry import _NO_CACHE_ACCELERATION
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
|
||||
|
||||
|
||||
def test_shard_loaded_weight_narrows_column_parallel_output():
|
||||
class _Owner:
|
||||
tp_rank = 1
|
||||
|
||||
def weight_loader(self, param, loaded):
|
||||
del param, loaded
|
||||
|
||||
param = nn.Parameter(torch.empty(2, 4))
|
||||
param.output_dim = 0
|
||||
param.weight_loader = _Owner().weight_loader
|
||||
loaded = torch.arange(16, dtype=torch.float32).reshape(4, 4)
|
||||
|
||||
shard = _shard_loaded_weight(param, loaded)
|
||||
|
||||
assert tuple(shard.shape) == (2, 4)
|
||||
torch.testing.assert_close(shard, loaded[2:4])
|
||||
|
||||
|
||||
def test_row_bias_is_not_sharded_when_shapes_match():
|
||||
class _Owner:
|
||||
tp_rank = 1
|
||||
|
||||
def weight_loader(self, param, loaded):
|
||||
del param, loaded
|
||||
|
||||
param = nn.Parameter(torch.empty(4))
|
||||
param.output_dim = 0
|
||||
param.weight_loader = _Owner().weight_loader
|
||||
loaded = torch.arange(4, dtype=torch.float32)
|
||||
|
||||
shard = _shard_loaded_weight(param, loaded)
|
||||
|
||||
torch.testing.assert_close(shard, loaded)
|
||||
|
||||
|
||||
def test_hsdp_predicate_matches_visual_blocks_only_when_indexed():
|
||||
assert Kandinsky6Transformer3DModel._is_transformer_block("visual_transformer_blocks.3", nn.Identity())
|
||||
assert Kandinsky6Transformer3DModel._is_transformer_block("video_text_transformer_blocks.0", nn.Identity())
|
||||
assert not Kandinsky6Transformer3DModel._is_transformer_block("visual_embeddings", nn.Identity())
|
||||
assert Kandinsky6Transformer3DModel._hsdp_shard_conditions
|
||||
|
||||
|
||||
def test_sp_plan_shards_visual_tokens_and_gathers_before_the_head():
|
||||
plan = Kandinsky6Transformer3DModel._sp_plan
|
||||
assert "_sp_visual_shard" in plan
|
||||
assert "_sp_visual_rope" in plan
|
||||
assert plan["_sp_visual_gather"].gather_dim == 1
|
||||
assert Kandinsky6Transformer3DModel._magcache_block_attrs == ("visual_transformer_blocks",)
|
||||
|
||||
|
||||
def test_k6_is_not_on_the_no_cache_list_and_enablers_are_registered():
|
||||
import vllm_omni.diffusion.cache.cachedit # noqa: F401 registers enablers
|
||||
from vllm_omni.diffusion.cache.cachedit.backend import CUSTOM_DIT_ENABLERS
|
||||
from vllm_omni.diffusion.cache.teacache.backend import CUSTOM_TEACACHE_ENABLERS
|
||||
|
||||
assert "Kandinsky6TI2VAPipeline" not in _NO_CACHE_ACCELERATION
|
||||
assert "Kandinsky6TI2VAPipeline" in CUSTOM_TEACACHE_ENABLERS
|
||||
assert "Kandinsky6TI2VAPipeline" in CUSTOM_DIT_ENABLERS
|
||||
assert len(K6_PRO_MAG_RATIOS) == 100
|
||||
|
||||
|
||||
def test_step_cache_skips_only_after_a_real_step():
|
||||
cache = K6StepCache(
|
||||
kind="mag",
|
||||
threshold=0.24,
|
||||
max_skip_steps=3,
|
||||
retention_ratio=0.0,
|
||||
mag_ratios=(1.0, 1.0),
|
||||
)
|
||||
assert cache.should_skip(0) is False
|
||||
cache.store(torch.zeros(1), torch.zeros(1))
|
||||
assert cache.should_skip(1) is True
|
||||
@@ -0,0 +1,246 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
|
||||
|
||||
|
||||
def test_kandinsky6_pipeline_import_and_registry():
|
||||
from vllm_omni.diffusion.models.kandinsky6 import (
|
||||
AutoencoderKLHunyuanVideo,
|
||||
Kandinsky6AudioVAE,
|
||||
Kandinsky6TI2VAPipeline,
|
||||
Kandinsky6Transformer3DModel,
|
||||
KandinskyFlowMatchScheduler,
|
||||
get_kandinsky6_post_process_func,
|
||||
get_kandinsky6_pre_process_func,
|
||||
)
|
||||
from vllm_omni.diffusion.registry import (
|
||||
_DIFFUSION_MODELS,
|
||||
_DIFFUSION_POST_PROCESS_FUNCS,
|
||||
_DIFFUSION_PRE_PROCESS_FUNCS,
|
||||
_NO_CACHE_ACCELERATION,
|
||||
)
|
||||
|
||||
assert Kandinsky6TI2VAPipeline is not None
|
||||
assert Kandinsky6Transformer3DModel is not None
|
||||
assert AutoencoderKLHunyuanVideo is not None
|
||||
assert Kandinsky6AudioVAE is not None
|
||||
assert KandinskyFlowMatchScheduler is not None
|
||||
assert get_kandinsky6_post_process_func is not None
|
||||
assert get_kandinsky6_pre_process_func is not None
|
||||
|
||||
assert _DIFFUSION_MODELS["Kandinsky6TI2VAPipeline"] == (
|
||||
"kandinsky6",
|
||||
"pipeline_kandinsky6",
|
||||
"Kandinsky6TI2VAPipeline",
|
||||
)
|
||||
assert _DIFFUSION_POST_PROCESS_FUNCS["Kandinsky6TI2VAPipeline"] == "get_kandinsky6_post_process_func"
|
||||
assert _DIFFUSION_PRE_PROCESS_FUNCS["Kandinsky6TI2VAPipeline"] == "get_kandinsky6_pre_process_func"
|
||||
assert "Kandinsky6TI2VAPipeline" not in _NO_CACHE_ACCELERATION
|
||||
|
||||
|
||||
def test_kandinsky6_component_discovery_declarations():
|
||||
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6TI2VAPipeline
|
||||
|
||||
assert Kandinsky6TI2VAPipeline._dit_modules == ["transformer"]
|
||||
assert Kandinsky6TI2VAPipeline._encoder_modules == ["text_encoder", "text_encoder_2"]
|
||||
assert Kandinsky6TI2VAPipeline._vae_modules == ["vae", "audio_vae"]
|
||||
assert Kandinsky6TI2VAPipeline.supports_step_execution is True
|
||||
assert Kandinsky6TI2VAPipeline.support_audio_output is True
|
||||
assert Kandinsky6TI2VAPipeline.support_image_input is True
|
||||
|
||||
|
||||
def test_kandinsky6_pipeline_satisfies_capability_protocols():
|
||||
"""isinstance checks against the @runtime_checkable protocols the
|
||||
framework actually uses to detect these capabilities (io_support.py's
|
||||
supports_audio_output, etc.) — stronger than just checking the class
|
||||
attribute exists."""
|
||||
from vllm_omni.diffusion.models.interface import (
|
||||
SupportAudioOutput,
|
||||
SupportImageInput,
|
||||
SupportsComponentDiscovery,
|
||||
)
|
||||
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6TI2VAPipeline
|
||||
|
||||
assert isinstance(Kandinsky6TI2VAPipeline, SupportAudioOutput)
|
||||
assert isinstance(Kandinsky6TI2VAPipeline, SupportImageInput)
|
||||
assert isinstance(Kandinsky6TI2VAPipeline, SupportsComponentDiscovery)
|
||||
|
||||
|
||||
class _FakeAudioVAE:
|
||||
downsample_factor = 1024
|
||||
|
||||
|
||||
def test_kandinsky6_post_process_func_packages_video_and_audio():
|
||||
"""Pure-function test: no model/weights needed. Confirms the post-process
|
||||
payload shape matches what io_support.py -> output_formatter.py ->
|
||||
media_utils.py's PyAV muxing expects (the same flat {"video", "audio",
|
||||
"audio_sample_rate", "fps"} shape MiniMax H3's own post-process function
|
||||
produces), and that int16 PCM from ``postprocess_audio`` is rescaled to
|
||||
the float32 [-1, 1] waveform the muxer consumes."""
|
||||
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
|
||||
|
||||
post_process = get_kandinsky6_post_process_func(od_config=None)
|
||||
video = np.zeros((1, 4, 8, 8, 3), dtype=np.uint8)
|
||||
audio = np.array([0, 32767, -32767], dtype=np.int16)
|
||||
|
||||
result = post_process({"video": video, "audio": audio, "audio_sample_rate": 44100}, output_type="np")
|
||||
|
||||
assert result["video"] is video
|
||||
assert result["audio"].dtype == np.float32
|
||||
np.testing.assert_allclose(result["audio"], [0.0, 1.0, -1.0])
|
||||
assert result["audio_sample_rate"] == 44100
|
||||
assert result["fps"] == 24.0
|
||||
|
||||
|
||||
def test_kandinsky6_post_process_sample_rate_survives_output_formatter():
|
||||
"""The formatter only lifts ``audio_sample_rate`` into metadata from the
|
||||
flat payload form; an envelope would silently lose it and the MP4 would
|
||||
be muxed at the 24 kHz default."""
|
||||
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
|
||||
from vllm_omni.diffusion.output_formatter import normalize_diffusion_postprocess_output
|
||||
|
||||
post_process = get_kandinsky6_post_process_func(od_config=None)
|
||||
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
|
||||
audio = np.zeros((10,), dtype=np.int16)
|
||||
|
||||
normalized = normalize_diffusion_postprocess_output(
|
||||
post_process({"video": video, "audio": audio, "audio_sample_rate": 44100}, output_type="np")
|
||||
)
|
||||
|
||||
assert normalized.primary_key == "video"
|
||||
assert normalized.metadata["audio"]["sample_rate"] == 44100
|
||||
assert normalized.metadata["video"]["fps"] == 24.0
|
||||
assert "audio_sample_rate" not in normalized.outputs
|
||||
|
||||
|
||||
def test_kandinsky6_post_process_func_omits_audio_when_none():
|
||||
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
|
||||
|
||||
post_process = get_kandinsky6_post_process_func(od_config=None)
|
||||
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
|
||||
|
||||
result = post_process({"video": video, "audio": None, "audio_sample_rate": None}, output_type="np")
|
||||
|
||||
assert result["video"] is video
|
||||
assert "audio" not in result
|
||||
assert "audio_sample_rate" not in result
|
||||
|
||||
|
||||
def test_kandinsky6_post_process_func_unwraps_batched_audio_list():
|
||||
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
|
||||
|
||||
post_process = get_kandinsky6_post_process_func(od_config=None)
|
||||
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
|
||||
audio_item = np.full((50,), 16384, dtype=np.int16)
|
||||
|
||||
result = post_process({"video": video, "audio": [audio_item], "audio_sample_rate": 44100}, output_type="np")
|
||||
|
||||
assert result["audio"].shape == (50,)
|
||||
np.testing.assert_allclose(result["audio"], 16384 / 32767, rtol=1e-6)
|
||||
|
||||
|
||||
class _StubAudioVAE:
|
||||
"""Audio VAE stand-in: identity "decode" that returns the latent as a waveform."""
|
||||
|
||||
scaling_factor = 0.5
|
||||
mean_value = 0.0
|
||||
device = "cpu"
|
||||
|
||||
def wrapped_decode(self, latents):
|
||||
# (1, audio_dim, A) -> (A,) waveform: take the first channel.
|
||||
return latents[0, 0]
|
||||
|
||||
|
||||
def test_postprocess_audio_normalizes_by_default_and_clips_on_request():
|
||||
"""The default mode must match the k6_video production pipeline, which
|
||||
peak-normalizes each decoded waveform to full scale; ``clip`` keeps the
|
||||
raw amplitude (saturated to [-1, 1]) for callers that want it."""
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import LatentBundle, postprocess_audio
|
||||
|
||||
# Latent (A=4, audio_dim=1); the stub scales by 1/scaling_factor -> [0.2, -0.4, 0.6, 3.0].
|
||||
audio = torch.tensor([[0.1], [-0.2], [0.3], [1.5]], dtype=torch.float32)
|
||||
bundle = LatentBundle(
|
||||
video=None,
|
||||
audio=audio,
|
||||
video_cu_seqlens=None,
|
||||
audio_cu_seqlens=torch.tensor([0, 4], dtype=torch.int32),
|
||||
)
|
||||
vae = _StubAudioVAE()
|
||||
|
||||
normalized = postprocess_audio(bundle, vae)
|
||||
assert normalized is not None and len(normalized) == 1
|
||||
assert normalized[0].dtype == np.int16
|
||||
np.testing.assert_allclose(
|
||||
normalized[0].astype(np.float32) / 32767,
|
||||
np.array([0.2, -0.4, 0.6, 3.0], dtype=np.float32) / 3.0,
|
||||
atol=1e-4,
|
||||
)
|
||||
|
||||
clipped = postprocess_audio(bundle, vae, normalization_mode="clip")
|
||||
np.testing.assert_allclose(
|
||||
clipped[0].astype(np.float32) / 32767,
|
||||
np.array([0.2, -0.4, 0.6, 1.0], dtype=np.float32),
|
||||
atol=1e-4,
|
||||
)
|
||||
|
||||
assert postprocess_audio(LatentBundle(None, None, None, None), vae) is None
|
||||
with pytest.raises(ValueError, match="normalization_mode"):
|
||||
postprocess_audio(bundle, vae, normalization_mode="loud")
|
||||
|
||||
|
||||
def test_kandinsky6_pre_process_func_is_identity_for_now():
|
||||
"""v1 scope: the pre-process hook is registered (matching the framework's
|
||||
convention for image-conditioned models like SanaImageToVideoPipeline)
|
||||
but does not yet validate/transform the request — documented follow-up,
|
||||
not a silent gap."""
|
||||
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_pre_process_func
|
||||
|
||||
pre_process = get_kandinsky6_pre_process_func(od_config=None)
|
||||
sentinel = object()
|
||||
|
||||
assert pre_process(sentinel) is sentinel
|
||||
|
||||
|
||||
def test_hub_component_keys_map_onto_pipeline_modules():
|
||||
"""Hub folder keys from Kandinsky-6.0-Pro-5s-Diffusers land on the modules."""
|
||||
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import (
|
||||
_WEIGHT_SUBFOLDERS,
|
||||
_adapt_k6_weight_name,
|
||||
)
|
||||
|
||||
assert _WEIGHT_SUBFOLDERS == (
|
||||
("transformer", "transformer."),
|
||||
("vae", "vae."),
|
||||
("text_encoder", "text_encoder."),
|
||||
("text_encoder_2", "text_encoder_2."),
|
||||
("audio_vae", "audio_vae."),
|
||||
)
|
||||
assert _adapt_k6_weight_name("transformer.visual_transformer_blocks.0.videoT.self_attention.to_query.weight") == (
|
||||
"transformer.visual_transformer_blocks.0.video_dec_block.self_attention.to_query.weight"
|
||||
)
|
||||
assert _adapt_k6_weight_name("transformer.visual_transformer_blocks.0.audioT.feed_forward.in_layer.weight") == (
|
||||
"transformer.visual_transformer_blocks.0.audio_dec_block.feed_forward.in_layer.weight"
|
||||
)
|
||||
assert _adapt_k6_weight_name("vae.decoder.conv_in.conv.weight") == "vae.decoder.conv_in.conv.weight"
|
||||
assert _adapt_k6_weight_name("text_encoder_2.encoder.layers.0.mlp.fc1.weight") == (
|
||||
"text_encoder_2.encoder.layers.0.mlp.fc1.weight"
|
||||
)
|
||||
assert (
|
||||
_adapt_k6_weight_name("text_encoder.model.layers.0.input_layernorm.weight")
|
||||
== "text_encoder.model.language_model.layers.0.input_layernorm.weight"
|
||||
)
|
||||
assert _adapt_k6_weight_name("text_encoder.visual.blocks.0.norm1.weight") == (
|
||||
"text_encoder.model.visual.blocks.0.norm1.weight"
|
||||
)
|
||||
assert (
|
||||
_adapt_k6_weight_name("audio_vae.vae.decoder.conv_in.weight")
|
||||
== "audio_vae.native.tod.vae.decoder.conv_in.weight"
|
||||
)
|
||||
assert _adapt_k6_weight_name("audio_vae.vocoder.conv_pre.weight") == "audio_vae.native.tod.vocoder.conv_pre.weight"
|
||||
assert _adapt_k6_weight_name("audio_vae.mel_converter.hann_window") == "audio_vae.native.mel_converter.hann_window"
|
||||
@@ -0,0 +1,325 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _init_distributed(monkeypatch):
|
||||
"""The native transformer uses vLLM parallel linear layers, which require a
|
||||
tensor-parallel group; initialize a single-process group for CPU tests
|
||||
(mirrors tests/diffusion/models/sana_video/test_transformer_sana_video.py)."""
|
||||
from vllm.distributed.parallel_state import (
|
||||
cleanup_dist_env_and_memory,
|
||||
init_distributed_environment,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm.model_executor.layers.utils import default_unquantized_gemm
|
||||
|
||||
monkeypatch.setattr(
|
||||
"vllm.model_executor.layers.linear.dispatch_unquantized_gemm",
|
||||
lambda *_args, **_kwargs: default_unquantized_gemm,
|
||||
)
|
||||
os.environ.setdefault("MASTER_ADDR", "localhost")
|
||||
os.environ.setdefault("MASTER_PORT", "29502")
|
||||
init_distributed_environment(world_size=1, rank=0, local_rank=0, distributed_init_method="env://")
|
||||
initialize_model_parallel()
|
||||
yield
|
||||
cleanup_dist_env_and_memory()
|
||||
|
||||
|
||||
# model_dim must be an exact multiple of head_dim = sum(axes_dims); this is
|
||||
# an architectural constraint of the native model (DiffusionTransformer3D),
|
||||
# not specific to this port.
|
||||
_TINY_T2V_CONFIG = {
|
||||
"in_visual_dim": 4,
|
||||
"out_visual_dim": 4,
|
||||
"in_text_dim": 8,
|
||||
"in_text_dim2": 6,
|
||||
"time_dim": 16,
|
||||
"patch_size": (1, 2, 2),
|
||||
"model_dim": 24,
|
||||
"ff_dim": 32,
|
||||
"num_text_blocks": 1,
|
||||
"num_visual_blocks": 2,
|
||||
"axes_dims": (4, 4, 4),
|
||||
"visual_cond": False,
|
||||
"is_multimodal": False,
|
||||
"attention_engine": "sdpa",
|
||||
}
|
||||
|
||||
# time_dim_a is intentionally omitted (defaults to time_dim): the fused
|
||||
# block's cross-modal modulation (va_mod/av_mod) consumes the *other*
|
||||
# modality's time embedding by default (fix_modulation=False), so time_dim
|
||||
# and time_dim_a must match unless fix_modulation=True.
|
||||
_TINY_T2VA_CONFIG = {
|
||||
**_TINY_T2V_CONFIG,
|
||||
"is_multimodal": True,
|
||||
"in_audio_dim": 6,
|
||||
"model_dim_a": 12,
|
||||
"ff_dim_a": 16,
|
||||
"axes_dims_a": (2, 2, 2),
|
||||
}
|
||||
|
||||
|
||||
def _sdpa_config():
|
||||
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec
|
||||
|
||||
return SimpleNamespace(
|
||||
diffusion_attention_config=AttentionConfig(default=AttentionSpec(backend="TORCH_SDPA")),
|
||||
parallel_config=SimpleNamespace(ring_degree=1, allgather_degree=1),
|
||||
)
|
||||
|
||||
|
||||
def _build_transformer(config):
|
||||
from vllm_omni.diffusion.config import set_current_diffusion_config
|
||||
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6Transformer3DModel
|
||||
|
||||
with set_current_diffusion_config(_sdpa_config()):
|
||||
model = Kandinsky6Transformer3DModel(**config)
|
||||
model.eval()
|
||||
return model
|
||||
|
||||
|
||||
def test_t2v_only_forward_shape():
|
||||
model = _build_transformer(_TINY_T2V_CONFIG)
|
||||
time_frames, height, width = 2, 4, 4
|
||||
x_video = torch.randn(time_frames, height, width, 4)
|
||||
text_embed = torch.randn(5, 8)
|
||||
pooled = torch.randn(1, 6)
|
||||
time = torch.tensor([500.0])
|
||||
visual_rope = model.visual_rope_embeddings(
|
||||
shape=(time_frames, height // 2, width // 2),
|
||||
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
|
||||
scale_factor=(1.0, 1.0, 1.0),
|
||||
)
|
||||
text_rope = model.text_rope_embeddings(torch.arange(5))
|
||||
|
||||
with torch.no_grad():
|
||||
out = model(
|
||||
x_video=x_video,
|
||||
x_audio=None,
|
||||
text_embed=text_embed,
|
||||
pooled_text_embed=pooled,
|
||||
time=time,
|
||||
visual_rope=visual_rope,
|
||||
audio_rope=None,
|
||||
text_rope=text_rope,
|
||||
)
|
||||
|
||||
assert out.shape == (time_frames, height, width, 4)
|
||||
|
||||
|
||||
def test_visual_cond_channel_concat_path():
|
||||
"""visual_cond=True doubles+1 the visual input channel count (cond +
|
||||
mask channels appended by _build_video_input, stripped again by
|
||||
OutLayer) — the I2V conditioning path."""
|
||||
config = {**_TINY_T2V_CONFIG, "visual_cond": True}
|
||||
model = _build_transformer(config)
|
||||
time_frames, height, width = 2, 4, 4
|
||||
x_video = torch.randn(time_frames, height, width, 2 * 4 + 1)
|
||||
text_embed = torch.randn(5, 8)
|
||||
pooled = torch.randn(1, 6)
|
||||
time = torch.tensor([500.0])
|
||||
visual_rope = model.visual_rope_embeddings(
|
||||
shape=(time_frames, height // 2, width // 2),
|
||||
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
|
||||
scale_factor=(1.0, 1.0, 1.0),
|
||||
)
|
||||
text_rope = model.text_rope_embeddings(torch.arange(5))
|
||||
|
||||
with torch.no_grad():
|
||||
out = model(
|
||||
x_video=x_video,
|
||||
x_audio=None,
|
||||
text_embed=text_embed,
|
||||
pooled_text_embed=pooled,
|
||||
time=time,
|
||||
visual_rope=visual_rope,
|
||||
audio_rope=None,
|
||||
text_rope=text_rope,
|
||||
)
|
||||
|
||||
assert out.shape == (time_frames, height, width, 4)
|
||||
|
||||
|
||||
def test_t2va_joint_forward_shapes():
|
||||
"""Exercises the fused cross-modal decoder blocks (Kandinsky6FusedTransformerDecoderBlock)."""
|
||||
model = _build_transformer(_TINY_T2VA_CONFIG)
|
||||
time_frames, height, width = 2, 4, 4
|
||||
audio_len = 7
|
||||
x_video = torch.randn(time_frames, height, width, 4)
|
||||
x_audio = torch.randn(audio_len, 6)
|
||||
text_embed = torch.randn(5, 8)
|
||||
pooled = torch.randn(1, 6)
|
||||
time = torch.tensor([500.0])
|
||||
visual_rope = model.visual_rope_embeddings(
|
||||
shape=(time_frames, height // 2, width // 2),
|
||||
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
|
||||
scale_factor=(1.0, 1.0, 1.0),
|
||||
)
|
||||
audio_rope = model.audio_rope_embeddings(torch.arange(audio_len))
|
||||
video_text_rope = model.video_text_rope_embeddings(torch.arange(5))
|
||||
audio_text_rope = model.audio_text_rope_embeddings(torch.arange(5))
|
||||
|
||||
with torch.no_grad():
|
||||
video_out, audio_out = model(
|
||||
x_video=x_video,
|
||||
x_audio=x_audio,
|
||||
text_embed=[text_embed, text_embed],
|
||||
pooled_text_embed=[pooled, pooled],
|
||||
time=[time, time],
|
||||
visual_rope=visual_rope,
|
||||
audio_rope=audio_rope,
|
||||
text_rope=[video_text_rope, audio_text_rope],
|
||||
)
|
||||
|
||||
assert video_out.shape == (time_frames, height, width, 4)
|
||||
assert audio_out.shape == (audio_len, 6)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("modality", ["video", "audio"])
|
||||
def test_single_modality_through_multimodal_model(modality):
|
||||
"""A multimodal (is_multimodal=True) model must still support a pure
|
||||
video-only or audio-only forward call (denoise_loop's partial-sampling
|
||||
path uses this)."""
|
||||
model = _build_transformer(_TINY_T2VA_CONFIG)
|
||||
time_frames, height, width = 2, 4, 4
|
||||
audio_len = 7
|
||||
text_embed = torch.randn(5, 8)
|
||||
pooled = torch.randn(1, 6)
|
||||
time = torch.tensor([500.0])
|
||||
|
||||
if modality == "video":
|
||||
x_video = torch.randn(time_frames, height, width, 4)
|
||||
visual_rope = model.visual_rope_embeddings(
|
||||
shape=(time_frames, height // 2, width // 2),
|
||||
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
|
||||
scale_factor=(1.0, 1.0, 1.0),
|
||||
)
|
||||
text_rope = model.video_text_rope_embeddings(torch.arange(5))
|
||||
with torch.no_grad():
|
||||
out = model(
|
||||
x_video=x_video,
|
||||
x_audio=None,
|
||||
text_embed=text_embed,
|
||||
pooled_text_embed=pooled,
|
||||
time=time,
|
||||
visual_rope=visual_rope,
|
||||
audio_rope=None,
|
||||
text_rope=text_rope,
|
||||
)
|
||||
assert out.shape == (time_frames, height, width, 4)
|
||||
else:
|
||||
x_audio = torch.randn(audio_len, 6)
|
||||
audio_rope = model.audio_rope_embeddings(torch.arange(audio_len))
|
||||
text_rope = model.audio_text_rope_embeddings(torch.arange(5))
|
||||
with torch.no_grad():
|
||||
out = model(
|
||||
x_video=None,
|
||||
x_audio=x_audio,
|
||||
text_embed=text_embed,
|
||||
pooled_text_embed=pooled,
|
||||
time=time,
|
||||
visual_rope=None,
|
||||
audio_rope=audio_rope,
|
||||
text_rope=text_rope,
|
||||
)
|
||||
assert out.shape == (audio_len, 6)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("batched", [False, True])
|
||||
def test_self_attention_attends_over_tokens_for_batched_and_unbatched_inputs(batched):
|
||||
"""Self-attention must contract over the *token* axis whether its input is
|
||||
unbatched text ``(S, C)`` or batched=1 visual/audio ``(1, N, C)``.
|
||||
|
||||
Regression: an unconditional ``unsqueeze(0)`` on the already-batched
|
||||
visual path handed SDPA a 5-D ``(1, 1, N, H, D)`` tensor, which it
|
||||
accepts silently but then treats the head axis as the sequence — every
|
||||
generated video decoded to pure noise."""
|
||||
from vllm_omni.diffusion.config import set_current_diffusion_config
|
||||
from vllm_omni.diffusion.models.kandinsky6.kandinsky6_transformer import Kandinsky6Attention, apply_rotary
|
||||
|
||||
torch.manual_seed(0)
|
||||
channels, head_dim, tokens = 24, 12, 7
|
||||
with set_current_diffusion_config(_sdpa_config()):
|
||||
attn = Kandinsky6Attention(channels, head_dim, engine="sdpa").eval()
|
||||
with torch.no_grad():
|
||||
# vLLM parallel linears allocate with torch.empty (no init).
|
||||
for param in attn.parameters():
|
||||
param.normal_(std=0.2)
|
||||
x = torch.randn(tokens, channels)
|
||||
rope = torch.randn(tokens, 1, head_dim // 2, 2, 2)
|
||||
hidden = x.unsqueeze(0) if batched else x
|
||||
|
||||
with torch.no_grad():
|
||||
out = attn(hidden, rotary_emb=rope)
|
||||
|
||||
heads = channels // head_dim
|
||||
q = attn.query_norm(attn.to_query(x).reshape(tokens, heads, head_dim))
|
||||
k = attn.key_norm(attn.to_key(x).reshape(tokens, heads, head_dim))
|
||||
v = attn.to_value(x).reshape(tokens, heads, head_dim)
|
||||
q = apply_rotary(q, rope).type_as(q)
|
||||
k = apply_rotary(k, rope).type_as(k)
|
||||
ref = torch.nn.functional.scaled_dot_product_attention(
|
||||
q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1)
|
||||
).transpose(0, 1)
|
||||
expected = attn.out_layer(ref.reshape(tokens, channels))
|
||||
|
||||
assert out.shape == (hidden.shape[:-1] + (channels,))
|
||||
torch.testing.assert_close(out.reshape(tokens, channels), expected, rtol=1e-4, atol=1e-5)
|
||||
|
||||
|
||||
def test_modulation_projections_are_zero_initialized():
|
||||
"""Modulation layers are zero-initialized at construction (standard
|
||||
AdaLN-zero init) — confirms the zero-init in Kandinsky6Modulation.__init__
|
||||
reaches the tensor-parallel out_layer correctly (it's a
|
||||
ColumnParallelLinear, not plain nn.Linear)."""
|
||||
model = _build_transformer(_TINY_T2V_CONFIG)
|
||||
for block in model.text_transformer_blocks:
|
||||
assert torch.all(block.text_modulation.out_layer.weight == 0)
|
||||
for block in model.visual_transformer_blocks:
|
||||
assert torch.all(block.visual_modulation.out_layer.weight == 0)
|
||||
|
||||
|
||||
def test_dit_constructs_from_a_diffusers_config_dict():
|
||||
from vllm_omni.diffusion.config import set_current_diffusion_config
|
||||
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6Transformer3DModel
|
||||
|
||||
with set_current_diffusion_config(_sdpa_config()):
|
||||
model = Kandinsky6Transformer3DModel.from_diffusers_config(
|
||||
{**_TINY_T2VA_CONFIG, "patch_size": [1, 2, 2], "axes_dims": [4, 4, 4]}
|
||||
)
|
||||
assert tuple(model.patch_size) == (1, 2, 2)
|
||||
assert model._repeated_blocks == [
|
||||
"Kandinsky6FusedTransformerDecoderBlock",
|
||||
"Kandinsky6TransformerEncoderBlock",
|
||||
"Kandinsky6TransformerDecoderBlock",
|
||||
]
|
||||
visual = model.visual_transformer_blocks[0].video_dec_block.self_attention.attn
|
||||
audio = model.visual_transformer_blocks[0].audio_dec_block.self_attention.attn
|
||||
text = model.video_text_transformer_blocks[0].self_attention.attn
|
||||
cross = model.visual_transformer_blocks[0].va_cross_attention.attn
|
||||
assert visual.role == "kandinsky6.visual_self"
|
||||
assert visual.skip_sequence_parallel is False
|
||||
assert audio.skip_sequence_parallel is True
|
||||
assert text.role == "kandinsky6.text_self"
|
||||
assert text.skip_sequence_parallel is True
|
||||
assert cross.role == "kandinsky6.video_audio_cross"
|
||||
assert cross.skip_sequence_parallel is True
|
||||
|
||||
|
||||
def test_cfg_combine_matches_apply_cfg_without_normalization():
|
||||
from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin
|
||||
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import apply_cfg
|
||||
|
||||
positive = torch.tensor([1.0, 3.0])
|
||||
negative = torch.tensor([0.25, -1.0])
|
||||
combined = CFGParallelMixin.combine_cfg_noise(object(), positive, negative, 5.0, False)
|
||||
torch.testing.assert_close(combined, apply_cfg(positive, negative, 5.0))
|
||||
@@ -0,0 +1,118 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Kandinsky 6 VAE patch parallel keeps the single-GPU tile grid."""
|
||||
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
import torch.multiprocessing as mp
|
||||
|
||||
from vllm_omni.diffusion.models.kandinsky6.modeling_kandinsky6_vae import AutoencoderKLHunyuanVideo
|
||||
|
||||
|
||||
def _stub_vae() -> AutoencoderKLHunyuanVideo:
|
||||
vae = AutoencoderKLHunyuanVideo.__new__(AutoencoderKLHunyuanVideo)
|
||||
vae.spatial_compression_ratio = 8
|
||||
vae.temporal_compression_ratio = 4
|
||||
vae.use_tiling = True
|
||||
vae.use_framewise_decoding = True
|
||||
vae.tile_sample_min_height = 32
|
||||
vae.tile_sample_min_width = 32
|
||||
vae.tile_sample_min_num_frames = 16
|
||||
vae.tile_sample_stride_height = 24
|
||||
vae.tile_sample_stride_width = 24
|
||||
vae.tile_sample_stride_num_frames = 12
|
||||
vae.tile_size = None
|
||||
vae.distributed_executor = None
|
||||
vae.post_quant_conv = lambda tensor: tensor
|
||||
vae._seen_widths = []
|
||||
|
||||
def decoder(tensor: torch.Tensor) -> torch.Tensor:
|
||||
vae._seen_widths.append(int(tensor.shape[-1]))
|
||||
return tensor + 1
|
||||
|
||||
vae.decoder = decoder
|
||||
return vae
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_distributed_decode_tiles_match_local_tiled_decode():
|
||||
vae = _stub_vae()
|
||||
latent = torch.arange(1 * 4 * 2 * 8 * 12, dtype=torch.float32).reshape(1, 4, 2, 8, 12)
|
||||
|
||||
local = vae.tiled_decode(latent).sample
|
||||
tasks, spec = vae._decode_tile_split(latent)
|
||||
assert len(tasks) > 1
|
||||
assert max(int(task.tensor.shape[-1]) for task in tasks) < latent.shape[-1]
|
||||
|
||||
decoded = {task.grid_coord: vae._decode_tile_exec(task) for task in tasks}
|
||||
merged = vae._decode_tile_merge(decoded, spec)
|
||||
|
||||
torch.testing.assert_close(merged, local)
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_decode_still_applies_optimal_tiling_when_patch_parallel_is_on():
|
||||
vae = _stub_vae()
|
||||
seen = {}
|
||||
|
||||
def tiling(shape, device=None):
|
||||
del device
|
||||
seen["shape"] = tuple(shape)
|
||||
return (1, 17, 32, 32), (8, 24, 24)
|
||||
|
||||
from diffusers.models.autoencoders.vae import DecoderOutput
|
||||
|
||||
vae.get_dec_optimal_tiling = tiling
|
||||
vae._decode = lambda z, return_dict=True: DecoderOutput(sample=z)
|
||||
vae.distributed_executor = SimpleNamespace(parallel_size=2)
|
||||
vae.is_distributed_enabled = lambda: True
|
||||
|
||||
out = vae.decode(torch.zeros(1, 16, 8, 8, 12))
|
||||
|
||||
assert seen["shape"] == (1, 16, 8, 8, 12)
|
||||
assert tuple(out.sample.shape) == (1, 16, 8, 8, 12)
|
||||
|
||||
|
||||
def _patch_parallel_worker(rank: int, port: str, shapes: dict) -> None:
|
||||
os.environ["MASTER_ADDR"] = "127.0.0.1"
|
||||
os.environ["MASTER_PORT"] = port
|
||||
os.environ["RANK"] = str(rank)
|
||||
os.environ["WORLD_SIZE"] = "2"
|
||||
dist.init_process_group("gloo", rank=rank, world_size=2)
|
||||
try:
|
||||
import vllm_omni.diffusion.distributed.autoencoders.distributed_vae_executor as executor_mod
|
||||
|
||||
world = SimpleNamespace(device_group=dist.group.WORLD)
|
||||
executor_mod.get_world_group = lambda: world
|
||||
|
||||
vae = _stub_vae()
|
||||
latent = torch.arange(1 * 4 * 2 * 8 * 12, dtype=torch.float32).reshape(1, 4, 2, 8, 12)
|
||||
reference = vae.tiled_decode(latent).sample.clone()
|
||||
vae._seen_widths.clear()
|
||||
vae.set_parallel_size(2, mode="tile")
|
||||
decoded = vae.tiled_decode(latent).sample
|
||||
torch.testing.assert_close(decoded, reference)
|
||||
if rank == 0:
|
||||
shapes["widths"] = list(vae._seen_widths)
|
||||
shapes["full_width"] = int(latent.shape[-1])
|
||||
finally:
|
||||
dist.destroy_process_group()
|
||||
|
||||
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.cpu
|
||||
def test_patch_parallel_decode_splits_tiles_across_ranks():
|
||||
manager = mp.Manager()
|
||||
shapes = manager.dict()
|
||||
port = str(29500 + (os.getpid() % 1000))
|
||||
mp.spawn(_patch_parallel_worker, args=(port, shapes), nprocs=2, join=True)
|
||||
|
||||
assert shapes["widths"]
|
||||
assert max(shapes["widths"]) < shapes["full_width"]
|
||||
@@ -0,0 +1,79 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Offline smoke for Kandinsky 6 TI2VA.
|
||||
|
||||
Short request (256x256, 5 frames, 2 steps) so it finishes inside the default
|
||||
online poll window. Pro geometry (480x864, 125 frames, 50 steps) stays the
|
||||
serving default, not this smoke.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.helpers.mark import hardware_test
|
||||
from tests.helpers.runtime import OfflineOmniClient, get_model_prefix
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
_HUB_MODEL = "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers"
|
||||
_LOCAL_BUNDLE = Path(__file__).resolve().parents[4] / "kandinsky6_bundle"
|
||||
|
||||
|
||||
def _resolve_model() -> str:
|
||||
env = os.environ.get("KANDINSKY6_MODEL")
|
||||
if env:
|
||||
return env
|
||||
if _LOCAL_BUNDLE.is_dir():
|
||||
return str(_LOCAL_BUNDLE)
|
||||
return _HUB_MODEL
|
||||
|
||||
|
||||
def _checkpoint_on_disk(model: str) -> bool:
|
||||
if Path(model).is_dir():
|
||||
return True
|
||||
prefix = get_model_prefix()
|
||||
return bool(prefix) and Path(prefix + model).is_dir()
|
||||
|
||||
|
||||
MODEL = _resolve_model()
|
||||
PROMPT = "A golden retriever runs along a sunny beach, waves crashing."
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.skipif(
|
||||
not _checkpoint_on_disk(MODEL),
|
||||
reason=(
|
||||
"Kandinsky 6 Diffusers weights are gated. Set KANDINSKY6_MODEL or "
|
||||
"MODEL_PREFIX to a local copy of "
|
||||
"kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers."
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards={"cuda": 1})
|
||||
@pytest.mark.parametrize(
|
||||
"omni_runner",
|
||||
[(MODEL, None, {"enable_cpu_offload": True, "enforce_eager": True})],
|
||||
indirect=True,
|
||||
)
|
||||
def test_text_to_video_and_audio_001(offline_client: OfflineOmniClient):
|
||||
sampling = OmniDiffusionSamplingParams(
|
||||
height=256,
|
||||
width=256,
|
||||
num_frames=5,
|
||||
fps=24,
|
||||
num_inference_steps=2,
|
||||
guidance_scale=5.0,
|
||||
seed=42,
|
||||
extra_args={"sample_audio": True},
|
||||
)
|
||||
request_config = {
|
||||
"model": MODEL,
|
||||
"prompt": PROMPT,
|
||||
"sampling_params": sampling,
|
||||
}
|
||||
offline_client.send_diffusion_request(request_config)
|
||||
@@ -0,0 +1,89 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Online smoke for Kandinsky 6 TI2VA via ``/v1/videos``.
|
||||
|
||||
Short geometry so the job finishes inside the helper's default 300 s poll.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.helpers.mark import hardware_marks
|
||||
from tests.helpers.runtime import OmniServer, OmniServerParams, OnlineOmniClient, get_model_prefix
|
||||
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
|
||||
_HUB_MODEL = "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers"
|
||||
_LOCAL_BUNDLE = Path(__file__).resolve().parents[4] / "kandinsky6_bundle"
|
||||
|
||||
|
||||
def _resolve_model() -> str:
|
||||
env = os.environ.get("KANDINSKY6_MODEL")
|
||||
if env:
|
||||
return env
|
||||
if _LOCAL_BUNDLE.is_dir():
|
||||
return str(_LOCAL_BUNDLE)
|
||||
return _HUB_MODEL
|
||||
|
||||
|
||||
def _checkpoint_on_disk(model: str) -> bool:
|
||||
if Path(model).is_dir():
|
||||
return True
|
||||
prefix = get_model_prefix()
|
||||
return bool(prefix) and Path(prefix + model).is_dir()
|
||||
|
||||
|
||||
MODEL = _resolve_model()
|
||||
PROMPT = "A golden retriever runs along a sunny beach, waves crashing."
|
||||
|
||||
pytestmark = [
|
||||
pytest.mark.skipif(
|
||||
not _checkpoint_on_disk(MODEL),
|
||||
reason=(
|
||||
"Kandinsky 6 Diffusers weights are gated. Set KANDINSKY6_MODEL or "
|
||||
"MODEL_PREFIX to a local copy of "
|
||||
"kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers."
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
SINGLE_CARD_FEATURE_MARKS = hardware_marks(res={"cuda": "H100"})
|
||||
|
||||
|
||||
def _get_diffusion_feature_cases(model: str):
|
||||
return [
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=model,
|
||||
server_args=["--enable-cpu-offload", "--enforce-eager"],
|
||||
init_timeout=3600,
|
||||
stage_init_timeout=3600,
|
||||
),
|
||||
id="cpu_offload",
|
||||
marks=SINGLE_CARD_FEATURE_MARKS,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parametrize("omni_server", _get_diffusion_feature_cases(MODEL), indirect=True)
|
||||
def test_text_to_video_and_audio_001(omni_server: OmniServer, online_client: OnlineOmniClient) -> None:
|
||||
"""Short TI2VA job completes and returns an MP4."""
|
||||
request_config = {
|
||||
"model": omni_server.model,
|
||||
"form_data": {
|
||||
"prompt": PROMPT,
|
||||
"height": 256,
|
||||
"width": 256,
|
||||
"num_frames": 5,
|
||||
"fps": 24,
|
||||
"num_inference_steps": 2,
|
||||
"guidance_scale": 5.0,
|
||||
"seed": 42,
|
||||
},
|
||||
}
|
||||
online_client.send_video_diffusion_request(request_config)
|
||||
@@ -999,6 +999,38 @@ def test_magi2_serving_applies_native_defaults_and_rejects_explicit_frame_mismat
|
||||
)
|
||||
|
||||
|
||||
def test_kandinsky6_serving_applies_defaults_but_allows_explicit_frame_count():
|
||||
"""Kandinsky 6 publishes Pro-geometry defaults (480x864, 125 frames, 50
|
||||
steps) but is not a fixed-duration model: an explicit ``num_frames``
|
||||
must be honoured rather than rejected."""
|
||||
engine = FakeAsyncOmni()
|
||||
engine.model_class_name = "Kandinsky6TI2VAPipeline"
|
||||
handler = OmniOpenAIServingVideo.for_diffusion(
|
||||
diffusion_engine=engine,
|
||||
model_name="kandinsky6-bundle",
|
||||
)
|
||||
|
||||
asyncio.run(handler._run_and_extract(VideoGenerationRequest(prompt="A dog runs on a beach"), "defaults"))
|
||||
sampling = engine.captured_sampling_params_list[0]
|
||||
assert (sampling.width, sampling.height) == (864, 480)
|
||||
assert sampling.num_frames == 125
|
||||
assert sampling.num_inference_steps == 50
|
||||
assert sampling.fps == 24.0
|
||||
|
||||
asyncio.run(
|
||||
handler._run_and_extract(
|
||||
VideoGenerationRequest(
|
||||
prompt="A dog runs on a beach", width=512, height=320, num_frames=25, num_inference_steps=10
|
||||
),
|
||||
"explicit",
|
||||
)
|
||||
)
|
||||
sampling = engine.captured_sampling_params_list[-1]
|
||||
assert (sampling.width, sampling.height) == (512, 320)
|
||||
assert sampling.num_frames == 25
|
||||
assert sampling.num_inference_steps == 10
|
||||
|
||||
|
||||
def test_i2v_video_generation_with_image_reference_form(test_client, mocker: MockerFixture):
|
||||
mocker.patch(
|
||||
"vllm_omni.entrypoints.openai.serving_video._encode_video_bytes",
|
||||
|
||||
@@ -76,6 +76,9 @@ EXCLUDED_MODELS = [
|
||||
# The 270+ GiB multimodal checkpoint needs 4/8-way native sequence/head
|
||||
# parallel execution; covered by focused native contract and GPU tests.
|
||||
"Magi2Pipeline",
|
||||
# Pro DiT is ~56 GiB bf16 and the Hub checkpoint is gated. Covered by
|
||||
# tests/diffusion/models/kandinsky6 and the real-checkpoint E2E smoke.
|
||||
"Kandinsky6TI2VAPipeline",
|
||||
"OmniVoicePipeline",
|
||||
"OmniVoice",
|
||||
"Cosmos3OmniDiffusersPipeline",
|
||||
|
||||
@@ -908,6 +908,35 @@ def enable_cache_for_mammothmoda2(pipeline: Any, cache_config: Any) -> CacheDiTE
|
||||
return CacheDiTEnableResult(refresh=refresh, targets=(block_adapter,))
|
||||
|
||||
|
||||
def enable_cache_for_kandinsky6(pipeline: Any, cache_config: Any) -> RefreshCacheContextFunc:
|
||||
"""Cache-DiT over K6 visual blocks only.
|
||||
|
||||
Text and audio block lists are not one residual chain with the visual
|
||||
stack. If the block adapter rejects the fused ``(video, audio)`` return,
|
||||
fall back to the same step-level velocity cache MagCache uses.
|
||||
"""
|
||||
transformer = pipeline.transformer
|
||||
block_adapter = BlockAdapter(
|
||||
transformer=transformer,
|
||||
blocks=[transformer.visual_transformer_blocks],
|
||||
forward_pattern=[ForwardPattern.Pattern_0],
|
||||
has_separate_cfg=True,
|
||||
check_forward_pattern=False,
|
||||
)
|
||||
try:
|
||||
return enable_cache_for_dit(pipeline, cache_config, block_adapter)
|
||||
except Exception as exc:
|
||||
logger.warning("Cache-DiT block adapter failed for Kandinsky 6 (%s); using step cache", exc)
|
||||
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_mag_cache
|
||||
|
||||
attach_mag_cache(transformer, threshold=0.24, max_skip_steps=3, retention_ratio=0.1)
|
||||
|
||||
def _refresh(*_args, **_kwargs):
|
||||
return None
|
||||
|
||||
return _refresh
|
||||
|
||||
|
||||
def register_custom_dit_enablers() -> None:
|
||||
"""Register model-specific Cache-DiT enablers.
|
||||
|
||||
@@ -925,6 +954,7 @@ def register_custom_dit_enablers() -> None:
|
||||
"Cosmos3OmniDiffusersPipeline": enable_cache_for_cosmos3,
|
||||
"Cosmos3OmniPipeline": enable_cache_for_cosmos3,
|
||||
"Krea2Pipeline": enable_cache_for_krea2,
|
||||
"Kandinsky6TI2VAPipeline": enable_cache_for_kandinsky6,
|
||||
"Magi2Pipeline": enable_cache_for_magi2,
|
||||
"MammothModa2DiTPipeline": enable_cache_for_mammothmoda2,
|
||||
}
|
||||
|
||||
+14
-1
@@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""
|
||||
MagCache backend implementation.
|
||||
@@ -73,6 +73,19 @@ class MagCacheBackend(CacheBackend):
|
||||
transformer = pipeline.transformer
|
||||
transformer_type = transformer.__class__.__name__
|
||||
|
||||
if transformer_type == "Kandinsky6Transformer3DModel":
|
||||
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_mag_cache
|
||||
|
||||
attach_mag_cache(
|
||||
transformer,
|
||||
threshold=self.config.mag_threshold,
|
||||
max_skip_steps=self.config.mag_max_skip_steps,
|
||||
retention_ratio=self.config.mag_retention_ratio,
|
||||
)
|
||||
self._registered = True
|
||||
logger.info("MagCache: Kandinsky 6 step-cache attached (visual/audio velocity reuse)")
|
||||
return
|
||||
|
||||
num_inference_steps = self.config.num_inference_steps or 28
|
||||
|
||||
mag_ratios = self.config.mag_ratios
|
||||
|
||||
+4
-1
@@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""
|
||||
Hook-based MagCache implementation for vLLM-Omni.
|
||||
@@ -543,9 +543,12 @@ def apply_mag_cache_hook(
|
||||
state_manager = StateManager(MagCacheState, (), {})
|
||||
remaining_blocks = []
|
||||
|
||||
allowed = getattr(module, "_magcache_block_attrs", None)
|
||||
for name, submodule in module.named_children():
|
||||
if not isinstance(submodule, torch.nn.ModuleList):
|
||||
continue
|
||||
if allowed is not None and name not in allowed:
|
||||
continue
|
||||
for index, block in enumerate(submodule):
|
||||
remaining_blocks.append((f"{name}.{index}", block))
|
||||
|
||||
|
||||
+20
-1
@@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""
|
||||
TeaCache backend implementation.
|
||||
@@ -143,11 +143,30 @@ def enable_flux2_klein_teacache(pipeline: Any, config: DiffusionCacheConfig) ->
|
||||
)
|
||||
|
||||
|
||||
def enable_kandinsky6_teacache(pipeline: Any, config: DiffusionCacheConfig) -> None:
|
||||
"""TeaCache for Kandinsky 6 via the denoise-loop step cache.
|
||||
|
||||
Coefficients are uncalibrated (see ``K6_TEACACHE_COEFFICIENTS``).
|
||||
"""
|
||||
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_tea_cache
|
||||
|
||||
attach_tea_cache(
|
||||
pipeline.transformer,
|
||||
rel_l1_thresh=config.rel_l1_thresh if config.rel_l1_thresh is not None else 0.2,
|
||||
coefficients=tuple(config.coefficients) if config.coefficients is not None else None,
|
||||
)
|
||||
logger.info(
|
||||
"TeaCache: Kandinsky 6 step-cache attached (uncalibrated coefficients, thresh=%s)",
|
||||
config.rel_l1_thresh,
|
||||
)
|
||||
|
||||
|
||||
CUSTOM_TEACACHE_ENABLERS = {
|
||||
"BagelPipeline": enable_bagel_teacache,
|
||||
"Flux2KleinPipeline": enable_flux2_klein_teacache,
|
||||
"HunyuanImage3Pipeline": enable_hunyuan_image3_teacache,
|
||||
"MammothModa2DiTPipeline": enable_mammoth_moda2_teacache,
|
||||
"Kandinsky6TI2VAPipeline": enable_kandinsky6_teacache,
|
||||
"MiniMaxH3Pipeline": enable_minimax_h3_teacache,
|
||||
"SenseNovaU1Pipeline": enable_sensenova_u1_teacache,
|
||||
}
|
||||
|
||||
+10
-1
@@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
@@ -8,6 +8,15 @@ from dataclasses import dataclass
|
||||
# Source: TeaCache paper and ComfyUI-TeaCache empirical tuning
|
||||
_MODEL_COEFFICIENTS = {
|
||||
# FLUX transformer coefficients from TeaCache paper
|
||||
# Uncalibrated stand-in so Kandinsky 6 TeaCache can be enabled.
|
||||
# Copied from Qwen-Image; refit before treating skips as quality-neutral.
|
||||
"Kandinsky6Transformer3DModel": [
|
||||
-4.50000000e02,
|
||||
2.80000000e02,
|
||||
-4.50000000e01,
|
||||
3.20000000e00,
|
||||
-2.00000000e-02,
|
||||
],
|
||||
"FluxTransformer2DModel": [
|
||||
4.98651651e02,
|
||||
-2.83781631e02,
|
||||
|
||||
@@ -81,6 +81,15 @@ _DIFFUSION_MODEL_METADATA: dict[str, DiffusionModelMetadata] = {
|
||||
max_multimodal_image_inputs=1,
|
||||
final_output_type="video",
|
||||
),
|
||||
# Joint text/image-to-video-and-audio, same shape as MiniMaxH3Pipeline
|
||||
# above (an MP4 with both tracks) — declared "video" for the same reason:
|
||||
# the final container is a video file, so /v1/videos* is the right API
|
||||
# surface even though the model also produces audio.
|
||||
"Kandinsky6TI2VAPipeline": DiffusionModelMetadata(
|
||||
supports_multimodal_inputs=True,
|
||||
max_multimodal_image_inputs=1,
|
||||
final_output_type="video",
|
||||
),
|
||||
"WanPipeline": DiffusionModelMetadata(
|
||||
attention_mask_free=True,
|
||||
final_output_type="video",
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from .kandinsky6_transformer import Kandinsky6Transformer3DModel
|
||||
from .modeling_kandinsky6_audio import Kandinsky6AudioVAE
|
||||
from .modeling_kandinsky6_vae import AutoencoderKLHunyuanVideo
|
||||
from .pipeline_kandinsky6 import (
|
||||
Kandinsky6TI2VAPipeline,
|
||||
get_kandinsky6_post_process_func,
|
||||
get_kandinsky6_pre_process_func,
|
||||
)
|
||||
from .scheduling_kandinsky6 import KandinskyFlowMatchScheduler
|
||||
|
||||
__all__ = [
|
||||
"AutoencoderKLHunyuanVideo",
|
||||
"Kandinsky6AudioVAE",
|
||||
"Kandinsky6TI2VAPipeline",
|
||||
"Kandinsky6Transformer3DModel",
|
||||
"KandinskyFlowMatchScheduler",
|
||||
"get_kandinsky6_post_process_func",
|
||||
"get_kandinsky6_pre_process_func",
|
||||
]
|
||||
@@ -0,0 +1,199 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Kandinsky 6 cache adapters.
|
||||
|
||||
Generic MagCache / TeaCache hooks assume a single residual stream named
|
||||
``hidden_states``. K6's fused visual blocks return ``(video, audio)``, so
|
||||
the denoise loop consults :class:`K6StepCache` instead and reuses the
|
||||
previous velocity when the schedule says the step can be skipped.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# Pro geometry ratios from k6_video
|
||||
# ``k6_pro_125_480_864_mCache_mOffload.yaml`` (cache.magcache.mag_ratios).
|
||||
K6_PRO_MAG_RATIOS: tuple[float, ...] = (
|
||||
1.0,
|
||||
1.0,
|
||||
0.94526286,
|
||||
0.94622554,
|
||||
1.13130749,
|
||||
1.11243134,
|
||||
1.02926022,
|
||||
1.02675114,
|
||||
1.011996,
|
||||
1.01235353,
|
||||
1.03039566,
|
||||
1.02848639,
|
||||
1.00101278,
|
||||
1.00307236,
|
||||
1.02238594,
|
||||
1.02108294,
|
||||
1.0248162,
|
||||
1.02491556,
|
||||
0.99771836,
|
||||
0.99882495,
|
||||
1.01694329,
|
||||
1.01367628,
|
||||
1.01096435,
|
||||
1.00959183,
|
||||
1.01537554,
|
||||
1.01504827,
|
||||
0.99237553,
|
||||
0.99345238,
|
||||
1.02368983,
|
||||
1.02088727,
|
||||
1.01396087,
|
||||
1.01278108,
|
||||
0.99773024,
|
||||
0.99927632,
|
||||
1.01573298,
|
||||
1.01489232,
|
||||
1.00129728,
|
||||
1.00120319,
|
||||
1.01759963,
|
||||
1.01691798,
|
||||
0.99335877,
|
||||
0.99548372,
|
||||
1.02925427,
|
||||
1.02299559,
|
||||
0.99749375,
|
||||
0.99682709,
|
||||
0.99694342,
|
||||
1.00064403,
|
||||
1.01101129,
|
||||
1.0117984,
|
||||
1.04420642,
|
||||
1.03624643,
|
||||
0.9599572,
|
||||
0.97176547,
|
||||
1.00452036,
|
||||
1.0046107,
|
||||
1.01447715,
|
||||
1.01351055,
|
||||
0.9999695,
|
||||
1.00270806,
|
||||
1.00770242,
|
||||
1.00271775,
|
||||
0.98795461,
|
||||
0.98880837,
|
||||
1.01113352,
|
||||
1.00860993,
|
||||
1.00116034,
|
||||
1.00251894,
|
||||
1.05147006,
|
||||
1.04040677,
|
||||
0.99651072,
|
||||
0.99212138,
|
||||
0.99040673,
|
||||
0.98881456,
|
||||
1.01265486,
|
||||
1.01241698,
|
||||
0.97343865,
|
||||
0.97302804,
|
||||
1.02539269,
|
||||
1.02113221,
|
||||
1.02117306,
|
||||
1.02987275,
|
||||
0.98607016,
|
||||
0.98262938,
|
||||
0.98404577,
|
||||
0.98453141,
|
||||
1.01065119,
|
||||
1.00807871,
|
||||
0.99986077,
|
||||
0.99668861,
|
||||
0.98597074,
|
||||
0.98133325,
|
||||
0.98239303,
|
||||
0.98370113,
|
||||
0.93155548,
|
||||
0.93534231,
|
||||
0.94231988,
|
||||
0.93388087,
|
||||
0.84200656,
|
||||
0.83657581,
|
||||
)
|
||||
|
||||
# Uncalibrated. Borrowed from the Qwen-Image TeaCache polynomial so the flag
|
||||
# skips steps instead of raising. Quality will not match a fitted K6 curve.
|
||||
K6_TEACACHE_COEFFICIENTS: tuple[float, ...] = (
|
||||
-4.50000000e02,
|
||||
2.80000000e02,
|
||||
-4.50000000e01,
|
||||
3.20000000e00,
|
||||
-2.00000000e-02,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class K6StepCache:
|
||||
"""Step-level skip state shared by MagCache and TeaCache."""
|
||||
|
||||
kind: str
|
||||
threshold: float
|
||||
max_skip_steps: int
|
||||
retention_ratio: float
|
||||
mag_ratios: tuple[float, ...] | None = None
|
||||
coefficients: tuple[float, ...] | None = None
|
||||
rel_l1_thresh: float = 0.2
|
||||
last_cond: object = None
|
||||
last_uncond: object = None
|
||||
accumulated_err: float = 0.0
|
||||
accumulated_ratio: float = 1.0
|
||||
accumulated_steps: int = 0
|
||||
|
||||
def should_skip(self, step_index: int) -> bool:
|
||||
if self.last_cond is None or step_index <= 0:
|
||||
return False
|
||||
if self.kind == "mag":
|
||||
ratios = self.mag_ratios or ()
|
||||
scale = float(ratios[step_index]) if step_index < len(ratios) else 1.0
|
||||
self.accumulated_ratio *= scale
|
||||
self.accumulated_steps += 1
|
||||
self.accumulated_err += abs(1.0 - self.accumulated_ratio)
|
||||
if (
|
||||
self.accumulated_err <= self.threshold
|
||||
and self.accumulated_steps <= self.max_skip_steps
|
||||
and step_index >= max(1, int(self.retention_ratio * max(step_index, 1)))
|
||||
):
|
||||
return True
|
||||
self.accumulated_ratio = 1.0
|
||||
self.accumulated_steps = 0
|
||||
self.accumulated_err = 0.0
|
||||
return False
|
||||
# TeaCache stand-in: uncalibrated polynomial of the step index.
|
||||
# The real TeaCache distance is the modulated input; this only makes
|
||||
# the flag skip steps instead of raising.
|
||||
coeff = self.coefficients or K6_TEACACHE_COEFFICIENTS
|
||||
x = float(step_index)
|
||||
predicted = sum(weight * (x ** (len(coeff) - 1 - i)) for i, weight in enumerate(coeff))
|
||||
return abs(predicted) < self.rel_l1_thresh * 1.0e4
|
||||
|
||||
def store(self, cond: object, uncond: object) -> None:
|
||||
self.last_cond = cond
|
||||
self.last_uncond = uncond
|
||||
|
||||
|
||||
def attach_mag_cache(transformer, *, threshold: float, max_skip_steps: int, retention_ratio: float) -> None:
|
||||
transformer._k6_step_cache = K6StepCache(
|
||||
kind="mag",
|
||||
threshold=threshold,
|
||||
max_skip_steps=max_skip_steps,
|
||||
retention_ratio=retention_ratio,
|
||||
mag_ratios=K6_PRO_MAG_RATIOS,
|
||||
)
|
||||
|
||||
|
||||
def attach_tea_cache(transformer, *, rel_l1_thresh: float, coefficients: tuple[float, ...] | None) -> None:
|
||||
transformer._k6_step_cache = K6StepCache(
|
||||
kind="tea",
|
||||
threshold=rel_l1_thresh if rel_l1_thresh is not None else 0.2,
|
||||
max_skip_steps=3,
|
||||
retention_ratio=0.0,
|
||||
coefficients=coefficients or K6_TEACACHE_COEFFICIENTS,
|
||||
rel_l1_thresh=rel_l1_thresh if rel_l1_thresh is not None else 0.2,
|
||||
)
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,103 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Native K6 shifted flow-matching Euler scheduler.
|
||||
|
||||
Self-contained (no ``diffusers`` dependency), matching the duck-typed
|
||||
scheduler interface (``set_timesteps`` / ``timesteps`` / ``sigmas`` /
|
||||
``step``) that other native vLLM-Omni diffusion pipelines use for their own
|
||||
per-model schedulers — see ``WanEulerScheduler`` in
|
||||
``vllm_omni.diffusion.models.wan2_2.scheduling_wan_euler`` for the sibling
|
||||
pattern this mirrors.
|
||||
|
||||
The math reproduces ``kandinsky.core.algo.flow_matching.flow_match_timesteps``
|
||||
exactly: for ``t`` linearly spaced from 1.0 down to 0.0,
|
||||
``sigma = scale * t / (1 + (scale - 1) * t)``. This is the same shifted
|
||||
rectified-flow schedule family Wan's own scheduler uses, parameterized by
|
||||
``scheduler_scale`` (K6's name for what Wan calls ``shift``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from types import SimpleNamespace
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
@dataclass
|
||||
class KandinskyFlowMatchSchedulerOutput:
|
||||
prev_sample: torch.Tensor
|
||||
|
||||
|
||||
class KandinskyFlowMatchScheduler:
|
||||
"""Stateful Euler stepper over K6's shifted flow-matching sigma schedule.
|
||||
|
||||
``timesteps`` holds the model-scale (``[0, 1000]``-ish) values fed to the
|
||||
DiT's ``time`` input; ``sigmas`` holds the same schedule unscaled
|
||||
(``num_inference_steps + 1`` entries, running 1.0 -> 0.0), needed to
|
||||
derive a step size without consuming a stateful ``step()`` call when one
|
||||
modality is stepped implicitly alongside another (see the ``vllm`` port's
|
||||
``denoise_loop`` override).
|
||||
"""
|
||||
|
||||
order = 1
|
||||
|
||||
def __init__(self, scheduler_scale: float = 1.0, device: torch.device | str = "cpu") -> None:
|
||||
self.scheduler_scale = float(scheduler_scale)
|
||||
self.device = device
|
||||
self.config = SimpleNamespace(scheduler_scale=self.scheduler_scale)
|
||||
|
||||
self._step_index: int | None = None
|
||||
self.timesteps = torch.empty(0, dtype=torch.float32)
|
||||
self.sigmas = torch.empty(0, dtype=torch.float32)
|
||||
|
||||
self.set_timesteps(num_inference_steps=1, device=self.device)
|
||||
|
||||
@property
|
||||
def step_index(self) -> int | None:
|
||||
return self._step_index
|
||||
|
||||
def set_timesteps(
|
||||
self,
|
||||
num_inference_steps: int,
|
||||
device: torch.device | str | None = None,
|
||||
**kwargs, # noqa: ARG002 - kept for scheduler API compatibility
|
||||
) -> None:
|
||||
device = device or self.device
|
||||
t = torch.linspace(1.0, 0.0, int(num_inference_steps) + 1, device=device, dtype=torch.float32)
|
||||
scale = self.scheduler_scale
|
||||
self.sigmas = scale * t / (1 + (scale - 1) * t)
|
||||
# DiT `time` input is model-scale, i.e. sigma * 1000 (matches the core
|
||||
# algo loop's `t.unsqueeze(0).expand(bs) * 1000`).
|
||||
self.timesteps = self.sigmas[:-1] * 1000.0
|
||||
self._step_index = None
|
||||
|
||||
def index_for_timestep(self, timestep: torch.Tensor) -> int:
|
||||
timestep_t = torch.as_tensor(timestep, device=self.timesteps.device, dtype=self.timesteps.dtype)
|
||||
return int(torch.argmin(torch.abs(self.timesteps - timestep_t)).item())
|
||||
|
||||
def step(
|
||||
self,
|
||||
model_output: torch.Tensor,
|
||||
timestep: torch.Tensor,
|
||||
sample: torch.Tensor,
|
||||
return_dict: bool = True,
|
||||
**kwargs, # noqa: ARG002 - kept for scheduler API compatibility
|
||||
) -> KandinskyFlowMatchSchedulerOutput | tuple[torch.Tensor]:
|
||||
if self._step_index is None:
|
||||
self._step_index = self.index_for_timestep(timestep)
|
||||
|
||||
sigma = self.sigmas[self._step_index]
|
||||
sigma_next = self.sigmas[self._step_index + 1]
|
||||
dt = (sigma_next - sigma).to(device=sample.device, dtype=sample.dtype)
|
||||
prev_sample = sample + dt * model_output
|
||||
|
||||
self._step_index += 1
|
||||
|
||||
if not return_dict:
|
||||
return (prev_sample,)
|
||||
return KandinskyFlowMatchSchedulerOutput(prev_sample=prev_sample)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return int(self.timesteps.shape[0])
|
||||
@@ -396,6 +396,11 @@ _DIFFUSION_MODELS = {
|
||||
"pipeline_krea2",
|
||||
"Krea2Pipeline",
|
||||
),
|
||||
"Kandinsky6TI2VAPipeline": (
|
||||
"kandinsky6",
|
||||
"pipeline_kandinsky6",
|
||||
"Kandinsky6TI2VAPipeline",
|
||||
),
|
||||
}
|
||||
_DIFFUSION_MODELS["MingImageLayeredDiffusionPipeline"] = _DIFFUSION_MODELS["MingImageDiffusionPipeline"]
|
||||
|
||||
@@ -710,6 +715,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
|
||||
"StableDiffusionXLPipeline": "get_sdxl_image_post_process_func",
|
||||
"Krea2Pipeline": "get_krea2_post_process_func",
|
||||
"HunyuanImage3ForCausalMM": "get_hunyuan_image3_post_process_func",
|
||||
"Kandinsky6TI2VAPipeline": "get_kandinsky6_post_process_func",
|
||||
}
|
||||
_DIFFUSION_POST_PROCESS_FUNCS["MingImageLayeredDiffusionPipeline"] = _DIFFUSION_POST_PROCESS_FUNCS[
|
||||
"MingImageDiffusionPipeline"
|
||||
@@ -756,6 +762,7 @@ _DIFFUSION_PRE_PROCESS_FUNCS = {
|
||||
"Cosmos3OmniDiffusersPipeline": "get_cosmos3_pre_process_func",
|
||||
"Cosmos3OmniPipeline": "get_cosmos3_pre_process_func",
|
||||
"MammothModa2DiTPipeline": "get_mammoth_moda2_pre_process_func",
|
||||
"Kandinsky6TI2VAPipeline": "get_kandinsky6_pre_process_func",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -158,7 +158,13 @@ def is_diffusion_model(model_name: str) -> bool:
|
||||
logger.debug("Failed to read local %s: %s", filename, e)
|
||||
|
||||
# Strategy 2: Check using vllm's utility (works for both local and remote models)
|
||||
config_dict = get_diffusion_model_index(model_name)
|
||||
# get_hf_file_to_dict can raise for local paths that don't resolve to a Hub
|
||||
# repo id (see resolve_model_class_name's identical guard for the same call).
|
||||
try:
|
||||
config_dict = get_diffusion_model_index(model_name)
|
||||
except Exception as e:
|
||||
logger.debug("Failed to fetch diffusion model index via vllm utility: %s", e)
|
||||
config_dict = None
|
||||
if config_dict is not None and config_dict.get("_class_name") and config_dict.get("_diffusers_version"):
|
||||
logger.debug("Detected diffusion model via a standard Diffusers index")
|
||||
return True
|
||||
|
||||
@@ -334,7 +334,12 @@ class OmniOpenAIServingVideo:
|
||||
)
|
||||
vp.width = vp.width or video_defaults.width
|
||||
vp.height = vp.height or video_defaults.height
|
||||
if num_frames_provided and vp.num_frames != video_defaults.num_frames:
|
||||
# Only fixed-duration pipelines (those declaring
|
||||
# ``duration_seconds``, e.g. MAGI-2) pin the frame count; models
|
||||
# that merely publish serving defaults (e.g. Kandinsky 6, any
|
||||
# 4k+1 frame count) still accept an explicit ``num_frames``.
|
||||
fixed_frame_count = video_defaults.duration_seconds is not None
|
||||
if fixed_frame_count and num_frames_provided and vp.num_frames != video_defaults.num_frames:
|
||||
raise HTTPException(
|
||||
status_code=HTTPStatus.BAD_REQUEST.value,
|
||||
detail=(f"This diffusion model requires {video_defaults.num_frames} frames; got {vp.num_frames}."),
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Request fields and example defaults for the Kandinsky 6 TI2VA pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from vllm_omni.model_extras.video_generation import VideoGenerationDefaults
|
||||
|
||||
# Forwarded to ``OmniDiffusionSamplingParams.extra_args`` by the shared examples
|
||||
# and the OpenAI video API's ``extra_body``.
|
||||
KANDINSKY6_EXTRA_BODY_PARAMS = frozenset(
|
||||
{
|
||||
# bool, default True: also sample the audio track (TI2VA). False -> video only.
|
||||
"sample_audio",
|
||||
# "pretrain" (text-only) | "tail_cond_first_frame" (reference image).
|
||||
"visual_cond_scheme",
|
||||
}
|
||||
)
|
||||
KANDINSKY6_EXTRA_OUTPUT_PARAMS: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def get_kandinsky6_video_generation_defaults(
|
||||
extra_body: Mapping[str, Any] | None = None,
|
||||
) -> VideoGenerationDefaults:
|
||||
"""Kandinsky 6 Pro production geometry: 480x864, 125 frames @ 24 fps, 50 steps, CFG 5.
|
||||
|
||||
These are defaults, not a fixed contract: any ``4k+1`` frame count and
|
||||
16-divisible resolution can be requested explicitly, so ``duration_seconds``
|
||||
is left unset (the serving layer only pins ``num_frames`` for
|
||||
fixed-duration pipelines such as MAGI-2).
|
||||
"""
|
||||
del extra_body
|
||||
return VideoGenerationDefaults(
|
||||
width=864,
|
||||
height=480,
|
||||
num_frames=125,
|
||||
num_inference_steps=50,
|
||||
fps=24.0,
|
||||
guidance_scale=5.0,
|
||||
output="kandinsky6_output.mp4",
|
||||
# ``None`` lets the pipeline apply its own tuned negative prompt.
|
||||
default_negative_prompt=None,
|
||||
duration_seconds=None,
|
||||
)
|
||||
@@ -40,6 +40,11 @@ from vllm_omni.model_extras.hunyuan_image3 import build_x_to_text_prompt as buil
|
||||
from vllm_omni.model_extras.hunyuan_image3 import (
|
||||
validate_ar_tokenizer as validate_hunyuan_image3_ar_tokenizer,
|
||||
)
|
||||
from vllm_omni.model_extras.kandinsky6 import (
|
||||
KANDINSKY6_EXTRA_BODY_PARAMS,
|
||||
KANDINSKY6_EXTRA_OUTPUT_PARAMS,
|
||||
get_kandinsky6_video_generation_defaults,
|
||||
)
|
||||
from vllm_omni.model_extras.lingbot_video import LINGBOT_VIDEO_EXTRA_BODY_PARAMS
|
||||
from vllm_omni.model_extras.ltx2 import (
|
||||
LTX_EXTRA_BODY_PARAMS,
|
||||
@@ -241,6 +246,13 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
|
||||
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
},
|
||||
"Kandinsky6TI2VAPipeline": {
|
||||
"extra_body_params": KANDINSKY6_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": KANDINSKY6_EXTRA_OUTPUT_PARAMS,
|
||||
"video_generation_defaults_builder": get_kandinsky6_video_generation_defaults,
|
||||
# forward() returns uint8 frames already; declared for completeness.
|
||||
"output_tensor_range": "zero_to_one",
|
||||
},
|
||||
"HeliosPyramidPipeline": {
|
||||
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
|
||||
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
|
||||
|
||||
Reference in new issue
Block a user