Kandinsky6 (#8537)

Signed-off-by: leffff <levnovitskiy@gmail.com>
Signed-off-by: Artem Borisov <divotionchannel@gmail.com>
Co-authored-by: leffff <levnovitskiy@gmail.com>
Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
This commit is contained in:
authored and GitHub committed 2026-10-06 15:55:15 +08:00
1 parent 1f63e97a40
commit 5f95115e70
38 files changed
+8574 -8

No files matched your search

@@ -149,6 +149,10 @@ x-lingbot-video: &lingbot_video
x-lingbot-world: &lingbot_world
- vllm_omni/diffusion/models/lingbot_world/
x-kandinsky6: &kandinsky6
- vllm_omni/diffusion/models/kandinsky6/
- vllm_omni/model_extras/kandinsky6.py
source_file_dependencies:
# Hardware presets and shared test helpers. Not a job key — do not attach this
@@ -491,6 +495,11 @@ source_file_dependencies:
- tests/dfx/perf/scripts/run_benchmark.py
- tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json
diffusion_kandinsky6_function:
- *kandinsky6
- tests/e2e/offline_inference/test_kandinsky6.py
- tests/e2e/online_serving/test_kandinsky6.py
# Invalid-param jobs: model business code only, plus the reliability scripts.
reliability_invalid_param_h100:
+9
View File
@@ -189,6 +189,15 @@ steps:
'
mirror_hardwares: h100_1
- label: "Diffusion · Kandinsky6 Test"
source_file_dependencies: diffusion_kandinsky6_function
commands:
- |
timeout 40m bash -c "
pytest -s -v tests/e2e/offline_inference/test_kandinsky6.py tests/e2e/online_serving/test_kandinsky6.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
"
mirror_hardwares: h100_1
- label: "Diffusion · Wan22 Test"
source_file_dependencies: diffusion_wan22_function
commands:
+1
View File
@@ -106,6 +106,7 @@ th {
| `SenseNovaU1Pipeline` | SenseNova-U1-A3B (MoE) | `sensenova/SenseNova-U1-A3B-MoT` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.md#a3b-moe-variant) |
| `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
| `Kandinsky6TI2VAPipeline` | Kandinsky 6 TI2VA | `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Kandinsky/Kandinsky6-TI2VA.md) |
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
+1
View File
@@ -196,6 +196,7 @@ The following tables show which models support each feature:
| **SANA-Video-2B T2V I2V** | ❌ | ❌ | ✅ (`--usp`, frame-sharded) | ✅ | ✅ (TP=2 only) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **Kandinsky 6 TI2VA** | ✅* (uncalibrated) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ (FP8) | ✅ (1 request) |
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
@@ -1,7 +1,7 @@
# Image-To-Video
This shared example generates videos from images with VACE, Wan2.2, LTX-2,
HunyuanVideo-1.5, SANA-Video, Cosmos3, MAGI-2, and other compatible pipelines.
HunyuanVideo-1.5, SANA-Video, Cosmos3, MAGI-2, Kandinsky 6, and other compatible pipelines.
- `image_to_video.py`: command-line script for single video generation with advanced options.
@@ -30,6 +30,7 @@ This folder provides a unified CLI script for image-to-video generation using vL
| `sand-ai/MAGI-2-preview` | 512 x 896 | 125 | 100 | Model-fixed | Native four-GPU TP/SP; resident SP4 default; DLO available |
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | 480 x 832 | 81 | 50 | 6.0 | Native `SanaImageToVideoPipeline`; Wan VAE |
| `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | 704 x 1280 | 81 | 50 | 6.0 | Native `SanaImageToVideoPipeline`; LTX-2 Video VAE |
| `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | 480 x 864 | 125 | 50 | 5.0 | ~75 GiB reserved (H100, CPU offload); gated Hub repo |
MAGI-2 native Preview setup, four-GPU topology and DLO choices, I2VA commands,
and eight-GPU validation status are documented in the
@@ -14,6 +14,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
| `nvidia/Cosmos3-Nano` | 720x1280 | 189 | 35 | 6.0 | ~46 GiB (peak, 720p) |
| `BestWishYsh/Helios-Base` / `Helios-Mid` / `Helios-Distilled` | 384x640 | 99 | 50 | 5.0 / 5.0 / 1.0 | — |
| `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | 480x864 | 125 | 50 | 5.0 | ~75 GiB reserved (H100, CPU offload, SDPA) |
| `sand-ai/MAGI-2-preview` | 512x896 | 125 | 100 | Model-fixed | Native four-GPU TP/SP; resident SP4 default; DLO available |
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | 480x832 | 81 | 50 | 6.0 | BF16 DiT + FP32 Wan VAE wrapper |
| `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | 704x1280 | 81 | 50 | 6.0 | BF16 DiT + LTX-2 Video VAE wrapper |
@@ -249,6 +250,24 @@ python text_to_video.py \
> conditioning tensors that cannot be passed through the JSON `--extra-body`
> flag; they are out of scope for this text-to-video example.
### Kandinsky 6 (TI2VA)
The Hub checkpoint is gated; export `HF_TOKEN` if download returns 401.
`--enable-cpu-offload` is required on an 80 GB H100. Audio is on by default.
```bash
python text_to_video.py \
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
--enable-cpu-offload \
--output kandinsky6_t2va.mp4
```
Video only: add `--extra-body '{"sample_audio": false}'`. Image-to-video uses
the shared [`image_to_video.py`](../image_to_video/README.md) example with
`--image`. Serving numbers and flags are in the
[`Kandinsky 6 TI2VA recipe`](../../../recipes/Kandinsky/Kandinsky6-TI2VA.md).
## Key Arguments
### Common
@@ -235,7 +235,7 @@ def parse_extra_body(value: str) -> dict[str, Any]:
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Generate a video from a text prompt. "
"Supports Wan2.2, HunyuanVideo-1.5, Helios, LingBot-Video, MAGI-2, "
"Supports Wan2.2, HunyuanVideo-1.5, Helios, LingBot-Video, MAGI-2, Kandinsky 6, "
"and other text-to-video models."
)
parser.add_argument(
+12
View File
@@ -232,6 +232,11 @@ ignore = [
# F-correctness rules still apply.
"vllm_omni/model_executor/models/nemotron_voicechat/nemo_vendored/**" = ["E501", "E721", "E741", "F401", "F841", "I001", "N803", "N814", "UP006", "UP007", "UP008", "UP011", "UP032", "UP035", "W291", "W293"]
"vllm_omni/experimental/fullduplex/joyvl/decision/prompts.py" = ["E501"] # System/summarizer prompts are long prose lines
# Kandinsky 6 keeps native/BigVGAN names (T/H/W, `l`) and NVIDIA vocoder
# docstring length so the port stays diffable against the source modules.
"vllm_omni/diffusion/models/kandinsky6/kandinsky6_transformer.py" = ["N803", "E741"]
"vllm_omni/diffusion/models/kandinsky6/pipeline_kandinsky6.py" = ["N803"]
"vllm_omni/diffusion/models/kandinsky6/modeling_kandinsky6_audio.py" = ["E501", "E741", "N803", "UP008"]
[tool.ruff.lint.flake8-tidy-imports.banned-api]
"librosa".msg = "The librosa module is banned, use vllm.multimodal helpers instead"
@@ -364,6 +369,8 @@ extend-ignore-identifiers-re = [
# (output feature dim), not "doubt".
".*dout.*",
".*DOUT.*",
# Kandinsky nabla threshold argument (`thr`), not the word "the".
"^thr$",
]
extend-ignore-re = [
"\\.\\.\\.n,d->\\.\\.\\.nd",
@@ -404,3 +411,8 @@ IY = "IY"
# Speech DAC codec identifiers (convnet_type, transconvnet_type) and is not a
# typo of "convent".
convnet = "convnet"
# Upstream Kandinsky system prompt. "promt" and "scren" are part of the
# conditioning text and must not be corrected.
thr = "thr"
promt = "promt"
scren = "scren"
+247
View File
@@ -0,0 +1,247 @@
# Kandinsky 6 TI2VA — H100 (v1, single GPU, CPU offload)
> Text/image-to-video-and-audio serving with Kandinsky 6 Pro
## Summary
- Vendor: Kandinsky Lab ([`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers))
- Model: Kandinsky 6 Pro (TI2VA — text/image-to-video-and-audio)
- Task: Joint text/image-to-video-and-audio generation
- Mode: Offline (`Omni(...)`) and online serving with the OpenAI-compatible
`/v1/videos` API
- Hardware: 1x NVIDIA H100 80GB HBM3 with component-level CPU offload
- Maintainer: Community
## When to use this recipe
Use this recipe to serve Kandinsky 6's TI2VA pipeline — a single DiT that
jointly denoises video and audio latents, conditioned on a text prompt and,
optionally, a reference image (image-to-video). Tensor parallel, CFG
parallel, Ulysses, ring, pipeline parallel, HSDP, layerwise offload,
MagCache, TeaCache, Cache-DiT, FP8, and tiled VAE patch parallel are wired.
NaviCache and a registered NABLA backend are not. TeaCache coefficients are
uncalibrated. See "Known limitations".
## Supported model contract
| Task | Entrypoint | Input | Output |
|---|---|---|---|
| Text-to-video-and-audio | `vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni` / `Omni(model="kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers")` | text prompt | synchronized MP4 (H.264 video + AAC 44.1 kHz mono audio) |
| Image-to-video-and-audio | same | text prompt + reference image | synchronized MP4, first frame conditioned on the reference image |
- Audio generation is on by default (`sample_audio=True`); pass
`extra_args.sample_audio=false` to generate video only.
- Batch size is 1 request per generation call (upstream limitation of the
ported denoise loop; concurrent requests are still served by vLLM-Omni's
own request scheduler, just not batched together within one DiT forward
pass yet).
- Serving defaults follow the Pro production geometry registered in
`vllm_omni/model_extras/kandinsky6.py`: **864x480, 125 frames @ 24 fps
(5.2 s), 50 steps, CFG 5.0**. Any `4k+1` frame count and 16-divisible
resolution can be requested explicitly via `height`/`width`/`num_frames`/
`num_inference_steps` — the model is *not* fixed-duration.
- Audio is always 44.1 kHz; its length is derived from the video duration.
- Negative prompt defaults to the pipeline's built-in K6 negative prompt
when omitted.
## References
- Hub checkpoint: [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
- Joint video+audio native-model reference: [`recipes/MiniMaxAI/MiniMax-H3.md`](../MiniMaxAI/MiniMax-H3.md)
- Offline example: [`examples/offline_inference/text_to_video/text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py)
- Image-to-video example: [`examples/offline_inference/image_to_video/image_to_video.py`](../../examples/offline_inference/image_to_video/image_to_video.py)
## Hardware
- Accelerator model and per-device memory: NVIDIA H100 80GB HBM3
(driver 570.133.20)
- Number of devices: 1 (multi-GPU tensor parallelism is implemented in the
transformer's linear layers but not yet validated end-to-end)
- Device interconnect: N/A (single-device path)
- Host memory: 1.4 TB installed; the bf16 DiT (~56 GB) plus the text
encoders are staged on the host under `--enable-cpu-offload`, so budget
at least ~90 GB of free host RAM.
- Qualification scope: single-GPU T2VA at the Pro default geometry and at a
reduced smoke geometry, offline and via `vllm serve`. Numbers below were
measured once on the shared box described above; treat them as
indicative, not as a benchmark.
## Software environment
- OS: Ubuntu 22.04.5 LTS
- Python: 3.12.13
- Driver / runtime: CUDA 12.9 (torch 2.13.0+cu129)
- vLLM version: 0.29.0
- vLLM-Omni version or commit: this repository, `vllm_omni/diffusion/models/kandinsky6/`
- diffusers 0.40.0, transformers 5.14.1
- Attention: FlashAttention-3 on this box (`flash_attn_interface`). The Pro
timings below use it. Without FA3, `attention_engine: "auto"` falls back
to PyTorch SDPA; an earlier SDPA Pro run on the same GPU was 26.9 s/it
and 1377 s.
## Weights
`Kandinsky6TI2VAPipeline._load_components` loads the public Diffusers layout
from [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
(`model_index.json`, `transformer/`, `vae/`, `text_encoder/`, `tokenizer/`,
`text_encoder_2/`, `tokenizer_2/`, `audio_vae/`, `scheduler/`). Pass that
repo id (or a local snapshot of it) to `vllm serve` / `Omni(model=...)`.
The Hub repo is gated; export `HF_TOKEN` if download returns 401.
A raw Kandinsky 6 weight tree is not required. If you already have a local
Diffusers-layout snapshot, point `--model` at that directory instead of the
Hub id.
Each weight folder is loaded on its own (`transformer/`, `vae/`, `text_encoder/`, `text_encoder_2/`, `audio_vae/`). The DiT is built with initialization skipped, then the framework loader fills it from `transformer/`.
## Command
```bash
vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni \
--host 127.0.0.1 --port 8091 \
--num-gpus 1 --enable-cpu-offload
```
`--enable-cpu-offload` enables model-level offload with mutual exclusion
between the transformer and the two text encoders (they never co-reside on
the GPU). Without it the bf16 DiT (~56 GB) plus Qwen2.5-VL-7B (~16 GB) plus
the VAEs and activations do not fit an 80 GB device at the Pro geometry.
Offline (writes an MP4 with the audio track muxed in):
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
--seed 42 --enable-cpu-offload --output kandinsky6_t2va.mp4
# Video only: add --extra-body '{"sample_audio": false}'
```
Image-to-video-and-audio uses the shared image-to-video example:
```bash
python examples/offline_inference/image_to_video/image_to_video.py \
--model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
--image first_frame.png \
--prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
--seed 42 --enable-cpu-offload --output kandinsky6_i2va.mp4
```
The generic scripts pick up the Pro defaults from
`vllm_omni/model_extras/kandinsky6.py`.
## Verification
`/v1/videos` takes multipart form fields (not a JSON body). Submit, poll,
then download:
```bash
VID=$(curl -s -X POST http://127.0.0.1:8091/v1/videos \
-F prompt="A golden retriever runs along a sunny beach, waves crashing" \
-F size=864x480 -F num_frames=125 -F num_inference_steps=50 -F seed=42 \
| python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")
# Poll until "status": "completed"
curl -s http://127.0.0.1:8091/v1/videos/$VID
curl -s -o kandinsky6.mp4 http://127.0.0.1:8091/v1/videos/$VID/content
# Expect an MP4 with an H.264 video stream (125 frames, 24 fps, 864x480)
# and an AAC audio stream (44100 Hz, mono) of matching duration.
```
Smoke-size request for quick validation (~30 s end-to-end on one H100):
`-F size=512x320 -F num_frames=25 -F num_inference_steps=10`.
## Measurements
Single H100 80GB, seed 42, prompt
"A golden retriever runs along a sunny beach, waves crashing, cinematic footage",
negative prompt left at the pipeline default, MagCache off, prompt expansion
off. Pro geometry: 864x480, 125 frames, 50 steps, CFG 5.0, audio on.
vLLM-Omni: `text_to_video.py --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --enable-cpu-offload --model-class-name Kandinsky6TI2VAPipeline`,
FlashAttention-3.
| Wall / generation | Step time | Peak GPU | After load |
|---|---|---|---|
| 751.7 s generation (12.5 min); engine init 41 s | 14.4 s/it average | 78.39 GiB reserved (78.0 GiB nvidia-smi) | 17.74 GiB |
An earlier SDPA vLLM-Omni Pro run on the same GPU was 26.9 s/it and 1377 s,
peak reserved 74.97 GiB. A 512x320 / 25-frame / 10-step SDPA smoke was 1.4 s/it
and 29.5 s at the same reserved peak.
The MP4 is H.264 864x480, 125 frames at 24 fps (5.22 s), plus AAC 44.1 kHz
mono (230400 samples): a golden retriever on a sunny beach with breaking waves.
vLLM-Omni's VAE decode logged two failed CUDA allocations (7.15 GiB, then
4.27 GiB) and still wrote a complete file.
Audio is peak-normalized to full scale by default; pass
`extra_args.audio_normalization="clip"` to keep the raw decoded amplitude
instead.
## Notes
- Key flags: `extra_args.sample_audio` (bool, default true),
`extra_args.audio_normalization` (`"normalize"` default | `"clip"`),
`extra_args.visual_cond_scheme` (defaults to `pretrain` for text-only,
`tail_cond_first_frame` when an image is supplied).
- The post-process payload is a flat dict (`video`, `audio` as float32 in
`[-1, 1]`, `audio_sample_rate=44100`, `fps`) so the serving muxer picks
up the correct sample rate.
- Acceleration that is wired:
- Tensor parallel: the DiT loader narrows full checkpoint tensors onto
each rank's `ColumnParallelLinear` / `RowParallelLinear` shard.
- CFG parallel (`--cfg-parallel-size 2`): rank 0 runs the conditional
DiT, rank 1 the unconditional one, then both all-gather video and
audio velocities.
- HSDP: `_hsdp_shard_conditions` covers the indexed transformer blocks.
Do not combine with tensor parallel.
- Sequence parallel: visual tokens (and their RoPE) are sharded;
Ulysses all-to-all runs inside visual self-attention, Ring uses the
PyTorch ring kernel. Text stays replicated. Audio queries gather the
full visual sequence.
- Pipeline parallel: `visual_transformer_blocks` are split with
`PPMissingLayer`. Embeddings stay on the first stage, output heads on
the last. Activations and the final velocity are sent between stages.
- VAE patch parallel: Hunyuan decode splits the latent width when
`vae_patch_parallel_size > 1` and tiling is on.
- MagCache / TeaCache / Cache-DiT are registered. MagCache reuses the
Pro `mag_ratios`. TeaCache coefficients are **uncalibrated** (copied
from Qwen-Image) so skips are not quality-neutral. Cache-DiT targets
`visual_transformer_blocks` only and falls back to the same step cache
if the fused `(video, audio)` block return is rejected.
- FP8: online quantization runs in the DiT loader via
`process_weights_after_loading` after the sharded assign.
- `attention_engine: auto` uses FlashAttention-3 when
`flash_attn_interface` imports.
- Known limitations:
- NABLA sparse attention is not registered in the framework attention
backend. NaviCache is not wired.
- Prompt expansion, NF4 Qwen, and the super-resolution cascade are not
part of this port.
- Distributed layerwise offload still uses one process unless another
parallel axis sets `world_size > 1` (for example tensor parallel).
- `_load_components` loads the Diffusers checkpoint directly rather than the
streamed-prefetch loader.
## Supported features
| Feature | Status | Guide |
|---|---|---|
| Text-to-video-and-audio | Supported (validated on H100) | — |
| Image-to-video-and-audio | Supported (not yet validated on hardware) | — |
| CPU offload (`--enable-cpu-offload`) | Supported (required on 80 GB) | — |
| Layerwise offload (`--enable-layerwise-offload`) | Supported (about 36 GiB peak on the smoke geometry) | — |
| Tensor parallelism | Wired (sharded DiT load) | — |
| CFG parallelism | Wired (`--cfg-parallel-size 2`) | — |
| Ulysses / Ring sequence parallel | Wired (visual tokens only) | — |
| Pipeline parallelism | Wired (visual block split) | — |
| HSDP | Wired (not with tensor parallel) | — |
| VAE patch parallel | Wired (width-split decode) | — |
| MagCache | Wired (Pro ratios, step reuse) | — |
| TeaCache | Wired, coefficients uncalibrated | — |
| Cache-DiT | Wired on `visual_transformer_blocks` | — |
| FP8 | Wired (online quant; smoke peak about 46 GiB vs 75 GiB bf16) | [`FP8 quantization`](../../docs/user_guide/quantization/fp8.md) |
| NaviCache | Not supported | — |
| LoRA | Not supported | — |
+1
View File
@@ -64,6 +64,7 @@ recipes/
| [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) |
| [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB / 1x Intel Arc Pro B70 32GB |
| [`Kandinsky/Kandinsky6-TI2VA.md`](./Kandinsky/Kandinsky6-TI2VA.md) | Text/image-to-video-and-audio, offline and `/v1/videos` | 1x H100 80GB with CPU offload; 2x H100 for parallel modes |
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU / 1x Intel Arc Pro B70 32GB |
| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified / 1x Intel Arc Pro B70 32GB |
| [`m-a-p/SheetSage2-H200.md`](./m-a-p/SheetSage2-H200.md) | Official audio-to-score preprocessing and YuE2 request export | 1x H200 141GB |
+1
View File
@@ -59,6 +59,7 @@ def test_custom_cache_dit_enablers_are_registered_explicitly():
"Cosmos3OmniDiffusersPipeline": cd_model_specific.enable_cache_for_cosmos3,
"Cosmos3OmniPipeline": cd_model_specific.enable_cache_for_cosmos3,
"Krea2Pipeline": cd_model_specific.enable_cache_for_krea2,
"Kandinsky6TI2VAPipeline": cd_model_specific.enable_cache_for_kandinsky6,
"Magi2Pipeline": cd_model_specific.enable_cache_for_magi2,
"MammothModa2DiTPipeline": cd_model_specific.enable_cache_for_mammothmoda2,
}
Whitespace-only changes.
@@ -0,0 +1,89 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""CPU checks that Kandinsky 6 acceleration hooks are actually wired."""
import pytest
import torch
from torch import nn
from vllm_omni.diffusion.models.kandinsky6.cache_accel import K6_PRO_MAG_RATIOS, K6StepCache
from vllm_omni.diffusion.models.kandinsky6.kandinsky6_transformer import Kandinsky6Transformer3DModel
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import _shard_loaded_weight
from vllm_omni.diffusion.registry import _NO_CACHE_ACCELERATION
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
def test_shard_loaded_weight_narrows_column_parallel_output():
class _Owner:
tp_rank = 1
def weight_loader(self, param, loaded):
del param, loaded
param = nn.Parameter(torch.empty(2, 4))
param.output_dim = 0
param.weight_loader = _Owner().weight_loader
loaded = torch.arange(16, dtype=torch.float32).reshape(4, 4)
shard = _shard_loaded_weight(param, loaded)
assert tuple(shard.shape) == (2, 4)
torch.testing.assert_close(shard, loaded[2:4])
def test_row_bias_is_not_sharded_when_shapes_match():
class _Owner:
tp_rank = 1
def weight_loader(self, param, loaded):
del param, loaded
param = nn.Parameter(torch.empty(4))
param.output_dim = 0
param.weight_loader = _Owner().weight_loader
loaded = torch.arange(4, dtype=torch.float32)
shard = _shard_loaded_weight(param, loaded)
torch.testing.assert_close(shard, loaded)
def test_hsdp_predicate_matches_visual_blocks_only_when_indexed():
assert Kandinsky6Transformer3DModel._is_transformer_block("visual_transformer_blocks.3", nn.Identity())
assert Kandinsky6Transformer3DModel._is_transformer_block("video_text_transformer_blocks.0", nn.Identity())
assert not Kandinsky6Transformer3DModel._is_transformer_block("visual_embeddings", nn.Identity())
assert Kandinsky6Transformer3DModel._hsdp_shard_conditions
def test_sp_plan_shards_visual_tokens_and_gathers_before_the_head():
plan = Kandinsky6Transformer3DModel._sp_plan
assert "_sp_visual_shard" in plan
assert "_sp_visual_rope" in plan
assert plan["_sp_visual_gather"].gather_dim == 1
assert Kandinsky6Transformer3DModel._magcache_block_attrs == ("visual_transformer_blocks",)
def test_k6_is_not_on_the_no_cache_list_and_enablers_are_registered():
import vllm_omni.diffusion.cache.cachedit # noqa: F401 registers enablers
from vllm_omni.diffusion.cache.cachedit.backend import CUSTOM_DIT_ENABLERS
from vllm_omni.diffusion.cache.teacache.backend import CUSTOM_TEACACHE_ENABLERS
assert "Kandinsky6TI2VAPipeline" not in _NO_CACHE_ACCELERATION
assert "Kandinsky6TI2VAPipeline" in CUSTOM_TEACACHE_ENABLERS
assert "Kandinsky6TI2VAPipeline" in CUSTOM_DIT_ENABLERS
assert len(K6_PRO_MAG_RATIOS) == 100
def test_step_cache_skips_only_after_a_real_step():
cache = K6StepCache(
kind="mag",
threshold=0.24,
max_skip_steps=3,
retention_ratio=0.0,
mag_ratios=(1.0, 1.0),
)
assert cache.should_skip(0) is False
cache.store(torch.zeros(1), torch.zeros(1))
assert cache.should_skip(1) is True
@@ -0,0 +1,246 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import numpy as np
import pytest
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
def test_kandinsky6_pipeline_import_and_registry():
from vllm_omni.diffusion.models.kandinsky6 import (
AutoencoderKLHunyuanVideo,
Kandinsky6AudioVAE,
Kandinsky6TI2VAPipeline,
Kandinsky6Transformer3DModel,
KandinskyFlowMatchScheduler,
get_kandinsky6_post_process_func,
get_kandinsky6_pre_process_func,
)
from vllm_omni.diffusion.registry import (
_DIFFUSION_MODELS,
_DIFFUSION_POST_PROCESS_FUNCS,
_DIFFUSION_PRE_PROCESS_FUNCS,
_NO_CACHE_ACCELERATION,
)
assert Kandinsky6TI2VAPipeline is not None
assert Kandinsky6Transformer3DModel is not None
assert AutoencoderKLHunyuanVideo is not None
assert Kandinsky6AudioVAE is not None
assert KandinskyFlowMatchScheduler is not None
assert get_kandinsky6_post_process_func is not None
assert get_kandinsky6_pre_process_func is not None
assert _DIFFUSION_MODELS["Kandinsky6TI2VAPipeline"] == (
"kandinsky6",
"pipeline_kandinsky6",
"Kandinsky6TI2VAPipeline",
)
assert _DIFFUSION_POST_PROCESS_FUNCS["Kandinsky6TI2VAPipeline"] == "get_kandinsky6_post_process_func"
assert _DIFFUSION_PRE_PROCESS_FUNCS["Kandinsky6TI2VAPipeline"] == "get_kandinsky6_pre_process_func"
assert "Kandinsky6TI2VAPipeline" not in _NO_CACHE_ACCELERATION
def test_kandinsky6_component_discovery_declarations():
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6TI2VAPipeline
assert Kandinsky6TI2VAPipeline._dit_modules == ["transformer"]
assert Kandinsky6TI2VAPipeline._encoder_modules == ["text_encoder", "text_encoder_2"]
assert Kandinsky6TI2VAPipeline._vae_modules == ["vae", "audio_vae"]
assert Kandinsky6TI2VAPipeline.supports_step_execution is True
assert Kandinsky6TI2VAPipeline.support_audio_output is True
assert Kandinsky6TI2VAPipeline.support_image_input is True
def test_kandinsky6_pipeline_satisfies_capability_protocols():
"""isinstance checks against the @runtime_checkable protocols the
framework actually uses to detect these capabilities (io_support.py's
supports_audio_output, etc.) — stronger than just checking the class
attribute exists."""
from vllm_omni.diffusion.models.interface import (
SupportAudioOutput,
SupportImageInput,
SupportsComponentDiscovery,
)
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6TI2VAPipeline
assert isinstance(Kandinsky6TI2VAPipeline, SupportAudioOutput)
assert isinstance(Kandinsky6TI2VAPipeline, SupportImageInput)
assert isinstance(Kandinsky6TI2VAPipeline, SupportsComponentDiscovery)
class _FakeAudioVAE:
downsample_factor = 1024
def test_kandinsky6_post_process_func_packages_video_and_audio():
"""Pure-function test: no model/weights needed. Confirms the post-process
payload shape matches what io_support.py -> output_formatter.py ->
media_utils.py's PyAV muxing expects (the same flat {"video", "audio",
"audio_sample_rate", "fps"} shape MiniMax H3's own post-process function
produces), and that int16 PCM from ``postprocess_audio`` is rescaled to
the float32 [-1, 1] waveform the muxer consumes."""
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
post_process = get_kandinsky6_post_process_func(od_config=None)
video = np.zeros((1, 4, 8, 8, 3), dtype=np.uint8)
audio = np.array([0, 32767, -32767], dtype=np.int16)
result = post_process({"video": video, "audio": audio, "audio_sample_rate": 44100}, output_type="np")
assert result["video"] is video
assert result["audio"].dtype == np.float32
np.testing.assert_allclose(result["audio"], [0.0, 1.0, -1.0])
assert result["audio_sample_rate"] == 44100
assert result["fps"] == 24.0
def test_kandinsky6_post_process_sample_rate_survives_output_formatter():
"""The formatter only lifts ``audio_sample_rate`` into metadata from the
flat payload form; an envelope would silently lose it and the MP4 would
be muxed at the 24 kHz default."""
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
from vllm_omni.diffusion.output_formatter import normalize_diffusion_postprocess_output
post_process = get_kandinsky6_post_process_func(od_config=None)
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
audio = np.zeros((10,), dtype=np.int16)
normalized = normalize_diffusion_postprocess_output(
post_process({"video": video, "audio": audio, "audio_sample_rate": 44100}, output_type="np")
)
assert normalized.primary_key == "video"
assert normalized.metadata["audio"]["sample_rate"] == 44100
assert normalized.metadata["video"]["fps"] == 24.0
assert "audio_sample_rate" not in normalized.outputs
def test_kandinsky6_post_process_func_omits_audio_when_none():
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
post_process = get_kandinsky6_post_process_func(od_config=None)
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
result = post_process({"video": video, "audio": None, "audio_sample_rate": None}, output_type="np")
assert result["video"] is video
assert "audio" not in result
assert "audio_sample_rate" not in result
def test_kandinsky6_post_process_func_unwraps_batched_audio_list():
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_post_process_func
post_process = get_kandinsky6_post_process_func(od_config=None)
video = np.zeros((1, 2, 4, 4, 3), dtype=np.uint8)
audio_item = np.full((50,), 16384, dtype=np.int16)
result = post_process({"video": video, "audio": [audio_item], "audio_sample_rate": 44100}, output_type="np")
assert result["audio"].shape == (50,)
np.testing.assert_allclose(result["audio"], 16384 / 32767, rtol=1e-6)
class _StubAudioVAE:
"""Audio VAE stand-in: identity "decode" that returns the latent as a waveform."""
scaling_factor = 0.5
mean_value = 0.0
device = "cpu"
def wrapped_decode(self, latents):
# (1, audio_dim, A) -> (A,) waveform: take the first channel.
return latents[0, 0]
def test_postprocess_audio_normalizes_by_default_and_clips_on_request():
"""The default mode must match the k6_video production pipeline, which
peak-normalizes each decoded waveform to full scale; ``clip`` keeps the
raw amplitude (saturated to [-1, 1]) for callers that want it."""
import torch
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import LatentBundle, postprocess_audio
# Latent (A=4, audio_dim=1); the stub scales by 1/scaling_factor -> [0.2, -0.4, 0.6, 3.0].
audio = torch.tensor([[0.1], [-0.2], [0.3], [1.5]], dtype=torch.float32)
bundle = LatentBundle(
video=None,
audio=audio,
video_cu_seqlens=None,
audio_cu_seqlens=torch.tensor([0, 4], dtype=torch.int32),
)
vae = _StubAudioVAE()
normalized = postprocess_audio(bundle, vae)
assert normalized is not None and len(normalized) == 1
assert normalized[0].dtype == np.int16
np.testing.assert_allclose(
normalized[0].astype(np.float32) / 32767,
np.array([0.2, -0.4, 0.6, 3.0], dtype=np.float32) / 3.0,
atol=1e-4,
)
clipped = postprocess_audio(bundle, vae, normalization_mode="clip")
np.testing.assert_allclose(
clipped[0].astype(np.float32) / 32767,
np.array([0.2, -0.4, 0.6, 1.0], dtype=np.float32),
atol=1e-4,
)
assert postprocess_audio(LatentBundle(None, None, None, None), vae) is None
with pytest.raises(ValueError, match="normalization_mode"):
postprocess_audio(bundle, vae, normalization_mode="loud")
def test_kandinsky6_pre_process_func_is_identity_for_now():
"""v1 scope: the pre-process hook is registered (matching the framework's
convention for image-conditioned models like SanaImageToVideoPipeline)
but does not yet validate/transform the request — documented follow-up,
not a silent gap."""
from vllm_omni.diffusion.models.kandinsky6 import get_kandinsky6_pre_process_func
pre_process = get_kandinsky6_pre_process_func(od_config=None)
sentinel = object()
assert pre_process(sentinel) is sentinel
def test_hub_component_keys_map_onto_pipeline_modules():
"""Hub folder keys from Kandinsky-6.0-Pro-5s-Diffusers land on the modules."""
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import (
_WEIGHT_SUBFOLDERS,
_adapt_k6_weight_name,
)
assert _WEIGHT_SUBFOLDERS == (
("transformer", "transformer."),
("vae", "vae."),
("text_encoder", "text_encoder."),
("text_encoder_2", "text_encoder_2."),
("audio_vae", "audio_vae."),
)
assert _adapt_k6_weight_name("transformer.visual_transformer_blocks.0.videoT.self_attention.to_query.weight") == (
"transformer.visual_transformer_blocks.0.video_dec_block.self_attention.to_query.weight"
)
assert _adapt_k6_weight_name("transformer.visual_transformer_blocks.0.audioT.feed_forward.in_layer.weight") == (
"transformer.visual_transformer_blocks.0.audio_dec_block.feed_forward.in_layer.weight"
)
assert _adapt_k6_weight_name("vae.decoder.conv_in.conv.weight") == "vae.decoder.conv_in.conv.weight"
assert _adapt_k6_weight_name("text_encoder_2.encoder.layers.0.mlp.fc1.weight") == (
"text_encoder_2.encoder.layers.0.mlp.fc1.weight"
)
assert (
_adapt_k6_weight_name("text_encoder.model.layers.0.input_layernorm.weight")
== "text_encoder.model.language_model.layers.0.input_layernorm.weight"
)
assert _adapt_k6_weight_name("text_encoder.visual.blocks.0.norm1.weight") == (
"text_encoder.model.visual.blocks.0.norm1.weight"
)
assert (
_adapt_k6_weight_name("audio_vae.vae.decoder.conv_in.weight")
== "audio_vae.native.tod.vae.decoder.conv_in.weight"
)
assert _adapt_k6_weight_name("audio_vae.vocoder.conv_pre.weight") == "audio_vae.native.tod.vocoder.conv_pre.weight"
assert _adapt_k6_weight_name("audio_vae.mel_converter.hann_window") == "audio_vae.native.mel_converter.hann_window"
@@ -0,0 +1,325 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import os
from types import SimpleNamespace
import pytest
import torch
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
@pytest.fixture(autouse=True)
def _init_distributed(monkeypatch):
"""The native transformer uses vLLM parallel linear layers, which require a
tensor-parallel group; initialize a single-process group for CPU tests
(mirrors tests/diffusion/models/sana_video/test_transformer_sana_video.py)."""
from vllm.distributed.parallel_state import (
cleanup_dist_env_and_memory,
init_distributed_environment,
initialize_model_parallel,
)
from vllm.model_executor.layers.utils import default_unquantized_gemm
monkeypatch.setattr(
"vllm.model_executor.layers.linear.dispatch_unquantized_gemm",
lambda *_args, **_kwargs: default_unquantized_gemm,
)
os.environ.setdefault("MASTER_ADDR", "localhost")
os.environ.setdefault("MASTER_PORT", "29502")
init_distributed_environment(world_size=1, rank=0, local_rank=0, distributed_init_method="env://")
initialize_model_parallel()
yield
cleanup_dist_env_and_memory()
# model_dim must be an exact multiple of head_dim = sum(axes_dims); this is
# an architectural constraint of the native model (DiffusionTransformer3D),
# not specific to this port.
_TINY_T2V_CONFIG = {
"in_visual_dim": 4,
"out_visual_dim": 4,
"in_text_dim": 8,
"in_text_dim2": 6,
"time_dim": 16,
"patch_size": (1, 2, 2),
"model_dim": 24,
"ff_dim": 32,
"num_text_blocks": 1,
"num_visual_blocks": 2,
"axes_dims": (4, 4, 4),
"visual_cond": False,
"is_multimodal": False,
"attention_engine": "sdpa",
}
# time_dim_a is intentionally omitted (defaults to time_dim): the fused
# block's cross-modal modulation (va_mod/av_mod) consumes the *other*
# modality's time embedding by default (fix_modulation=False), so time_dim
# and time_dim_a must match unless fix_modulation=True.
_TINY_T2VA_CONFIG = {
**_TINY_T2V_CONFIG,
"is_multimodal": True,
"in_audio_dim": 6,
"model_dim_a": 12,
"ff_dim_a": 16,
"axes_dims_a": (2, 2, 2),
}
def _sdpa_config():
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec
return SimpleNamespace(
diffusion_attention_config=AttentionConfig(default=AttentionSpec(backend="TORCH_SDPA")),
parallel_config=SimpleNamespace(ring_degree=1, allgather_degree=1),
)
def _build_transformer(config):
from vllm_omni.diffusion.config import set_current_diffusion_config
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6Transformer3DModel
with set_current_diffusion_config(_sdpa_config()):
model = Kandinsky6Transformer3DModel(**config)
model.eval()
return model
def test_t2v_only_forward_shape():
model = _build_transformer(_TINY_T2V_CONFIG)
time_frames, height, width = 2, 4, 4
x_video = torch.randn(time_frames, height, width, 4)
text_embed = torch.randn(5, 8)
pooled = torch.randn(1, 6)
time = torch.tensor([500.0])
visual_rope = model.visual_rope_embeddings(
shape=(time_frames, height // 2, width // 2),
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
scale_factor=(1.0, 1.0, 1.0),
)
text_rope = model.text_rope_embeddings(torch.arange(5))
with torch.no_grad():
out = model(
x_video=x_video,
x_audio=None,
text_embed=text_embed,
pooled_text_embed=pooled,
time=time,
visual_rope=visual_rope,
audio_rope=None,
text_rope=text_rope,
)
assert out.shape == (time_frames, height, width, 4)
def test_visual_cond_channel_concat_path():
"""visual_cond=True doubles+1 the visual input channel count (cond +
mask channels appended by _build_video_input, stripped again by
OutLayer) — the I2V conditioning path."""
config = {**_TINY_T2V_CONFIG, "visual_cond": True}
model = _build_transformer(config)
time_frames, height, width = 2, 4, 4
x_video = torch.randn(time_frames, height, width, 2 * 4 + 1)
text_embed = torch.randn(5, 8)
pooled = torch.randn(1, 6)
time = torch.tensor([500.0])
visual_rope = model.visual_rope_embeddings(
shape=(time_frames, height // 2, width // 2),
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
scale_factor=(1.0, 1.0, 1.0),
)
text_rope = model.text_rope_embeddings(torch.arange(5))
with torch.no_grad():
out = model(
x_video=x_video,
x_audio=None,
text_embed=text_embed,
pooled_text_embed=pooled,
time=time,
visual_rope=visual_rope,
audio_rope=None,
text_rope=text_rope,
)
assert out.shape == (time_frames, height, width, 4)
def test_t2va_joint_forward_shapes():
"""Exercises the fused cross-modal decoder blocks (Kandinsky6FusedTransformerDecoderBlock)."""
model = _build_transformer(_TINY_T2VA_CONFIG)
time_frames, height, width = 2, 4, 4
audio_len = 7
x_video = torch.randn(time_frames, height, width, 4)
x_audio = torch.randn(audio_len, 6)
text_embed = torch.randn(5, 8)
pooled = torch.randn(1, 6)
time = torch.tensor([500.0])
visual_rope = model.visual_rope_embeddings(
shape=(time_frames, height // 2, width // 2),
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
scale_factor=(1.0, 1.0, 1.0),
)
audio_rope = model.audio_rope_embeddings(torch.arange(audio_len))
video_text_rope = model.video_text_rope_embeddings(torch.arange(5))
audio_text_rope = model.audio_text_rope_embeddings(torch.arange(5))
with torch.no_grad():
video_out, audio_out = model(
x_video=x_video,
x_audio=x_audio,
text_embed=[text_embed, text_embed],
pooled_text_embed=[pooled, pooled],
time=[time, time],
visual_rope=visual_rope,
audio_rope=audio_rope,
text_rope=[video_text_rope, audio_text_rope],
)
assert video_out.shape == (time_frames, height, width, 4)
assert audio_out.shape == (audio_len, 6)
@pytest.mark.parametrize("modality", ["video", "audio"])
def test_single_modality_through_multimodal_model(modality):
"""A multimodal (is_multimodal=True) model must still support a pure
video-only or audio-only forward call (denoise_loop's partial-sampling
path uses this)."""
model = _build_transformer(_TINY_T2VA_CONFIG)
time_frames, height, width = 2, 4, 4
audio_len = 7
text_embed = torch.randn(5, 8)
pooled = torch.randn(1, 6)
time = torch.tensor([500.0])
if modality == "video":
x_video = torch.randn(time_frames, height, width, 4)
visual_rope = model.visual_rope_embeddings(
shape=(time_frames, height // 2, width // 2),
pos=[torch.arange(time_frames), torch.arange(height // 2), torch.arange(width // 2)],
scale_factor=(1.0, 1.0, 1.0),
)
text_rope = model.video_text_rope_embeddings(torch.arange(5))
with torch.no_grad():
out = model(
x_video=x_video,
x_audio=None,
text_embed=text_embed,
pooled_text_embed=pooled,
time=time,
visual_rope=visual_rope,
audio_rope=None,
text_rope=text_rope,
)
assert out.shape == (time_frames, height, width, 4)
else:
x_audio = torch.randn(audio_len, 6)
audio_rope = model.audio_rope_embeddings(torch.arange(audio_len))
text_rope = model.audio_text_rope_embeddings(torch.arange(5))
with torch.no_grad():
out = model(
x_video=None,
x_audio=x_audio,
text_embed=text_embed,
pooled_text_embed=pooled,
time=time,
visual_rope=None,
audio_rope=audio_rope,
text_rope=text_rope,
)
assert out.shape == (audio_len, 6)
@pytest.mark.parametrize("batched", [False, True])
def test_self_attention_attends_over_tokens_for_batched_and_unbatched_inputs(batched):
"""Self-attention must contract over the *token* axis whether its input is
unbatched text ``(S, C)`` or batched=1 visual/audio ``(1, N, C)``.
Regression: an unconditional ``unsqueeze(0)`` on the already-batched
visual path handed SDPA a 5-D ``(1, 1, N, H, D)`` tensor, which it
accepts silently but then treats the head axis as the sequence — every
generated video decoded to pure noise."""
from vllm_omni.diffusion.config import set_current_diffusion_config
from vllm_omni.diffusion.models.kandinsky6.kandinsky6_transformer import Kandinsky6Attention, apply_rotary
torch.manual_seed(0)
channels, head_dim, tokens = 24, 12, 7
with set_current_diffusion_config(_sdpa_config()):
attn = Kandinsky6Attention(channels, head_dim, engine="sdpa").eval()
with torch.no_grad():
# vLLM parallel linears allocate with torch.empty (no init).
for param in attn.parameters():
param.normal_(std=0.2)
x = torch.randn(tokens, channels)
rope = torch.randn(tokens, 1, head_dim // 2, 2, 2)
hidden = x.unsqueeze(0) if batched else x
with torch.no_grad():
out = attn(hidden, rotary_emb=rope)
heads = channels // head_dim
q = attn.query_norm(attn.to_query(x).reshape(tokens, heads, head_dim))
k = attn.key_norm(attn.to_key(x).reshape(tokens, heads, head_dim))
v = attn.to_value(x).reshape(tokens, heads, head_dim)
q = apply_rotary(q, rope).type_as(q)
k = apply_rotary(k, rope).type_as(k)
ref = torch.nn.functional.scaled_dot_product_attention(
q.transpose(0, 1), k.transpose(0, 1), v.transpose(0, 1)
).transpose(0, 1)
expected = attn.out_layer(ref.reshape(tokens, channels))
assert out.shape == (hidden.shape[:-1] + (channels,))
torch.testing.assert_close(out.reshape(tokens, channels), expected, rtol=1e-4, atol=1e-5)
def test_modulation_projections_are_zero_initialized():
"""Modulation layers are zero-initialized at construction (standard
AdaLN-zero init) — confirms the zero-init in Kandinsky6Modulation.__init__
reaches the tensor-parallel out_layer correctly (it's a
ColumnParallelLinear, not plain nn.Linear)."""
model = _build_transformer(_TINY_T2V_CONFIG)
for block in model.text_transformer_blocks:
assert torch.all(block.text_modulation.out_layer.weight == 0)
for block in model.visual_transformer_blocks:
assert torch.all(block.visual_modulation.out_layer.weight == 0)
def test_dit_constructs_from_a_diffusers_config_dict():
from vllm_omni.diffusion.config import set_current_diffusion_config
from vllm_omni.diffusion.models.kandinsky6 import Kandinsky6Transformer3DModel
with set_current_diffusion_config(_sdpa_config()):
model = Kandinsky6Transformer3DModel.from_diffusers_config(
{**_TINY_T2VA_CONFIG, "patch_size": [1, 2, 2], "axes_dims": [4, 4, 4]}
)
assert tuple(model.patch_size) == (1, 2, 2)
assert model._repeated_blocks == [
"Kandinsky6FusedTransformerDecoderBlock",
"Kandinsky6TransformerEncoderBlock",
"Kandinsky6TransformerDecoderBlock",
]
visual = model.visual_transformer_blocks[0].video_dec_block.self_attention.attn
audio = model.visual_transformer_blocks[0].audio_dec_block.self_attention.attn
text = model.video_text_transformer_blocks[0].self_attention.attn
cross = model.visual_transformer_blocks[0].va_cross_attention.attn
assert visual.role == "kandinsky6.visual_self"
assert visual.skip_sequence_parallel is False
assert audio.skip_sequence_parallel is True
assert text.role == "kandinsky6.text_self"
assert text.skip_sequence_parallel is True
assert cross.role == "kandinsky6.video_audio_cross"
assert cross.skip_sequence_parallel is True
def test_cfg_combine_matches_apply_cfg_without_normalization():
from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin
from vllm_omni.diffusion.models.kandinsky6.pipeline_kandinsky6 import apply_cfg
positive = torch.tensor([1.0, 3.0])
negative = torch.tensor([0.25, -1.0])
combined = CFGParallelMixin.combine_cfg_noise(object(), positive, negative, 5.0, False)
torch.testing.assert_close(combined, apply_cfg(positive, negative, 5.0))
@@ -0,0 +1,118 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Kandinsky 6 VAE patch parallel keeps the single-GPU tile grid."""
import os
from types import SimpleNamespace
import pytest
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from vllm_omni.diffusion.models.kandinsky6.modeling_kandinsky6_vae import AutoencoderKLHunyuanVideo
def _stub_vae() -> AutoencoderKLHunyuanVideo:
vae = AutoencoderKLHunyuanVideo.__new__(AutoencoderKLHunyuanVideo)
vae.spatial_compression_ratio = 8
vae.temporal_compression_ratio = 4
vae.use_tiling = True
vae.use_framewise_decoding = True
vae.tile_sample_min_height = 32
vae.tile_sample_min_width = 32
vae.tile_sample_min_num_frames = 16
vae.tile_sample_stride_height = 24
vae.tile_sample_stride_width = 24
vae.tile_sample_stride_num_frames = 12
vae.tile_size = None
vae.distributed_executor = None
vae.post_quant_conv = lambda tensor: tensor
vae._seen_widths = []
def decoder(tensor: torch.Tensor) -> torch.Tensor:
vae._seen_widths.append(int(tensor.shape[-1]))
return tensor + 1
vae.decoder = decoder
return vae
@pytest.mark.core_model
@pytest.mark.cpu
def test_distributed_decode_tiles_match_local_tiled_decode():
vae = _stub_vae()
latent = torch.arange(1 * 4 * 2 * 8 * 12, dtype=torch.float32).reshape(1, 4, 2, 8, 12)
local = vae.tiled_decode(latent).sample
tasks, spec = vae._decode_tile_split(latent)
assert len(tasks) > 1
assert max(int(task.tensor.shape[-1]) for task in tasks) < latent.shape[-1]
decoded = {task.grid_coord: vae._decode_tile_exec(task) for task in tasks}
merged = vae._decode_tile_merge(decoded, spec)
torch.testing.assert_close(merged, local)
@pytest.mark.core_model
@pytest.mark.cpu
def test_decode_still_applies_optimal_tiling_when_patch_parallel_is_on():
vae = _stub_vae()
seen = {}
def tiling(shape, device=None):
del device
seen["shape"] = tuple(shape)
return (1, 17, 32, 32), (8, 24, 24)
from diffusers.models.autoencoders.vae import DecoderOutput
vae.get_dec_optimal_tiling = tiling
vae._decode = lambda z, return_dict=True: DecoderOutput(sample=z)
vae.distributed_executor = SimpleNamespace(parallel_size=2)
vae.is_distributed_enabled = lambda: True
out = vae.decode(torch.zeros(1, 16, 8, 8, 12))
assert seen["shape"] == (1, 16, 8, 8, 12)
assert tuple(out.sample.shape) == (1, 16, 8, 8, 12)
def _patch_parallel_worker(rank: int, port: str, shapes: dict) -> None:
os.environ["MASTER_ADDR"] = "127.0.0.1"
os.environ["MASTER_PORT"] = port
os.environ["RANK"] = str(rank)
os.environ["WORLD_SIZE"] = "2"
dist.init_process_group("gloo", rank=rank, world_size=2)
try:
import vllm_omni.diffusion.distributed.autoencoders.distributed_vae_executor as executor_mod
world = SimpleNamespace(device_group=dist.group.WORLD)
executor_mod.get_world_group = lambda: world
vae = _stub_vae()
latent = torch.arange(1 * 4 * 2 * 8 * 12, dtype=torch.float32).reshape(1, 4, 2, 8, 12)
reference = vae.tiled_decode(latent).sample.clone()
vae._seen_widths.clear()
vae.set_parallel_size(2, mode="tile")
decoded = vae.tiled_decode(latent).sample
torch.testing.assert_close(decoded, reference)
if rank == 0:
shapes["widths"] = list(vae._seen_widths)
shapes["full_width"] = int(latent.shape[-1])
finally:
dist.destroy_process_group()
@pytest.mark.core_model
@pytest.mark.cpu
def test_patch_parallel_decode_splits_tiles_across_ranks():
manager = mp.Manager()
shapes = manager.dict()
port = str(29500 + (os.getpid() % 1000))
mp.spawn(_patch_parallel_worker, args=(port, shapes), nprocs=2, join=True)
assert shapes["widths"]
assert max(shapes["widths"]) < shapes["full_width"]
@@ -0,0 +1,79 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Offline smoke for Kandinsky 6 TI2VA.
Short request (256x256, 5 frames, 2 steps) so it finishes inside the default
online poll window. Pro geometry (480x864, 125 frames, 50 steps) stays the
serving default, not this smoke.
"""
import os
from pathlib import Path
import pytest
from tests.helpers.mark import hardware_test
from tests.helpers.runtime import OfflineOmniClient, get_model_prefix
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
_HUB_MODEL = "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers"
_LOCAL_BUNDLE = Path(__file__).resolve().parents[4] / "kandinsky6_bundle"
def _resolve_model() -> str:
env = os.environ.get("KANDINSKY6_MODEL")
if env:
return env
if _LOCAL_BUNDLE.is_dir():
return str(_LOCAL_BUNDLE)
return _HUB_MODEL
def _checkpoint_on_disk(model: str) -> bool:
if Path(model).is_dir():
return True
prefix = get_model_prefix()
return bool(prefix) and Path(prefix + model).is_dir()
MODEL = _resolve_model()
PROMPT = "A golden retriever runs along a sunny beach, waves crashing."
pytestmark = [
pytest.mark.skipif(
not _checkpoint_on_disk(MODEL),
reason=(
"Kandinsky 6 Diffusers weights are gated. Set KANDINSKY6_MODEL or "
"MODEL_PREFIX to a local copy of "
"kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers."
),
),
]
@pytest.mark.advanced_model
@pytest.mark.diffusion
@hardware_test(res={"cuda": "H100"}, num_cards={"cuda": 1})
@pytest.mark.parametrize(
"omni_runner",
[(MODEL, None, {"enable_cpu_offload": True, "enforce_eager": True})],
indirect=True,
)
def test_text_to_video_and_audio_001(offline_client: OfflineOmniClient):
sampling = OmniDiffusionSamplingParams(
height=256,
width=256,
num_frames=5,
fps=24,
num_inference_steps=2,
guidance_scale=5.0,
seed=42,
extra_args={"sample_audio": True},
)
request_config = {
"model": MODEL,
"prompt": PROMPT,
"sampling_params": sampling,
}
offline_client.send_diffusion_request(request_config)
@@ -0,0 +1,89 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Online smoke for Kandinsky 6 TI2VA via ``/v1/videos``.
Short geometry so the job finishes inside the helper's default 300 s poll.
"""
import os
from pathlib import Path
import pytest
from tests.helpers.mark import hardware_marks
from tests.helpers.runtime import OmniServer, OmniServerParams, OnlineOmniClient, get_model_prefix
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
_HUB_MODEL = "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers"
_LOCAL_BUNDLE = Path(__file__).resolve().parents[4] / "kandinsky6_bundle"
def _resolve_model() -> str:
env = os.environ.get("KANDINSKY6_MODEL")
if env:
return env
if _LOCAL_BUNDLE.is_dir():
return str(_LOCAL_BUNDLE)
return _HUB_MODEL
def _checkpoint_on_disk(model: str) -> bool:
if Path(model).is_dir():
return True
prefix = get_model_prefix()
return bool(prefix) and Path(prefix + model).is_dir()
MODEL = _resolve_model()
PROMPT = "A golden retriever runs along a sunny beach, waves crashing."
pytestmark = [
pytest.mark.skipif(
not _checkpoint_on_disk(MODEL),
reason=(
"Kandinsky 6 Diffusers weights are gated. Set KANDINSKY6_MODEL or "
"MODEL_PREFIX to a local copy of "
"kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers."
),
),
]
SINGLE_CARD_FEATURE_MARKS = hardware_marks(res={"cuda": "H100"})
def _get_diffusion_feature_cases(model: str):
return [
pytest.param(
OmniServerParams(
model=model,
server_args=["--enable-cpu-offload", "--enforce-eager"],
init_timeout=3600,
stage_init_timeout=3600,
),
id="cpu_offload",
marks=SINGLE_CARD_FEATURE_MARKS,
),
]
@pytest.mark.advanced_model
@pytest.mark.diffusion
@pytest.mark.parametrize("omni_server", _get_diffusion_feature_cases(MODEL), indirect=True)
def test_text_to_video_and_audio_001(omni_server: OmniServer, online_client: OnlineOmniClient) -> None:
"""Short TI2VA job completes and returns an MP4."""
request_config = {
"model": omni_server.model,
"form_data": {
"prompt": PROMPT,
"height": 256,
"width": 256,
"num_frames": 5,
"fps": 24,
"num_inference_steps": 2,
"guidance_scale": 5.0,
"seed": 42,
},
}
online_client.send_video_diffusion_request(request_config)
@@ -999,6 +999,38 @@ def test_magi2_serving_applies_native_defaults_and_rejects_explicit_frame_mismat
)
def test_kandinsky6_serving_applies_defaults_but_allows_explicit_frame_count():
"""Kandinsky 6 publishes Pro-geometry defaults (480x864, 125 frames, 50
steps) but is not a fixed-duration model: an explicit ``num_frames``
must be honoured rather than rejected."""
engine = FakeAsyncOmni()
engine.model_class_name = "Kandinsky6TI2VAPipeline"
handler = OmniOpenAIServingVideo.for_diffusion(
diffusion_engine=engine,
model_name="kandinsky6-bundle",
)
asyncio.run(handler._run_and_extract(VideoGenerationRequest(prompt="A dog runs on a beach"), "defaults"))
sampling = engine.captured_sampling_params_list[0]
assert (sampling.width, sampling.height) == (864, 480)
assert sampling.num_frames == 125
assert sampling.num_inference_steps == 50
assert sampling.fps == 24.0
asyncio.run(
handler._run_and_extract(
VideoGenerationRequest(
prompt="A dog runs on a beach", width=512, height=320, num_frames=25, num_inference_steps=10
),
"explicit",
)
)
sampling = engine.captured_sampling_params_list[-1]
assert (sampling.width, sampling.height) == (512, 320)
assert sampling.num_frames == 25
assert sampling.num_inference_steps == 10
def test_i2v_video_generation_with_image_reference_form(test_client, mocker: MockerFixture):
mocker.patch(
"vllm_omni.entrypoints.openai.serving_video._encode_video_bytes",
@@ -76,6 +76,9 @@ EXCLUDED_MODELS = [
# The 270+ GiB multimodal checkpoint needs 4/8-way native sequence/head
# parallel execution; covered by focused native contract and GPU tests.
"Magi2Pipeline",
# Pro DiT is ~56 GiB bf16 and the Hub checkpoint is gated. Covered by
# tests/diffusion/models/kandinsky6 and the real-checkpoint E2E smoke.
"Kandinsky6TI2VAPipeline",
"OmniVoicePipeline",
"OmniVoice",
"Cosmos3OmniDiffusersPipeline",
+30
View File
@@ -908,6 +908,35 @@ def enable_cache_for_mammothmoda2(pipeline: Any, cache_config: Any) -> CacheDiTE
return CacheDiTEnableResult(refresh=refresh, targets=(block_adapter,))
def enable_cache_for_kandinsky6(pipeline: Any, cache_config: Any) -> RefreshCacheContextFunc:
"""Cache-DiT over K6 visual blocks only.
Text and audio block lists are not one residual chain with the visual
stack. If the block adapter rejects the fused ``(video, audio)`` return,
fall back to the same step-level velocity cache MagCache uses.
"""
transformer = pipeline.transformer
block_adapter = BlockAdapter(
transformer=transformer,
blocks=[transformer.visual_transformer_blocks],
forward_pattern=[ForwardPattern.Pattern_0],
has_separate_cfg=True,
check_forward_pattern=False,
)
try:
return enable_cache_for_dit(pipeline, cache_config, block_adapter)
except Exception as exc:
logger.warning("Cache-DiT block adapter failed for Kandinsky 6 (%s); using step cache", exc)
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_mag_cache
attach_mag_cache(transformer, threshold=0.24, max_skip_steps=3, retention_ratio=0.1)
def _refresh(*_args, **_kwargs):
return None
return _refresh
def register_custom_dit_enablers() -> None:
"""Register model-specific Cache-DiT enablers.
@@ -925,6 +954,7 @@ def register_custom_dit_enablers() -> None:
"Cosmos3OmniDiffusersPipeline": enable_cache_for_cosmos3,
"Cosmos3OmniPipeline": enable_cache_for_cosmos3,
"Krea2Pipeline": enable_cache_for_krea2,
"Kandinsky6TI2VAPipeline": enable_cache_for_kandinsky6,
"Magi2Pipeline": enable_cache_for_magi2,
"MammothModa2DiTPipeline": enable_cache_for_mammothmoda2,
}
+14 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""
MagCache backend implementation.
@@ -73,6 +73,19 @@ class MagCacheBackend(CacheBackend):
transformer = pipeline.transformer
transformer_type = transformer.__class__.__name__
if transformer_type == "Kandinsky6Transformer3DModel":
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_mag_cache
attach_mag_cache(
transformer,
threshold=self.config.mag_threshold,
max_skip_steps=self.config.mag_max_skip_steps,
retention_ratio=self.config.mag_retention_ratio,
)
self._registered = True
logger.info("MagCache: Kandinsky 6 step-cache attached (visual/audio velocity reuse)")
return
num_inference_steps = self.config.num_inference_steps or 28
mag_ratios = self.config.mag_ratios
+4 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""
Hook-based MagCache implementation for vLLM-Omni.
@@ -543,9 +543,12 @@ def apply_mag_cache_hook(
state_manager = StateManager(MagCacheState, (), {})
remaining_blocks = []
allowed = getattr(module, "_magcache_block_attrs", None)
for name, submodule in module.named_children():
if not isinstance(submodule, torch.nn.ModuleList):
continue
if allowed is not None and name not in allowed:
continue
for index, block in enumerate(submodule):
remaining_blocks.append((f"{name}.{index}", block))
+20 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""
TeaCache backend implementation.
@@ -143,11 +143,30 @@ def enable_flux2_klein_teacache(pipeline: Any, config: DiffusionCacheConfig) ->
)
def enable_kandinsky6_teacache(pipeline: Any, config: DiffusionCacheConfig) -> None:
"""TeaCache for Kandinsky 6 via the denoise-loop step cache.
Coefficients are uncalibrated (see ``K6_TEACACHE_COEFFICIENTS``).
"""
from vllm_omni.diffusion.models.kandinsky6.cache_accel import attach_tea_cache
attach_tea_cache(
pipeline.transformer,
rel_l1_thresh=config.rel_l1_thresh if config.rel_l1_thresh is not None else 0.2,
coefficients=tuple(config.coefficients) if config.coefficients is not None else None,
)
logger.info(
"TeaCache: Kandinsky 6 step-cache attached (uncalibrated coefficients, thresh=%s)",
config.rel_l1_thresh,
)
CUSTOM_TEACACHE_ENABLERS = {
"BagelPipeline": enable_bagel_teacache,
"Flux2KleinPipeline": enable_flux2_klein_teacache,
"HunyuanImage3Pipeline": enable_hunyuan_image3_teacache,
"MammothModa2DiTPipeline": enable_mammoth_moda2_teacache,
"Kandinsky6TI2VAPipeline": enable_kandinsky6_teacache,
"MiniMaxH3Pipeline": enable_minimax_h3_teacache,
"SenseNovaU1Pipeline": enable_sensenova_u1_teacache,
}
+10 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from dataclasses import dataclass
@@ -8,6 +8,15 @@ from dataclasses import dataclass
# Source: TeaCache paper and ComfyUI-TeaCache empirical tuning
_MODEL_COEFFICIENTS = {
# FLUX transformer coefficients from TeaCache paper
# Uncalibrated stand-in so Kandinsky 6 TeaCache can be enabled.
# Copied from Qwen-Image; refit before treating skips as quality-neutral.
"Kandinsky6Transformer3DModel": [
-4.50000000e02,
2.80000000e02,
-4.50000000e01,
3.20000000e00,
-2.00000000e-02,
],
"FluxTransformer2DModel": [
4.98651651e02,
-2.83781631e02,
+9
View File
@@ -81,6 +81,15 @@ _DIFFUSION_MODEL_METADATA: dict[str, DiffusionModelMetadata] = {
max_multimodal_image_inputs=1,
final_output_type="video",
),
# Joint text/image-to-video-and-audio, same shape as MiniMaxH3Pipeline
# above (an MP4 with both tracks) — declared "video" for the same reason:
# the final container is a video file, so /v1/videos* is the right API
# surface even though the model also produces audio.
"Kandinsky6TI2VAPipeline": DiffusionModelMetadata(
supports_multimodal_inputs=True,
max_multimodal_image_inputs=1,
final_output_type="video",
),
"WanPipeline": DiffusionModelMetadata(
attention_mask_free=True,
final_output_type="video",
@@ -0,0 +1,22 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from .kandinsky6_transformer import Kandinsky6Transformer3DModel
from .modeling_kandinsky6_audio import Kandinsky6AudioVAE
from .modeling_kandinsky6_vae import AutoencoderKLHunyuanVideo
from .pipeline_kandinsky6 import (
Kandinsky6TI2VAPipeline,
get_kandinsky6_post_process_func,
get_kandinsky6_pre_process_func,
)
from .scheduling_kandinsky6 import KandinskyFlowMatchScheduler
__all__ = [
"AutoencoderKLHunyuanVideo",
"Kandinsky6AudioVAE",
"Kandinsky6TI2VAPipeline",
"Kandinsky6Transformer3DModel",
"KandinskyFlowMatchScheduler",
"get_kandinsky6_post_process_func",
"get_kandinsky6_pre_process_func",
]
@@ -0,0 +1,199 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Kandinsky 6 cache adapters.
Generic MagCache / TeaCache hooks assume a single residual stream named
``hidden_states``. K6's fused visual blocks return ``(video, audio)``, so
the denoise loop consults :class:`K6StepCache` instead and reuses the
previous velocity when the schedule says the step can be skipped.
"""
from __future__ import annotations
from dataclasses import dataclass
# Pro geometry ratios from k6_video
# ``k6_pro_125_480_864_mCache_mOffload.yaml`` (cache.magcache.mag_ratios).
K6_PRO_MAG_RATIOS: tuple[float, ...] = (
1.0,
1.0,
0.94526286,
0.94622554,
1.13130749,
1.11243134,
1.02926022,
1.02675114,
1.011996,
1.01235353,
1.03039566,
1.02848639,
1.00101278,
1.00307236,
1.02238594,
1.02108294,
1.0248162,
1.02491556,
0.99771836,
0.99882495,
1.01694329,
1.01367628,
1.01096435,
1.00959183,
1.01537554,
1.01504827,
0.99237553,
0.99345238,
1.02368983,
1.02088727,
1.01396087,
1.01278108,
0.99773024,
0.99927632,
1.01573298,
1.01489232,
1.00129728,
1.00120319,
1.01759963,
1.01691798,
0.99335877,
0.99548372,
1.02925427,
1.02299559,
0.99749375,
0.99682709,
0.99694342,
1.00064403,
1.01101129,
1.0117984,
1.04420642,
1.03624643,
0.9599572,
0.97176547,
1.00452036,
1.0046107,
1.01447715,
1.01351055,
0.9999695,
1.00270806,
1.00770242,
1.00271775,
0.98795461,
0.98880837,
1.01113352,
1.00860993,
1.00116034,
1.00251894,
1.05147006,
1.04040677,
0.99651072,
0.99212138,
0.99040673,
0.98881456,
1.01265486,
1.01241698,
0.97343865,
0.97302804,
1.02539269,
1.02113221,
1.02117306,
1.02987275,
0.98607016,
0.98262938,
0.98404577,
0.98453141,
1.01065119,
1.00807871,
0.99986077,
0.99668861,
0.98597074,
0.98133325,
0.98239303,
0.98370113,
0.93155548,
0.93534231,
0.94231988,
0.93388087,
0.84200656,
0.83657581,
)
# Uncalibrated. Borrowed from the Qwen-Image TeaCache polynomial so the flag
# skips steps instead of raising. Quality will not match a fitted K6 curve.
K6_TEACACHE_COEFFICIENTS: tuple[float, ...] = (
-4.50000000e02,
2.80000000e02,
-4.50000000e01,
3.20000000e00,
-2.00000000e-02,
)
@dataclass
class K6StepCache:
"""Step-level skip state shared by MagCache and TeaCache."""
kind: str
threshold: float
max_skip_steps: int
retention_ratio: float
mag_ratios: tuple[float, ...] | None = None
coefficients: tuple[float, ...] | None = None
rel_l1_thresh: float = 0.2
last_cond: object = None
last_uncond: object = None
accumulated_err: float = 0.0
accumulated_ratio: float = 1.0
accumulated_steps: int = 0
def should_skip(self, step_index: int) -> bool:
if self.last_cond is None or step_index <= 0:
return False
if self.kind == "mag":
ratios = self.mag_ratios or ()
scale = float(ratios[step_index]) if step_index < len(ratios) else 1.0
self.accumulated_ratio *= scale
self.accumulated_steps += 1
self.accumulated_err += abs(1.0 - self.accumulated_ratio)
if (
self.accumulated_err <= self.threshold
and self.accumulated_steps <= self.max_skip_steps
and step_index >= max(1, int(self.retention_ratio * max(step_index, 1)))
):
return True
self.accumulated_ratio = 1.0
self.accumulated_steps = 0
self.accumulated_err = 0.0
return False
# TeaCache stand-in: uncalibrated polynomial of the step index.
# The real TeaCache distance is the modulated input; this only makes
# the flag skip steps instead of raising.
coeff = self.coefficients or K6_TEACACHE_COEFFICIENTS
x = float(step_index)
predicted = sum(weight * (x ** (len(coeff) - 1 - i)) for i, weight in enumerate(coeff))
return abs(predicted) < self.rel_l1_thresh * 1.0e4
def store(self, cond: object, uncond: object) -> None:
self.last_cond = cond
self.last_uncond = uncond
def attach_mag_cache(transformer, *, threshold: float, max_skip_steps: int, retention_ratio: float) -> None:
transformer._k6_step_cache = K6StepCache(
kind="mag",
threshold=threshold,
max_skip_steps=max_skip_steps,
retention_ratio=retention_ratio,
mag_ratios=K6_PRO_MAG_RATIOS,
)
def attach_tea_cache(transformer, *, rel_l1_thresh: float, coefficients: tuple[float, ...] | None) -> None:
transformer._k6_step_cache = K6StepCache(
kind="tea",
threshold=rel_l1_thresh if rel_l1_thresh is not None else 0.2,
max_skip_steps=3,
retention_ratio=0.0,
coefficients=coefficients or K6_TEACACHE_COEFFICIENTS,
rel_l1_thresh=rel_l1_thresh if rel_l1_thresh is not None else 0.2,
)
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,103 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Native K6 shifted flow-matching Euler scheduler.
Self-contained (no ``diffusers`` dependency), matching the duck-typed
scheduler interface (``set_timesteps`` / ``timesteps`` / ``sigmas`` /
``step``) that other native vLLM-Omni diffusion pipelines use for their own
per-model schedulers — see ``WanEulerScheduler`` in
``vllm_omni.diffusion.models.wan2_2.scheduling_wan_euler`` for the sibling
pattern this mirrors.
The math reproduces ``kandinsky.core.algo.flow_matching.flow_match_timesteps``
exactly: for ``t`` linearly spaced from 1.0 down to 0.0,
``sigma = scale * t / (1 + (scale - 1) * t)``. This is the same shifted
rectified-flow schedule family Wan's own scheduler uses, parameterized by
``scheduler_scale`` (K6's name for what Wan calls ``shift``).
"""
from __future__ import annotations
from dataclasses import dataclass
from types import SimpleNamespace
import torch
@dataclass
class KandinskyFlowMatchSchedulerOutput:
prev_sample: torch.Tensor
class KandinskyFlowMatchScheduler:
"""Stateful Euler stepper over K6's shifted flow-matching sigma schedule.
``timesteps`` holds the model-scale (``[0, 1000]``-ish) values fed to the
DiT's ``time`` input; ``sigmas`` holds the same schedule unscaled
(``num_inference_steps + 1`` entries, running 1.0 -> 0.0), needed to
derive a step size without consuming a stateful ``step()`` call when one
modality is stepped implicitly alongside another (see the ``vllm`` port's
``denoise_loop`` override).
"""
order = 1
def __init__(self, scheduler_scale: float = 1.0, device: torch.device | str = "cpu") -> None:
self.scheduler_scale = float(scheduler_scale)
self.device = device
self.config = SimpleNamespace(scheduler_scale=self.scheduler_scale)
self._step_index: int | None = None
self.timesteps = torch.empty(0, dtype=torch.float32)
self.sigmas = torch.empty(0, dtype=torch.float32)
self.set_timesteps(num_inference_steps=1, device=self.device)
@property
def step_index(self) -> int | None:
return self._step_index
def set_timesteps(
self,
num_inference_steps: int,
device: torch.device | str | None = None,
**kwargs, # noqa: ARG002 - kept for scheduler API compatibility
) -> None:
device = device or self.device
t = torch.linspace(1.0, 0.0, int(num_inference_steps) + 1, device=device, dtype=torch.float32)
scale = self.scheduler_scale
self.sigmas = scale * t / (1 + (scale - 1) * t)
# DiT `time` input is model-scale, i.e. sigma * 1000 (matches the core
# algo loop's `t.unsqueeze(0).expand(bs) * 1000`).
self.timesteps = self.sigmas[:-1] * 1000.0
self._step_index = None
def index_for_timestep(self, timestep: torch.Tensor) -> int:
timestep_t = torch.as_tensor(timestep, device=self.timesteps.device, dtype=self.timesteps.dtype)
return int(torch.argmin(torch.abs(self.timesteps - timestep_t)).item())
def step(
self,
model_output: torch.Tensor,
timestep: torch.Tensor,
sample: torch.Tensor,
return_dict: bool = True,
**kwargs, # noqa: ARG002 - kept for scheduler API compatibility
) -> KandinskyFlowMatchSchedulerOutput | tuple[torch.Tensor]:
if self._step_index is None:
self._step_index = self.index_for_timestep(timestep)
sigma = self.sigmas[self._step_index]
sigma_next = self.sigmas[self._step_index + 1]
dt = (sigma_next - sigma).to(device=sample.device, dtype=sample.dtype)
prev_sample = sample + dt * model_output
self._step_index += 1
if not return_dict:
return (prev_sample,)
return KandinskyFlowMatchSchedulerOutput(prev_sample=prev_sample)
def __len__(self) -> int:
return int(self.timesteps.shape[0])
+7
View File
@@ -396,6 +396,11 @@ _DIFFUSION_MODELS = {
"pipeline_krea2",
"Krea2Pipeline",
),
"Kandinsky6TI2VAPipeline": (
"kandinsky6",
"pipeline_kandinsky6",
"Kandinsky6TI2VAPipeline",
),
}
_DIFFUSION_MODELS["MingImageLayeredDiffusionPipeline"] = _DIFFUSION_MODELS["MingImageDiffusionPipeline"]
@@ -710,6 +715,7 @@ _DIFFUSION_POST_PROCESS_FUNCS = {
"StableDiffusionXLPipeline": "get_sdxl_image_post_process_func",
"Krea2Pipeline": "get_krea2_post_process_func",
"HunyuanImage3ForCausalMM": "get_hunyuan_image3_post_process_func",
"Kandinsky6TI2VAPipeline": "get_kandinsky6_post_process_func",
}
_DIFFUSION_POST_PROCESS_FUNCS["MingImageLayeredDiffusionPipeline"] = _DIFFUSION_POST_PROCESS_FUNCS[
"MingImageDiffusionPipeline"
@@ -756,6 +762,7 @@ _DIFFUSION_PRE_PROCESS_FUNCS = {
"Cosmos3OmniDiffusersPipeline": "get_cosmos3_pre_process_func",
"Cosmos3OmniPipeline": "get_cosmos3_pre_process_func",
"MammothModa2DiTPipeline": "get_mammoth_moda2_pre_process_func",
"Kandinsky6TI2VAPipeline": "get_kandinsky6_pre_process_func",
}
+7 -1
View File
@@ -158,7 +158,13 @@ def is_diffusion_model(model_name: str) -> bool:
logger.debug("Failed to read local %s: %s", filename, e)
# Strategy 2: Check using vllm's utility (works for both local and remote models)
config_dict = get_diffusion_model_index(model_name)
# get_hf_file_to_dict can raise for local paths that don't resolve to a Hub
# repo id (see resolve_model_class_name's identical guard for the same call).
try:
config_dict = get_diffusion_model_index(model_name)
except Exception as e:
logger.debug("Failed to fetch diffusion model index via vllm utility: %s", e)
config_dict = None
if config_dict is not None and config_dict.get("_class_name") and config_dict.get("_diffusers_version"):
logger.debug("Detected diffusion model via a standard Diffusers index")
return True
@@ -334,7 +334,12 @@ class OmniOpenAIServingVideo:
)
vp.width = vp.width or video_defaults.width
vp.height = vp.height or video_defaults.height
if num_frames_provided and vp.num_frames != video_defaults.num_frames:
# Only fixed-duration pipelines (those declaring
# ``duration_seconds``, e.g. MAGI-2) pin the frame count; models
# that merely publish serving defaults (e.g. Kandinsky 6, any
# 4k+1 frame count) still accept an explicit ``num_frames``.
fixed_frame_count = video_defaults.duration_seconds is not None
if fixed_frame_count and num_frames_provided and vp.num_frames != video_defaults.num_frames:
raise HTTPException(
status_code=HTTPStatus.BAD_REQUEST.value,
detail=(f"This diffusion model requires {video_defaults.num_frames} frames; got {vp.num_frames}."),
+48
View File
@@ -0,0 +1,48 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Request fields and example defaults for the Kandinsky 6 TI2VA pipeline."""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from vllm_omni.model_extras.video_generation import VideoGenerationDefaults
# Forwarded to ``OmniDiffusionSamplingParams.extra_args`` by the shared examples
# and the OpenAI video API's ``extra_body``.
KANDINSKY6_EXTRA_BODY_PARAMS = frozenset(
{
# bool, default True: also sample the audio track (TI2VA). False -> video only.
"sample_audio",
# "pretrain" (text-only) | "tail_cond_first_frame" (reference image).
"visual_cond_scheme",
}
)
KANDINSKY6_EXTRA_OUTPUT_PARAMS: frozenset[str] = frozenset()
def get_kandinsky6_video_generation_defaults(
extra_body: Mapping[str, Any] | None = None,
) -> VideoGenerationDefaults:
"""Kandinsky 6 Pro production geometry: 480x864, 125 frames @ 24 fps, 50 steps, CFG 5.
These are defaults, not a fixed contract: any ``4k+1`` frame count and
16-divisible resolution can be requested explicitly, so ``duration_seconds``
is left unset (the serving layer only pins ``num_frames`` for
fixed-duration pipelines such as MAGI-2).
"""
del extra_body
return VideoGenerationDefaults(
width=864,
height=480,
num_frames=125,
num_inference_steps=50,
fps=24.0,
guidance_scale=5.0,
output="kandinsky6_output.mp4",
# ``None`` lets the pipeline apply its own tuned negative prompt.
default_negative_prompt=None,
duration_seconds=None,
)
+12
View File
@@ -40,6 +40,11 @@ from vllm_omni.model_extras.hunyuan_image3 import build_x_to_text_prompt as buil
from vllm_omni.model_extras.hunyuan_image3 import (
validate_ar_tokenizer as validate_hunyuan_image3_ar_tokenizer,
)
from vllm_omni.model_extras.kandinsky6 import (
KANDINSKY6_EXTRA_BODY_PARAMS,
KANDINSKY6_EXTRA_OUTPUT_PARAMS,
get_kandinsky6_video_generation_defaults,
)
from vllm_omni.model_extras.lingbot_video import LINGBOT_VIDEO_EXTRA_BODY_PARAMS
from vllm_omni.model_extras.ltx2 import (
LTX_EXTRA_BODY_PARAMS,
@@ -241,6 +246,13 @@ _EXTRA_SPECS: dict[str, dict[str, Any]] = {
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,
},
"Kandinsky6TI2VAPipeline": {
"extra_body_params": KANDINSKY6_EXTRA_BODY_PARAMS,
"extra_output_params": KANDINSKY6_EXTRA_OUTPUT_PARAMS,
"video_generation_defaults_builder": get_kandinsky6_video_generation_defaults,
# forward() returns uint8 frames already; declared for completeness.
"output_tensor_range": "zero_to_one",
},
"HeliosPyramidPipeline": {
"extra_body_params": HELIOS_EXTRA_BODY_PARAMS,
"extra_output_params": HELIOS_EXTRA_OUTPUT_PARAMS,