Kandinsky6 (#8537)

Signed-off-by: leffff <levnovitskiy@gmail.com> Signed-off-by: Artem Borisov <divotionchannel@gmail.com> Co-authored-by: leffff <levnovitskiy@gmail.com> Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
2026-10-06 15:55:15 +08:00
1 parent 1f63e97a40
commit 5f95115e70
@@ -0,0 +1,247 @@
+# Kandinsky 6 TI2VA — H100 (v1, single GPU, CPU offload)
+
+> Text/image-to-video-and-audio serving with Kandinsky 6 Pro
+
+## Summary
+
+- Vendor: Kandinsky Lab ([`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers))
+- Model: Kandinsky 6 Pro (TI2VA — text/image-to-video-and-audio)
+- Task: Joint text/image-to-video-and-audio generation
+- Mode: Offline (`Omni(...)`) and online serving with the OpenAI-compatible
+  `/v1/videos` API
+- Hardware: 1x NVIDIA H100 80GB HBM3 with component-level CPU offload
+- Maintainer: Community
+
+## When to use this recipe
+
+Use this recipe to serve Kandinsky 6's TI2VA pipeline — a single DiT that
+jointly denoises video and audio latents, conditioned on a text prompt and,
+optionally, a reference image (image-to-video). Tensor parallel, CFG
+parallel, Ulysses, ring, pipeline parallel, HSDP, layerwise offload,
+MagCache, TeaCache, Cache-DiT, FP8, and tiled VAE patch parallel are wired.
+NaviCache and a registered NABLA backend are not. TeaCache coefficients are
+uncalibrated. See "Known limitations".
+
+## Supported model contract
+
+| Task | Entrypoint | Input | Output |
+|---|---|---|---|
+| Text-to-video-and-audio | `vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni` / `Omni(model="kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers")` | text prompt | synchronized MP4 (H.264 video + AAC 44.1 kHz mono audio) |
+| Image-to-video-and-audio | same | text prompt + reference image | synchronized MP4, first frame conditioned on the reference image |
+
+- Audio generation is on by default (`sample_audio=True`); pass
+  `extra_args.sample_audio=false` to generate video only.
+- Batch size is 1 request per generation call (upstream limitation of the
+  ported denoise loop; concurrent requests are still served by vLLM-Omni's
+  own request scheduler, just not batched together within one DiT forward
+  pass yet).
+- Serving defaults follow the Pro production geometry registered in
+  `vllm_omni/model_extras/kandinsky6.py`: **864x480, 125 frames @ 24 fps
+  (5.2 s), 50 steps, CFG 5.0**. Any `4k+1` frame count and 16-divisible
+  resolution can be requested explicitly via `height`/`width`/`num_frames`/
+  `num_inference_steps` — the model is *not* fixed-duration.
+- Audio is always 44.1 kHz; its length is derived from the video duration.
+- Negative prompt defaults to the pipeline's built-in K6 negative prompt
+  when omitted.
+
+## References
+
+- Hub checkpoint: [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
+- Joint video+audio native-model reference: [`recipes/MiniMaxAI/MiniMax-H3.md`](../MiniMaxAI/MiniMax-H3.md)
+- Offline example: [`examples/offline_inference/text_to_video/text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py)
+- Image-to-video example: [`examples/offline_inference/image_to_video/image_to_video.py`](../../examples/offline_inference/image_to_video/image_to_video.py)
+
+## Hardware
+
+- Accelerator model and per-device memory: NVIDIA H100 80GB HBM3
+  (driver 570.133.20)
+- Number of devices: 1 (multi-GPU tensor parallelism is implemented in the
+  transformer's linear layers but not yet validated end-to-end)
+- Device interconnect: N/A (single-device path)
+- Host memory: 1.4 TB installed; the bf16 DiT (~56 GB) plus the text
+  encoders are staged on the host under `--enable-cpu-offload`, so budget
+  at least ~90 GB of free host RAM.
+- Qualification scope: single-GPU T2VA at the Pro default geometry and at a
+  reduced smoke geometry, offline and via `vllm serve`. Numbers below were
+  measured once on the shared box described above; treat them as
+  indicative, not as a benchmark.
+
+## Software environment
+
+- OS: Ubuntu 22.04.5 LTS
+- Python: 3.12.13
+- Driver / runtime: CUDA 12.9 (torch 2.13.0+cu129)
+- vLLM version: 0.29.0
+- vLLM-Omni version or commit: this repository, `vllm_omni/diffusion/models/kandinsky6/`
+- diffusers 0.40.0, transformers 5.14.1
+- Attention: FlashAttention-3 on this box (`flash_attn_interface`). The Pro
+  timings below use it. Without FA3, `attention_engine: "auto"` falls back
+  to PyTorch SDPA; an earlier SDPA Pro run on the same GPU was 26.9 s/it
+  and 1377 s.
+
+## Weights
+
+`Kandinsky6TI2VAPipeline._load_components` loads the public Diffusers layout
+from [`kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers`](https://huggingface.co/kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers)
+(`model_index.json`, `transformer/`, `vae/`, `text_encoder/`, `tokenizer/`,
+`text_encoder_2/`, `tokenizer_2/`, `audio_vae/`, `scheduler/`). Pass that
+repo id (or a local snapshot of it) to `vllm serve` / `Omni(model=...)`.
+The Hub repo is gated; export `HF_TOKEN` if download returns 401.
+
+A raw Kandinsky 6 weight tree is not required. If you already have a local
+Diffusers-layout snapshot, point `--model` at that directory instead of the
+Hub id.
+
+Each weight folder is loaded on its own (`transformer/`, `vae/`, `text_encoder/`, `text_encoder_2/`, `audio_vae/`). The DiT is built with initialization skipped, then the framework loader fills it from `transformer/`.
+
+## Command
+
+```bash
+vllm serve kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --omni \
+  --host 127.0.0.1 --port 8091 \
+  --num-gpus 1 --enable-cpu-offload
+```
+
+`--enable-cpu-offload` enables model-level offload with mutual exclusion
+between the transformer and the two text encoders (they never co-reside on
+the GPU). Without it the bf16 DiT (~56 GB) plus Qwen2.5-VL-7B (~16 GB) plus
+the VAEs and activations do not fit an 80 GB device at the Pro geometry.
+
+Offline (writes an MP4 with the audio track muxed in):
+
+```bash
+python examples/offline_inference/text_to_video/text_to_video.py \
+  --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
+  --prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
+  --seed 42 --enable-cpu-offload --output kandinsky6_t2va.mp4
+# Video only: add --extra-body '{"sample_audio": false}'
+```
+
+Image-to-video-and-audio uses the shared image-to-video example:
+
+```bash
+python examples/offline_inference/image_to_video/image_to_video.py \
+  --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
+  --image first_frame.png \
+  --prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
+  --seed 42 --enable-cpu-offload --output kandinsky6_i2va.mp4
+```
+
+The generic scripts pick up the Pro defaults from
+`vllm_omni/model_extras/kandinsky6.py`.
+
+## Verification
+
+`/v1/videos` takes multipart form fields (not a JSON body). Submit, poll,
+then download:
+
+```bash
+VID=$(curl -s -X POST http://127.0.0.1:8091/v1/videos \
+  -F prompt="A golden retriever runs along a sunny beach, waves crashing" \
+  -F size=864x480 -F num_frames=125 -F num_inference_steps=50 -F seed=42 \
+  | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")
+
+# Poll until "status": "completed"
+curl -s http://127.0.0.1:8091/v1/videos/$VID
+
+curl -s -o kandinsky6.mp4 http://127.0.0.1:8091/v1/videos/$VID/content
+# Expect an MP4 with an H.264 video stream (125 frames, 24 fps, 864x480)
+# and an AAC audio stream (44100 Hz, mono) of matching duration.
+```
+
+Smoke-size request for quick validation (~30 s end-to-end on one H100):
+`-F size=512x320 -F num_frames=25 -F num_inference_steps=10`.
+
+## Measurements
+
+Single H100 80GB, seed 42, prompt
+"A golden retriever runs along a sunny beach, waves crashing, cinematic footage",
+negative prompt left at the pipeline default, MagCache off, prompt expansion
+off. Pro geometry: 864x480, 125 frames, 50 steps, CFG 5.0, audio on.
+
+vLLM-Omni: `text_to_video.py --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers --enable-cpu-offload --model-class-name Kandinsky6TI2VAPipeline`,
+FlashAttention-3.
+
+| Wall / generation | Step time | Peak GPU | After load |
+|---|---|---|---|
+| 751.7 s generation (12.5 min); engine init 41 s | 14.4 s/it average | 78.39 GiB reserved (78.0 GiB nvidia-smi) | 17.74 GiB |
+
+An earlier SDPA vLLM-Omni Pro run on the same GPU was 26.9 s/it and 1377 s,
+peak reserved 74.97 GiB. A 512x320 / 25-frame / 10-step SDPA smoke was 1.4 s/it
+and 29.5 s at the same reserved peak.
+
+The MP4 is H.264 864x480, 125 frames at 24 fps (5.22 s), plus AAC 44.1 kHz
+mono (230400 samples): a golden retriever on a sunny beach with breaking waves.
+
+vLLM-Omni's VAE decode logged two failed CUDA allocations (7.15 GiB, then
+4.27 GiB) and still wrote a complete file.
+
+Audio is peak-normalized to full scale by default; pass
+`extra_args.audio_normalization="clip"` to keep the raw decoded amplitude
+instead.
+
+## Notes
+
+- Key flags: `extra_args.sample_audio` (bool, default true),
+  `extra_args.audio_normalization` (`"normalize"` default | `"clip"`),
+  `extra_args.visual_cond_scheme` (defaults to `pretrain` for text-only,
+  `tail_cond_first_frame` when an image is supplied).
+- The post-process payload is a flat dict (`video`, `audio` as float32 in
+  `[-1, 1]`, `audio_sample_rate=44100`, `fps`) so the serving muxer picks
+  up the correct sample rate.
+- Acceleration that is wired:
+  - Tensor parallel: the DiT loader narrows full checkpoint tensors onto
+    each rank's `ColumnParallelLinear` / `RowParallelLinear` shard.
+  - CFG parallel (`--cfg-parallel-size 2`): rank 0 runs the conditional
+    DiT, rank 1 the unconditional one, then both all-gather video and
+    audio velocities.
+  - HSDP: `_hsdp_shard_conditions` covers the indexed transformer blocks.
+    Do not combine with tensor parallel.
+  - Sequence parallel: visual tokens (and their RoPE) are sharded;
+    Ulysses all-to-all runs inside visual self-attention, Ring uses the
+    PyTorch ring kernel. Text stays replicated. Audio queries gather the
+    full visual sequence.
+  - Pipeline parallel: `visual_transformer_blocks` are split with
+    `PPMissingLayer`. Embeddings stay on the first stage, output heads on
+    the last. Activations and the final velocity are sent between stages.
+  - VAE patch parallel: Hunyuan decode splits the latent width when
+    `vae_patch_parallel_size > 1` and tiling is on.
+  - MagCache / TeaCache / Cache-DiT are registered. MagCache reuses the
+    Pro `mag_ratios`. TeaCache coefficients are **uncalibrated** (copied
+    from Qwen-Image) so skips are not quality-neutral. Cache-DiT targets
+    `visual_transformer_blocks` only and falls back to the same step cache
+    if the fused `(video, audio)` block return is rejected.
+  - FP8: online quantization runs in the DiT loader via
+    `process_weights_after_loading` after the sharded assign.
+  - `attention_engine: auto` uses FlashAttention-3 when
+    `flash_attn_interface` imports.
+- Known limitations:
+  - NABLA sparse attention is not registered in the framework attention
+    backend. NaviCache is not wired.
+  - Prompt expansion, NF4 Qwen, and the super-resolution cascade are not
+    part of this port.
+  - Distributed layerwise offload still uses one process unless another
+    parallel axis sets `world_size > 1` (for example tensor parallel).
+  - `_load_components` loads the Diffusers checkpoint directly rather than the
+    streamed-prefetch loader.
+
+## Supported features
+
+| Feature | Status | Guide |
+|---|---|---|
+| Text-to-video-and-audio | Supported (validated on H100) | — |
+| Image-to-video-and-audio | Supported (not yet validated on hardware) | — |
+| CPU offload (`--enable-cpu-offload`) | Supported (required on 80 GB) | — |
+| Layerwise offload (`--enable-layerwise-offload`) | Supported (about 36 GiB peak on the smoke geometry) | — |
+| Tensor parallelism | Wired (sharded DiT load) | — |
+| CFG parallelism | Wired (`--cfg-parallel-size 2`) | — |
+| Ulysses / Ring sequence parallel | Wired (visual tokens only) | — |
+| Pipeline parallelism | Wired (visual block split) | — |
+| HSDP | Wired (not with tensor parallel) | — |
+| VAE patch parallel | Wired (width-split decode) | — |
+| MagCache | Wired (Pro ratios, step reuse) | — |
+| TeaCache | Wired, coefficients uncalibrated | — |
+| Cache-DiT | Wired on `visual_transformer_blocks` | — |
+| FP8 | Wired (online quant; smoke peak about 46 GiB vs 75 GiB bf16) | [`FP8 quantization`](../../docs/user_guide/quantization/fp8.md) |
+| NaviCache | Not supported | — |
+| LoRA | Not supported | — |
@@ -64,6 +64,7 @@ recipes/
 | [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) |
 | [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
 | [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB / 1x Intel Arc Pro B70 32GB |
+| [`Kandinsky/Kandinsky6-TI2VA.md`](./Kandinsky/Kandinsky6-TI2VA.md) | Text/image-to-video-and-audio, offline and `/v1/videos` | 1x H100 80GB with CPU offload; 2x H100 for parallel modes |
 | [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU / 1x Intel Arc Pro B70 32GB |
 | [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified / 1x Intel Arc Pro B70 32GB |
 | [`m-a-p/SheetSage2-H200.md`](./m-a-p/SheetSage2-H200.md) | Official audio-to-score preprocessing and YuE2 request export | 1x H200 141GB |