From 62ebad2d7cd79da24653110d8043cc13fb744328 Mon Sep 17 00:00:00 2001 From: Joshna-Medisetty Date: Fri, 25 Sep 2026 13:42:08 -0700 Subject: [PATCH] Add Intel Arc BMG recipes for validated P0 diffusion models. (#7537) Signed-off-by: Joshna-Medisetty --- docs/models/supported_models.md | 42 +++++------ recipes/LTX/LTX-2.5.md | 28 +++++++ recipes/LTX/LTX-2.md | 57 ++++++++++++++ recipes/Qwen/Qwen-Image.md | 66 ++++++++++++++++- recipes/Qwen/Qwen3-Omni.md | 56 ++++++++++++-- recipes/README.md | 17 +++-- recipes/StabilityAI/Stable-Diffusion-3.5.md | 36 +++++++++ recipes/Tencent/HunyuanVideo-1.5.md | 72 ++++++++++++++++++ recipes/Tongyi-MAI/Z-Image-Turbo.md | 68 +++++++++++++++++ recipes/Wan-AI/Wan2.2-T2V.md | 82 ++++++++++++++++++--- recipes/black-forest-labs/FLUX.2-klein.md | 69 +++++++++++++++++ recipes/krea/Krea-2.md | 55 ++++++++++++++ 12 files changed, 603 insertions(+), 45 deletions(-) create mode 100644 recipes/Tencent/HunyuanVideo-1.5.md create mode 100644 recipes/Tongyi-MAI/Z-Image-Turbo.md create mode 100644 recipes/black-forest-labs/FLUX.2-klein.md diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index 1d9d52f77..1ac585526 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -19,9 +19,9 @@ th { | Architecture | Models | Example HF Models | NVIDIA GPU | AMD GPU | Ascend NPU | Intel GPU | Recipe | | -------------- | -------- | ------------------- | ------------ | --------- | ----- | ----------- | -------- | -| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) | +| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) | | `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | -| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen1) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | | — | +| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen1) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | ✅︎ | — | | `BagelForConditionalGeneration` | BAGEL (DiT-only) | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ | | ✅︎ | — | | `InternVLAA1Pipeline` | InternVLA-A1 | `InternRobotics/InternVLA-A1-3B` | ✅︎ | ✅︎ | | | — | | `Gr00tN1d7Pipeline` | GR00T N1.7 | `nvidia/GR00T-N1.7-3B` | ✅︎ | | | | — | @@ -33,9 +33,9 @@ th { | `QwenImageLayeredPipeline` | Qwen-Image-Layered | `Qwen/Qwen-Image-Layered` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `QwenImageEditPlusPipeline` | Qwen-Image-Edit-2511 | `Qwen/Qwen-Image-Edit-2511` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `GlmImagePipeline` | GLM-Image | `zai-org/GLM-Image` | ✅︎ | ✅︎ | | | — | -| `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | -| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | ✅︎ | — | -| `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | +| `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tongyi-MAI/Z-Image-Turbo.md) | +| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/krea/Krea-2.md) | +| `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Wan-AI/Wan2.2-T2V.md) | | `WanImageToVideoPipeline` | Wan2.2-I2V | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `LingBotWorldCausalDMDPipeline` | LingBot-World 2.0 v1 (experimental) | `robbyant/lingbot-world-v2-14b-causal-fast-diffusers` | Experimental | | | | — | | `Cosmos3OmniDiffusersPipeline` | Cosmos3 T2I, T2V, I2V, V2V, T2V with sound, action policy | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | @@ -43,16 +43,16 @@ th { | `Wan22S2VPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | ✅︎ | — | -| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | | — | -| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | | — | -| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | | — | -| `LTX2DistilledTwoStagePipeline` | LTX-2 / LTX-2.3 merged-distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` + matching Lightricks upsampler | ✅︎ | ✅︎ | | | — | -| `LTX2Pipeline` | LTX-2.5 Full/SFT one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) | +| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) | +| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) | +| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) | +| `LTX2DistilledTwoStagePipeline` | LTX-2 / LTX-2.3 merged-distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` + matching Lightricks upsampler | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) | +| `LTX2Pipeline` | LTX-2.5 Full/SFT one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) | | `LTX2TwoStagePipeline` | LTX-2.5 Full/SFT two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` + `Lightricks/LTX-2.5` LoRA | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) | | `LTX2DistilledOneStagePipeline` | LTX-2.5 merged-distilled one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) | | `LTX2DistilledTwoStagePipeline` | LTX-2.5 merged-distilled two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) | | `LingBotVideoPipeline` | LingBot-Video dense and MoE T2I, T2V, TI2V | `robbyant/lingbot-video-dense-1.3b`, `robbyant/lingbot-video-moe-30b-a3b` | ✅︎ | | | | — | -| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) | +| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) | | `SanaImageToVideoPipeline` | SANA-Video-2B I2V (native and Diffusers adapter validated at 480p/720p) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) | | `MiniMaxH3Pipeline` | MiniMax H3 T2VA, first/last-keyframe FL2VA, and mixed-reference Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | ✅︎ | | | [Published](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) | | `Magi2Pipeline` | MAGI-2 native Preview-stage T2VA and I2VA (272p/540p) | `sand-ai/MAGI-2-preview` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SandAI/MAGI-2-preview-L20X.md) | @@ -65,18 +65,18 @@ th { | `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `LongCatVideoAvatarPipeline` | LongCat-Video-Avatar-1.5 A2V/AI2V (native single-speaker and multi-speaker AI2V/AVC) | `meituan-longcat/LongCat-Video-Avatar-1.5` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/meituan-longcat/LongCat-Video-Avatar-1.5.md) | | `BooguImagePipeline` | Boogu-Image | `Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit`, `Boogu/Boogu-Image-0.1-Edit-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) | -| `BooguImageTurboPipeline` | Boogu-Image Turbo distilled T2I | `Boogu/Boogu-Image-0.1-Turbo` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) | +| `BooguImageTurboPipeline` | Boogu-Image Turbo distilled T2I | `Boogu/Boogu-Image-0.1-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) | | `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | -| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — | +| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/StabilityAI/Stable-Diffusion-3.5.md) | | `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — | | `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) | | `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — | | `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — | | `BreezeForConditionalGeneration` | Breeze-TTS-2 (voice design, voice clone, voice direction, CFG) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) | -| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — | +| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | ✅︎ | — | | `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — | -| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | -| `FluxKontextPipeline` | FLUX.1-Kontext-dev | `black-forest-labs/FLUX.1-Kontext-dev` | ✅︎ | ✅︎ | | | — | +| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/black-forest-labs/FLUX.2-klein.md) | +| `FluxKontextPipeline` | FLUX.1-Kontext-dev | `black-forest-labs/FLUX.1-Kontext-dev` | ✅︎ | ✅︎ | | ✅︎ | — | | `FluxPipeline` | FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | ✅︎ | ✅︎ | | ✅︎ | — | | `FluxPipeline` | FLUX.1-schnell | `black-forest-labs/FLUX.1-schnell` | ✅︎ | ✅︎ | | ✅︎ | — | | `OmniGen2Pipeline` | OmniGen2 | `OmniGen2/OmniGen2` | ✅︎ | ✅︎ | | ✅︎ | — | @@ -97,13 +97,13 @@ th { | `NextStep11Pipeline` | NextStep-1.1 | `stepfun-ai/NextStep-1.1` | ✅︎ | ✅︎ | | ✅︎ | — | | `MiMoAudioModel` | MiMo-Audio-7B-Instruct | `XiaomiMiMo/MiMo-Audio-7B-Instruct` | ✅︎ | ✅︎ | | | — | | `MiMoV2ASRForCausalLM` | MiMo-V2.5-ASR | `XiaomiMiMo/MiMo-V2.5-ASR` | ✅︎ | ✅︎ | | | — | -| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | | — | +| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | ✅︎ | — | | `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | ✅︎ | — | -| `Audio8TTSSlowARForConditionalGeneration` | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✅︎ | | | | — | -| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only), SenseNova-U1.5 | `SenseNova/SenseNova-U1-8B-MoT`, `sensenova/SenseNova-U1.5-8B-MoT` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.5.md) | +| `Audio8TTSSlowARForConditionalGeneration` | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✅︎ | | | ✅︎ | — | +| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only), SenseNova-U1.5 | `SenseNova/SenseNova-U1-8B-MoT`, `sensenova/SenseNova-U1.5-8B-MoT` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.5.md) | | `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — | -| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | | — | -| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — | +| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) | +| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) | | `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — | | `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — | | `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) | diff --git a/recipes/LTX/LTX-2.5.md b/recipes/LTX/LTX-2.5.md index 538eed1ce..ca9a83322 100644 --- a/recipes/LTX/LTX-2.5.md +++ b/recipes/LTX/LTX-2.5.md @@ -89,6 +89,7 @@ Install matching vLLM and vLLM-Omni versions, and ensure `ffmpeg` and | NVIDIA B200 or H200 | Capacity-based recommendation; not yet verified | All four canonical pipelines | | NVIDIA GB200 or GB300 | Capacity-based recommendation; not yet verified | All four canonical pipelines | | NVIDIA H100 80 GB | FP8 recipe | Distilled one-stage at 960x544 | +| Intel Arc Pro B70 32 GB | Verified | Offline Full/SFT one-stage T2V at 768x512 | The 1920x1088 two-stage examples require about 114 GB of peak GPU memory. 80 GB GPUs do not have enough safety margin for the canonical two-stage @@ -133,6 +134,33 @@ SP attention head counts must be divisible by `ulysses_degree`. Use each phase's `H, W` (half/full resolution for two-stage); non-divisible shapes fail. Each GPU must still fit the resident weights. Warm up with the production shape. +### 1x Intel Arc Pro B70 32 GB (XPU) + +Offline Full/SFT one-stage T2V at 768x512, 17 frames. Select the legacy +convolutional VAE with `ltx2_use_conv_vae`; the default DiffVAE decoder loads a +NATTEN kernel, which has no XPU build. FP8 with layerwise offload streams both +the transformer and the Gemma-4 text encoder. + +Environment: torch 2.14.0+xpu, vLLM 0.29.0, vLLM-Omni `main` at `d8d162d8`. + +```bash +python examples/offline_inference/text_to_video/text_to_video.py \ + --model Lightricks/LTX-2.5-Diffusers \ + --model-class-name LTX2Pipeline \ + --stage-overrides '{"0":{"extras":{"ltx2_use_conv_vae":true}}}' \ + --prompt "A serene lakeside sunrise with mist over the water." \ + --num-frames 17 \ + --quantization fp8 \ + --enable-layerwise-offload \ + --vae-use-tiling \ + --vae-use-slicing \ + --enforce-eager \ + --output ltx25_output.mp4 +``` + +Peaks at 16.3 GiB in 163 s. Two-stage, distilled, and I2V paths were not +exercised on this hardware. + ## Offline inference Choose values from the pipeline table. For example, the distilled two-stage diff --git a/recipes/LTX/LTX-2.md b/recipes/LTX/LTX-2.md index b84bebd66..408e576dd 100644 --- a/recipes/LTX/LTX-2.md +++ b/recipes/LTX/LTX-2.md @@ -165,6 +165,63 @@ curl -X POST http://localhost:8000/v1/videos/sync \ Use `image_reference` for a URL or JSON-safe image reference. Do not provide it together with `input_reference`. +## Hardware Support + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +Offline one-stage T2V at 768x512, 17 frames. FP8 with layerwise offload streams +both the transformer and the Gemma text encoder, which otherwise stays resident +beside it. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.14.0+xpu +- vLLM: 0.29.0 +- vLLM-Omni: `main` at `d8d162d8` + +#### Command + +```bash +python examples/offline_inference/text_to_video/text_to_video.py \ + --model diffusers/LTX-2.3-Diffusers \ + --prompt "A serene lakeside sunrise with mist over the water." \ + --num-frames 17 \ + --quantization fp8 \ + --enable-layerwise-offload \ + --vae-use-tiling \ + --vae-use-slicing \ + --enforce-eager \ + --output ltx23_output.mp4 +``` + +The same flags cover every one-stage checkpoint; only `--model` changes. The +distilled checkpoints additionally need their class and an explicit size, +because their preset defaults to 1536x1024: + +```bash + --model diffusers/LTX-2.3-Distilled-Diffusers \ + --model-class-name LTX2DistilledOneStagePipeline \ + --height 512 --width 768 +``` + +| Checkpoint | Peak | Time | +| --- | ---: | ---: | +| `Lightricks/LTX-2` | 14.8 GiB | 160 s | +| `diffusers/LTX-2.3-Diffusers` | 16.4 GiB | 167 s | +| `dg845/LTX-2.3-Diffusers` | 16.4 GiB | 155 s | +| `diffusers/LTX-2.3-Distilled-Diffusers` | 14.9 GiB | 128 s | + +#### Verification + +Each command writes its `--output` MP4. Confirm it decodes and that the sampled +frames match the prompt. + + + ## Guidance One-stage and ordinary Stage 1 support independent video/audio CFG, diff --git a/recipes/Qwen/Qwen-Image.md b/recipes/Qwen/Qwen-Image.md index c40d71c1e..6cd0e7704 100644 --- a/recipes/Qwen/Qwen-Image.md +++ b/recipes/Qwen/Qwen-Image.md @@ -7,7 +7,8 @@ - Vendor: Qwen - Model: `Qwen/Qwen-Image` - Task: Text-to-image generation -- Mode: Online serving with optional step-wise continuous batching +- Mode: Online serving with optional step-wise continuous batching; offline + inference - Maintainer: Community ## When to use this recipe @@ -25,11 +26,14 @@ the benchmark assets already bundled in this repository. [`examples/online_serving/text_to_image/README.md`](../../examples/online_serving/text_to_image/README.md) - Related benchmark: [`benchmarks/diffusion/diffusion_benchmark_serving.py`](../../benchmarks/diffusion/diffusion_benchmark_serving.py) +- Offline inference example used by the XPU section: + [`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py) ## Hardware Support -This recipe currently documents one CUDA GPU serving configuration. Extend it -with more hardware sections as community validation lands. +This recipe documents CUDA GPU serving configurations and one Intel XPU +offline-inference configuration. Extend it with more hardware sections as +community validation lands. ## GPU @@ -235,3 +239,59 @@ CUTLASS was the fastest validated backend for this checkpoint. - The benchmark table intentionally reports Qwen-Image-2512 as concurrency-1 request-level data; do not compare it directly with an online HTTP concurrency benchmark. + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +Offline text-to-image at 1024x1024, with layerwise offload streaming the +transformer blocks so the 20B model fits one card. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model Qwen/Qwen-Image \ + --prompt "a cup of coffee on the table" \ + --num-inference-steps 50 \ + --enable-layerwise-offload \ + --vae-use-tiling \ + --vae-use-slicing \ + --enforce-eager \ + --output qwen_image_output.png +``` + +Given spare cards, `--enable-cpu-offload` shards the transformer instead of +streaming it. It shortens the denoise loop but not the run, because the +per-request model swap costs more than the loop saves. + +```bash + --tensor-parallel-size 4 \ + --enable-cpu-offload +``` + +| Offload | Cards | Peak VRAM | Time | +| --- | ---: | ---: | ---: | +| `--enable-layerwise-offload` | 1 | 16.6 GiB | 50 s | +| `--enable-cpu-offload` | 2 | 26.4 GiB | 85 s | +| `--enable-cpu-offload` | 4 | 19.8 GiB | 70 s | +| `--enable-cpu-offload` | 8 | 16.9 GiB | 87 s | + +#### Verification + +Confirm `qwen_image_output.png` is written and looks coherent for the prompt. + +#### Notes + +- Memory usage: peak 16.6 GiB, about 55 s per image. +- Known limitations: only offline generation was qualified. Step-wise continuous + batching, ModelOpt quantization, and online serving are out of scope for this + profile. diff --git a/recipes/Qwen/Qwen3-Omni.md b/recipes/Qwen/Qwen3-Omni.md index 0ec67158c..e319dddfb 100644 --- a/recipes/Qwen/Qwen3-Omni.md +++ b/recipes/Qwen/Qwen3-Omni.md @@ -53,7 +53,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ --deploy-config /path/to/your_qwen3_omni_overrides.yaml ``` -#### Runtime tuning +### Runtime tuning Prefer CLI overrides for day-to-day tuning: @@ -75,7 +75,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ }' ``` -#### Stage-based launch (one stage per process) +### Stage-based launch (one stage per process) Use three terminals (one per stage). Start with the default commands below, then add `--max-num-seqs` only if you need explicit per-stage concurrency control. @@ -135,7 +135,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ If you use custom deploy YAML, add `--deploy-config` to each stage command. -#### Verification +### Verification After server startup, run a multimodal example client: @@ -185,7 +185,7 @@ python examples/online_serving/qwen3_omni/openai_realtime_client.py \ --output-wav realtime_output.wav ``` -#### Benchmark with `vllm bench` +### Benchmark with `vllm bench` After the server is up, you can run online serving benchmarks with `vllm bench serve --omni`. @@ -240,8 +240,54 @@ vllm bench serve \ --percentile-metrics ttft,tpot,itl,e2el ``` -#### Notes +### Notes - `/v1/realtime` is unsupported while `async_chunk` is enabled. - The default deploy uses `SharedMemoryConnector`; this is for single-host stage wiring. + +## XPU + +### 5x Intel Arc Pro B70 (32 GB) + +Offline end-to-end text query returning both text and speech. The 62 GiB of +BF16 thinker weights need four cards, so the `xpu:` block of +`vllm_omni/deploy/qwen3_omni_moe.yaml` runs the thinker at TP=4 on cards 0-3 +and the talker and code2wav on card 4. It is the registry default here, so +`--deploy-config` is not needed. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +The raised timeouts absorb the first decode step, which JIT-compiles the +Triton MoE kernels. + +```bash +VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ +python examples/offline_inference/qwen3_omni/end2end.py \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --query-type text \ + --stage-init-timeout 1800 \ + --init-timeout 1800 \ + --output-wav qwen3omni_output +``` + +#### Verification + +`qwen3omni_output/` holds one `.txt` answer and the matching `.wav`. Confirm +the transcript of the audio tracks the text response. + +#### Notes + +- Runtime: about 260 s for a single text query. +- The thinker MoE layers have no tuned Triton config for this shape and log + `Using default MoE config. Performance might be sub-optimal!`. +- Known limitations: only `--query-type text` was qualified. The audio, image, + and video query types and online serving are out of scope for this profile. diff --git a/recipes/README.md b/recipes/README.md index 22174a9bb..c6d02f0c5 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -45,6 +45,7 @@ recipes/ | [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB | | [`Bagel/BAGEL-7B-MoT.md`](./Bagel/BAGEL-7B-MoT.md) | Text-to-image with shared online/offline examples | 1x A100 80GB / 2x CUDA GPUs | | [`black-forest-labs/FLUX.1-schnell.md`](./black-forest-labs/FLUX.1-schnell.md) | Offline text-to-image generation with CPU offload | 1x NVIDIA GeForce RTX 5090 32GB | +| [`black-forest-labs/FLUX.2-klein.md`](./black-forest-labs/FLUX.2-klein.md) | Offline image editing (FLUX.2-klein-4B) | 1x Intel Arc Pro B70 32GB | | [`Boogu/Boogu-Image.md`](./Boogu/Boogu-Image.md) | Text-to-image online serving (Boogu-Image-0.1-Base) | 1x A100/H100 40GB+ | | [`BosonAI/Higgs-Audio-V3-TTS.md`](./BosonAI/Higgs-Audio-V3-TTS.md) | Online + offline multilingual TTS with voice cloning | 1x H100 80GB | | [`BreezeBlue/Breeze-TTS-2.md`](./BreezeBlue/Breeze-TTS-2.md) | Online + offline TTS (plain / voice design / voice clone / voice direction, streaming PCM) | 1x NVIDIA L20 48GB | @@ -62,9 +63,9 @@ recipes/ | [`MiniMaxAI/MiniMax-H3.md`](./MiniMaxAI/MiniMax-H3.md) | T2VA, FL2VA, image+audio Ref2VA, and multi-video Ref2VA serving | 1x GPU with CPU offload / 4x B300 | | [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) | | [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) | -| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB | -| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU | -| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified | +| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB / 1x Intel Arc Pro B70 32GB | +| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU / 1x Intel Arc Pro B70 32GB | +| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified / 1x Intel Arc Pro B70 32GB | | [`MammothModa2/MammothModa2.md`](./MammothModa2/MammothModa2.md) | Preview and Dev text-to-image (AR → DiT); Dev text/image understanding | Preview: 1x L40S 48GB / 1x ≥40GB GPU; Dev: 1x NVIDIA GPU with sufficient cache headroom | | [`NVIDIA/SANA-Video-2B.md`](./NVIDIA/SANA-Video-2B.md) | Native and Diffusers-adapter 480p/720p T2V+I2V | 1x RTX 5090 32GB | | [`meituan-longcat/LongCat-Video-Avatar-1.5.md`](./meituan-longcat/LongCat-Video-Avatar-1.5.md) | Audio-driven avatar video generation (AT2V / AI2V, single- and multi-speaker, AVC continuation) | 1x H100 80GB | @@ -82,10 +83,10 @@ recipes/ | [`NVIDIA/Nemotron-Labs-Audex.md`](./NVIDIA/Nemotron-Labs-Audex.md) | TTS / text-to-audio / audio understanding / cascaded S2S (Audex 2B + 30B-A3B) | 1x H100 80GB | | [`NVIDIA/NemotronLabs-VoiceChat.md`](./NVIDIA/NemotronLabs-VoiceChat.md) | Offline speech-to-speech voice chat (11B, frame-locked 12.5 Hz, 3-stage thinker/talker/code2wav) | 1x H100 80GB | | [`NVIDIA/SANA-WM.md`](./NVIDIA/SANA-WM.md) | First-frame image-to-video serving with camera control | 1x 24GB+ CUDA GPU; ~40GB disk | -| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 | +| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4; offline text-to-image on XPU | 1x A100 80GB / 2x B200 / 1x Intel Arc Pro B70 32GB | | [`Qwen/Qwen-Image-2512.md`](./Qwen/Qwen-Image-2512.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt FP8 / mixed FP8/NVFP4 | 1x A800 80GB / 2x B200 | | [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB | -| [`Qwen/Qwen3-Omni.md`](./Qwen/Qwen3-Omni.md) | Online serving for multimodal chat | 1x A100 80GB | +| [`Qwen/Qwen3-Omni.md`](./Qwen/Qwen3-Omni.md) | Online serving for multimodal chat; offline end-to-end text-to-speech on XPU | 1x A100 80GB / 5x Intel Arc Pro B70 32GB | | [`Qwen/Qwen3-TTS.md`](./Qwen/Qwen3-TTS.md) | Text-to-speech serving (CustomVoice / VoiceDesign / Base) | 1x H100/A100 80GB / 1x AMD MI300X | | [`SenseNova/SenseNova-U1.md`](./SenseNova/SenseNova-U1.md) | Unified image generation and understanding | 1x or 2x H200 / 1x AMD MI300X | | [`SenseNova/SenseNova-U1.5.md`](./SenseNova/SenseNova-U1.5.md) | Unified image generation and understanding, with an 8-step distilled LoRA | 1x A800 80GB | @@ -93,13 +94,15 @@ recipes/ | [`Tencent/AuK-H100.md`](./Tencent/AuK-H100.md) | Offline instruction-driven TTS, speech editing, enhancement and separation (AuK base and Flash) | 1x H100 80GB | | [`Tencent/Covo-Audio-Chat.md`](./Tencent/Covo-Audio-Chat.md) | Online serving for audio chat | 1x A100 80GB | | [`Tencent/HunyuanImage-3.0-Instruct.md`](./Tencent/HunyuanImage-3.0-Instruct.md) | DiT-only text-to-image serving and benchmark, including ModelOpt mixed FP8/NVFP4 | 4x H100/H800 80GB / 2x B200 | -| [`Wan-AI/Wan2.2-T2V.md`](./Wan-AI/Wan2.2-T2V.md) | Text-to-video serving (Wan2.2 14B) with Skip-Softmax sparse attention | 1x datacenter Blackwell (B200/B300) | +| [`Tencent/HunyuanVideo-1.5.md`](./Tencent/HunyuanVideo-1.5.md) | Offline text-to-video (HunyuanVideo-1.5 Diffusers) | 1x Intel Arc Pro B70 32GB | +| [`Tongyi-MAI/Z-Image-Turbo.md`](./Tongyi-MAI/Z-Image-Turbo.md) | Offline few-step text-to-image (Z-Image-Turbo) | 1x Intel Arc Pro B70 32GB | +| [`Wan-AI/Wan2.2-T2V.md`](./Wan-AI/Wan2.2-T2V.md) | Text-to-video serving (Wan2.2 14B) with Skip-Softmax sparse attention | 8x H20/H100/A100 / 1x datacenter Blackwell (B200/B300) / 1x Intel Arc Pro B70 32GB | | [`Wan-AI/Wan2.2-TI2V-5B-A100.md`](./Wan-AI/Wan2.2-TI2V-5B-A100.md) | Unified 720p text-to-video and image-to-video serving (Wan2.2 5B) | 1x A100-SXM4-80GB | | [`Wan-AI/Wan2.2-I2V.md`](./Wan-AI/Wan2.2-I2V.md) | Image-to-video serving (Wan2.2 14B) | 8x Ascend NPU (A2/A3) | | [`Wan-AI/Wan2.2-S2V.md`](./Wan-AI/Wan2.2-S2V.md) | Speech-to-video serving (Wan2.2 14B) | 2x A100/H100 80GB | | [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) | | [`XiaomiMiMo/MiMo-Audio.md`](./XiaomiMiMo/MiMo-Audio.md) | Offline + online omni audio (TTS, ASR, dialogue) | 1x RTX 5090 / 5090D 32GB | -| [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB | +| [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB / 1x Intel Arc Pro B70 32GB | | [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB / 2x non-standard RTX 4090 48GB (49,140 MiB each) | | [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB | | [`JD/JoyAI-VL-Interaction.md`](./JD/JoyAI-VL-Interaction.md) | Real-time streaming video-language interaction (proactive speak/silence/delegate) | 1x GPU 24GB+ | diff --git a/recipes/StabilityAI/Stable-Diffusion-3.5.md b/recipes/StabilityAI/Stable-Diffusion-3.5.md index 7c8f747aa..5012c144e 100644 --- a/recipes/StabilityAI/Stable-Diffusion-3.5.md +++ b/recipes/StabilityAI/Stable-Diffusion-3.5.md @@ -137,3 +137,39 @@ print('saved output.png') - Image quality is noticeably better than medium, with sharper details. - `negative_prompt` requires `guidance_scale > 1` to take effect; SD3 defaults to 1.0 (no CFG). - 2048×2048 fits within 48 GB but leaves little headroom (~45.5 GiB used out of 49 GiB). + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) — stable-diffusion-3.5-medium + +Offline text-to-image at 1024x1024 with BF16 weights fully resident. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model stabilityai/stable-diffusion-3.5-medium \ + --prompt "a sunset over mountains, photorealistic" \ + --num-inference-steps 28 \ + --vae-use-tiling \ + --output sd35_medium_output.png +``` + +#### Verification + +Confirm `sd35_medium_output.png` is written and matches the prompt. + +#### Notes + +- Memory usage: peak 19.5 GiB against 29.8 GiB free. +- Known limitations: only `stable-diffusion-3.5-medium` offline generation was + qualified. `stable-diffusion-3.5-large` and online serving are out of scope + for this profile. diff --git a/recipes/Tencent/HunyuanVideo-1.5.md b/recipes/Tencent/HunyuanVideo-1.5.md new file mode 100644 index 000000000..f4a359989 --- /dev/null +++ b/recipes/Tencent/HunyuanVideo-1.5.md @@ -0,0 +1,72 @@ +# HunyuanVideo 1.5 + +> Text-to-video with the HunyuanVideo-1.5 Diffusers checkpoints + +## Summary + +- Vendor: Tencent (community Diffusers ports) +- Model: `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` +- Task: Text-to-video +- Mode: Offline inference +- Maintainer: Community + +## When to use this recipe + +Use this recipe to generate a short clip with HunyuanVideo-1.5 on a memory- +constrained card, trading output size for capacity. + +## References + +- Model card: + +- Shared runnable example: + [`examples/offline_inference/text_to_video/text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py) +- CPU offload guide: + [`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md) + +## Hardware Support + +This recipe documents one validated Intel XPU configuration. Extend it with +more hardware sections as community validation lands. + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +BF16 weights with model-level CPU offload at a reduced 320x240, 17-frame +shape. The 480p and 720p T2V and I2V checkpoints all use the same command; only +`--model` changes. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_video/text_to_video.py \ + --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v \ + --prompt "A serene lakeside sunrise with mist over the water." \ + --height 240 --width 320 --num-frames 17 \ + --num-inference-steps 50 \ + --guidance-scale 6.0 \ + --enable-cpu-offload \ + --vae-use-tiling \ + --vae-use-slicing \ + --output hunyuan_video_15_output.mp4 +``` + +#### Verification + +Confirm `hunyuan_video_15_output.mp4` decodes and that the sampled frames match +the prompt. + +#### Notes + +- Memory usage: 18.3 GiB loaded, 25.7 GiB peak, about 39 s per clip. +- Known limitations: the native 720p geometry does not fit a 32 GB card with + CPU offload. Online serving is out of scope for this profile. diff --git a/recipes/Tongyi-MAI/Z-Image-Turbo.md b/recipes/Tongyi-MAI/Z-Image-Turbo.md new file mode 100644 index 000000000..c015327c5 --- /dev/null +++ b/recipes/Tongyi-MAI/Z-Image-Turbo.md @@ -0,0 +1,68 @@ +# Z-Image Turbo + +> Few-step text-to-image with `Tongyi-MAI/Z-Image-Turbo` + +## Summary + +- Vendor: Tongyi-MAI +- Model: `Tongyi-MAI/Z-Image-Turbo` +- Task: Text-to-image +- Mode: Offline inference +- Maintainer: Community + +## When to use this recipe + +Use this recipe to generate 1024x1024 images with the distilled Z-Image Turbo +checkpoint through the shared offline text-to-image example. + +## References + +- Model card: +- Shared runnable example: + [`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py) +- CPU offload guide: + [`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md) + +## Hardware Support + +This recipe documents one validated Intel XPU configuration. Extend it with +more hardware sections as community validation lands. + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +BF16 weights with model-level CPU offload and VAE tiling/slicing, 25 steps. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model Tongyi-MAI/Z-Image-Turbo \ + --prompt "a cup of coffee on the table" \ + --num-inference-steps 25 \ + --enable-cpu-offload \ + --vae-use-tiling \ + --vae-use-slicing \ + --enforce-eager \ + --output z_image_turbo_output.png +``` + +#### Verification + +Confirm `z_image_turbo_output.png` is written as a 1024x1024 PNG matching the +prompt. + +#### Notes + +- Memory usage: peak 15.6 GiB, about 58 s per image. +- Known limitations: only offline generation was qualified. Online serving is + out of scope for this profile. diff --git a/recipes/Wan-AI/Wan2.2-T2V.md b/recipes/Wan-AI/Wan2.2-T2V.md index 640129a47..c80032a06 100644 --- a/recipes/Wan-AI/Wan2.2-T2V.md +++ b/recipes/Wan-AI/Wan2.2-T2V.md @@ -85,14 +85,78 @@ For online serving client examples and request formats, see #### Notes - **Key flags:** - - `--omni` — enables vLLM-Omni diffusion serving. - - `--use-hsdp` — Hybrid Sharded Data Parallelism for the 14B DiT weights. - - `--usp ` — Unified (Ulysses) Sequence Parallelism degree. - - `--cfg-parallel-size ` — CFG parallelism; set to 2 for the official model, + - `--omni` — enables vLLM-Omni diffusion serving. + - `--use-hsdp` — Hybrid Sharded Data Parallelism for the 14B DiT weights. + - `--usp ` — Unified (Ulysses) Sequence Parallelism degree. + - `--cfg-parallel-size ` — CFG parallelism; set to 2 for the official model, omit for distilled checkpoints. - - `--vae-patch-parallel-size 8` / `--vae-use-tiling` — parallel + tiled VAE + - `--vae-patch-parallel-size 8` / `--vae-use-tiling` — parallel + tiled VAE decoding; disabling patch parallelism can significantly increase VAE latency. +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +Offline text-to-video at a reduced 576x320, 17-frame shape. Layerwise offload +streams the two 14B expert transformers; VAE slicing bounds the decode +activations. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_video/text_to_video.py \ + --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \ + --prompt "A serene lakeside sunrise with mist over the water." \ + --negative-prompt "blurry, noisy, artifacts, distorted, low quality, streaks, sparkles, blocky, pixelated" \ + --height 320 --width 576 --num-frames 17 \ + --num-inference-steps 60 \ + --guidance-scale 4.0 \ + --flow-shift 5.0 \ + --boundary-ratio 0.875 \ + --enable-layerwise-offload \ + --vae-use-slicing \ + --enforce-eager \ + --seed 12345 \ + --output wan22_t2v_output.mp4 +``` + +Given spare cards, `--enable-cpu-offload` shards the two expert transformers +instead of streaming them, which nearly halves the denoise loop. Four is the +fastest count; eight is slower. + +```bash + --tensor-parallel-size 4 \ + --enable-cpu-offload +``` + +| Offload | Cards | Peak VRAM | Denoise | +| --- | ---: | ---: | ---: | +| `--enable-layerwise-offload` | 1 | 9.9 GiB | 160 s | +| `--enable-cpu-offload` | 2 | 18.9 GiB | 113 s | +| `--enable-cpu-offload` | 4 | 12.4 GiB | 86 s | + +#### Verification + +Confirm `wan22_t2v_output.mp4` decodes and that frame 0 is clean. + +#### Notes + +- Key flags: `--boundary-ratio 0.875` sets the high-noise to low-noise expert + switch. `--flow-shift 5.0` with `--guidance-scale 4.0` over 60 steps clears + the frame-0 VAE corruption seen at the stock 50-step, `flow_shift=12.0` + settings. +- Known limitations: only offline T2V at 576x320 was qualified. Larger shapes, + TI2V, HSDP, CFG parallelism, and online serving are out of scope for this + profile. + ## Accelerating with TRTLLM_ATTN + Skip-Softmax (datacenter Blackwell) On **datacenter Blackwell** (sm_100 / sm_103), the mask-free Wan pipeline @@ -153,7 +217,7 @@ instead of `target_sparsity`): ### Skip-Softmax controls | Key | Valid values | Meaning | -|---|---|---| +| --- | --- | --- | | `target_sparsity` | finite, `[0, 1]` | Operating point on the checkpoint's calibrated curve (`a·exp(b·s)`). | | `threshold` | finite, `≥ 0` | Direct skip threshold, no calibration needed. Mutually exclusive with `target_sparsity`. | | `disabled_until_timestep` | finite, `[0, 1]` | Keeps early, high-noise steps dense; skip turns on once the normalized timestep `t ≤ D`. | @@ -169,7 +233,7 @@ Measured on B300 / SM103, 1280×720 / 81f / 50 steps, torch.compile, official ModelOpt calibration (speedup and divergence relative to dense): | config | speedup | LPIPS (divergence vs dense) | -|---|---|---| +| --- | --- | --- | | dense | 1.000x | - | | `s=0.65, D=0.86` | 1.062x | 0.041 | | `s=0.75, D=1.00` | 1.181x | 0.377 | @@ -181,6 +245,6 @@ ModelOpt calibration (speedup and divergence relative to dense): `flashinfer`. Elsewhere, or without FlashInfer, selecting `TRTLLM_ATTN` raises rather than silently degrading. - **Known limitations:** - - Skip-Softmax is not supported under ring sequence parallelism; use Ulysses SP + - Skip-Softmax is not supported under ring sequence parallelism; use Ulysses SP (`--usp`), or run Skip-Softmax without ring. - - `TRTLLM_ATTN` is BF16-only (no attention-level quantization). + - `TRTLLM_ATTN` is BF16-only (no attention-level quantization). diff --git a/recipes/black-forest-labs/FLUX.2-klein.md b/recipes/black-forest-labs/FLUX.2-klein.md new file mode 100644 index 000000000..630864ab8 --- /dev/null +++ b/recipes/black-forest-labs/FLUX.2-klein.md @@ -0,0 +1,69 @@ +# FLUX.2-klein + +> Text-guided image editing with `black-forest-labs/FLUX.2-klein-4B` + +## Summary + +- Vendor: Black Forest Labs +- Model: `black-forest-labs/FLUX.2-klein-4B` +- Task: Image editing (image-to-image) +- Mode: Offline inference +- Maintainer: Community + +## When to use this recipe + +Use this recipe to run single-image edits with FLUX.2-klein-4B through the +shared offline image-edit example. + +## References + +- Model card: +- Shared runnable example: + [`examples/offline_inference/image_to_image/image_edit.py`](../../examples/offline_inference/image_to_image/image_edit.py) +- CPU offload guide: + [`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md) + +## Hardware Support + +This recipe documents one validated Intel XPU configuration. Extend it with +more hardware sections as community validation lands. + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +BF16 weights with model-level CPU offload. `--image` takes any RGB image. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/image_to_image/image_edit.py \ + --model black-forest-labs/FLUX.2-klein-4B \ + --image test_input.png \ + --prompt "Add a sunset sky with orange and purple clouds" \ + --num-inference-steps 28 \ + --guidance-scale 3.5 \ + --enable-cpu-offload \ + --enforce-eager \ + --seed 42 \ + --output image_edit_output.png +``` + +#### Verification + +Confirm `image_edit_output.png` follows the prompt while preserving the rest +of the input image. + +#### Notes + +- Memory usage: 7.7 GiB loaded in 8.4 s, about 18 s per edit. +- Known limitations: only offline editing of the 4B checkpoint was qualified. + `FLUX.2-klein-9B` and online serving are out of scope for this profile. diff --git a/recipes/krea/Krea-2.md b/recipes/krea/Krea-2.md index 80e5105dd..800373a93 100644 --- a/recipes/krea/Krea-2.md +++ b/recipes/krea/Krea-2.md @@ -141,3 +141,58 @@ confirm a valid 1024×1024 PNG is written. no-CFG, True for Raw CFG). - Known limitations: TP / SP / CFG-Parallel are not yet wired for Krea 2. HSDP, layerwise CPU offload, and VAE-patch-parallel (decode) are supported. + +## XPU + +### 1x Intel Arc Pro B70 (32 GB) + +Offline text-to-image for the distilled Turbo checkpoint at 1024x1024, with +layerwise offload streaming the transformer blocks. + +#### Environment + +- OS: Linux +- Python: 3.10+ +- torch: 2.13.0+xpu +- vLLM: 0.29.0 (`98dff2a8`) +- vLLM-Omni: `main` at `4c7a98c2` + +#### Command + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model krea/Krea-2-Turbo \ + --prompt "a photo of a sunset over mountains" \ + --height 1024 --width 1024 \ + --num-inference-steps 8 \ + --enable-layerwise-offload \ + --vae-use-tiling \ + --enforce-eager \ + --output krea2_turbo_output.png +``` + +Given spare cards, `--enable-cpu-offload` replaces layerwise streaming. At 8 +steps the swap dominates the run, and cards past the second shard nothing +further. + +```bash + --tensor-parallel-size 2 \ + --enable-cpu-offload +``` + +| Offload | Cards | Peak VRAM | Time | +| --- | ---: | ---: | ---: | +| `--enable-layerwise-offload` | 1 | 12.3 GiB | 27 s | +| `--enable-cpu-offload` | 2 | 25.6 GiB | 59 s | +| `--enable-cpu-offload` | 4 | 25.6 GiB | 59 s | + +#### Verification + +Confirm `krea2_turbo_output.png` is written as a 1024x1024 PNG matching the +prompt. + +#### Notes + +- Memory usage: 7.8 GiB loaded, 12.3 GiB peak, about 26 s per image. +- Known limitations: only offline Turbo generation was qualified. Krea-2-Raw, + LoRA, Cache-DiT, and online serving are out of scope for this profile.