anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add Intel Arc BMG recipes for validated P0 diffusion models. (#7537)

Browse Source 
Signed-off-by: Joshna-Medisetty <joshna.medisetty@intel.com>
This commit is contained in:
Joshna-Medisetty authored and GitHub committed 2026-09-25 15:42:08 -05:00
1 parent 5e36b5a86b
commit 62ebad2d7c
12 files changed
+603 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/models/supported_models.md
+21 -21
View File
@@ -19,9 +19,9 @@ th {
| Architecture | Models | Example HF Models | NVIDIA GPU | AMD GPU | Ascend NPU | Intel GPU | Recipe |
| -------------- | -------- | ------------------- | ------------ | --------- | ----- | ----------- | -------- |
| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) |
| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) |
| `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | | — |
| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | ✅︎ | — |
| `BagelForConditionalGeneration` | BAGEL (DiT-only) | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ | | ✅︎ | — |
| `InternVLAA1Pipeline` | InternVLA-A1 | `InternRobotics/InternVLA-A1-3B` | ✅︎ | ✅︎ | | | — |
| `Gr00tN1d7Pipeline` | GR00T N1.7 | `nvidia/GR00T-N1.7-3B` | ✅︎ | | | | — |
@@ -33,9 +33,9 @@ th {
| `QwenImageLayeredPipeline` | Qwen-Image-Layered | `Qwen/Qwen-Image-Layered` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `QwenImageEditPlusPipeline` | Qwen-Image-Edit-2511 | `Qwen/Qwen-Image-Edit-2511` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GlmImagePipeline` | GLM-Image | `zai-org/GLM-Image` | ✅︎ | ✅︎ | | | — |
| `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | ✅︎ | — |
| `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tongyi-MAI/Z-Image-Turbo.md) |
| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/krea/Krea-2.md) |
| `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Wan-AI/Wan2.2-T2V.md) |
| `WanImageToVideoPipeline` | Wan2.2-I2V | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LingBotWorldCausalDMDPipeline` | LingBot-World 2.0 v1 (experimental) | `robbyant/lingbot-world-v2-14b-causal-fast-diffusers` | Experimental | | | | — |
| `Cosmos3OmniDiffusersPipeline` | Cosmos3 T2I, T2V, I2V, V2V, T2V with sound, action policy | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
@@ -43,16 +43,16 @@ th {
| `Wan22S2VPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | ✅︎ | — |
| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | | — |
| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | | — |
| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | | — |
| `LTX2DistilledTwoStagePipeline` | LTX-2 / LTX-2.3 merged-distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` + matching Lightricks upsampler | ✅︎ | ✅︎ | | | — |
| `LTX2Pipeline` | LTX-2.5 Full/SFT one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) |
| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) |
| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) |
| `LTX2DistilledTwoStagePipeline` | LTX-2 / LTX-2.3 merged-distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` + matching Lightricks upsampler | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.md) |
| `LTX2Pipeline` | LTX-2.5 Full/SFT one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2TwoStagePipeline` | LTX-2.5 Full/SFT two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` + `Lightricks/LTX-2.5` LoRA | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2DistilledOneStagePipeline` | LTX-2.5 merged-distilled one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2DistilledTwoStagePipeline` | LTX-2.5 merged-distilled two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LingBotVideoPipeline` | LingBot-Video dense and MoE T2I, T2V, TI2V | `robbyant/lingbot-video-dense-1.3b`, `robbyant/lingbot-video-moe-30b-a3b` | ✅︎ | | | | — |
| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
| `SanaImageToVideoPipeline` | SANA-Video-2B I2V (native and Diffusers adapter validated at 480p/720p) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
| `MiniMaxH3Pipeline` | MiniMax H3 T2VA, first/last-keyframe FL2VA, and mixed-reference Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | ✅︎ | | | [Published](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) |
| `Magi2Pipeline` | MAGI-2 native Preview-stage T2VA and I2VA (272p/540p) | `sand-ai/MAGI-2-preview` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SandAI/MAGI-2-preview-L20X.md) |
@@ -65,18 +65,18 @@ th {
| `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LongCatVideoAvatarPipeline` | LongCat-Video-Avatar-1.5 A2V/AI2V (native single-speaker and multi-speaker AI2V/AVC) | `meituan-longcat/LongCat-Video-Avatar-1.5` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/meituan-longcat/LongCat-Video-Avatar-1.5.md) |
| `BooguImagePipeline` | Boogu-Image | `Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit`, `Boogu/Boogu-Image-0.1-Edit-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `BooguImageTurboPipeline` | Boogu-Image Turbo distilled T2I | `Boogu/Boogu-Image-0.1-Turbo` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `BooguImageTurboPipeline` | Boogu-Image Turbo distilled T2I | `Boogu/Boogu-Image-0.1-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/StabilityAI/Stable-Diffusion-3.5.md) |
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — |
| `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (voice design, voice clone, voice direction, CFG) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | ✅︎ | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `FluxKontextPipeline` | FLUX.1-Kontext-dev | `black-forest-labs/FLUX.1-Kontext-dev` | ✅︎ | ✅︎ | | | — |
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/black-forest-labs/FLUX.2-klein.md) |
| `FluxKontextPipeline` | FLUX.1-Kontext-dev | `black-forest-labs/FLUX.1-Kontext-dev` | ✅︎ | ✅︎ | | ✅︎ | — |
| `FluxPipeline` | FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | ✅︎ | ✅︎ | | ✅︎ | — |
| `FluxPipeline` | FLUX.1-schnell | `black-forest-labs/FLUX.1-schnell` | ✅︎ | ✅︎ | | ✅︎ | — |
| `OmniGen2Pipeline` | OmniGen2 | `OmniGen2/OmniGen2` | ✅︎ | ✅︎ | | ✅︎ | — |
@@ -97,13 +97,13 @@ th {
| `NextStep11Pipeline` | NextStep-1.1 | `stepfun-ai/NextStep-1.1` | ✅︎ | ✅︎ | | ✅︎ | — |
| `MiMoAudioModel` | MiMo-Audio-7B-Instruct | `XiaomiMiMo/MiMo-Audio-7B-Instruct` | ✅︎ | ✅︎ | | | — |
| `MiMoV2ASRForCausalLM` | MiMo-V2.5-ASR | `XiaomiMiMo/MiMo-V2.5-ASR` | ✅︎ | ✅︎ | | | — |
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | | — |
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | ✅︎ | — |
| `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | ✅︎ | — |
| `Audio8TTSSlowARForConditionalGeneration` | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✅︎ | | | | — |
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only), SenseNova-U1.5 | `SenseNova/SenseNova-U1-8B-MoT`, `sensenova/SenseNova-U1.5-8B-MoT` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.5.md) |
| `Audio8TTSSlowARForConditionalGeneration` | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✅︎ | | | ✅︎ | — |
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only), SenseNova-U1.5 | `SenseNova/SenseNova-U1-8B-MoT`, `sensenova/SenseNova-U1.5-8B-MoT` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.5.md) |
| `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | | — |
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanVideo-1.5.md) |
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
recipes/LTX/LTX-2.5.md
+28
View File
@@ -89,6 +89,7 @@ Install matching vLLM and vLLM-Omni versions, and ensure `ffmpeg` and
| NVIDIA B200 or H200 | Capacity-based recommendation; not yet verified | All four canonical pipelines |
| NVIDIA GB200 or GB300 | Capacity-based recommendation; not yet verified | All four canonical pipelines |
| NVIDIA H100 80 GB | FP8 recipe | Distilled one-stage at 960x544 |
| Intel Arc Pro B70 32 GB | Verified | Offline Full/SFT one-stage T2V at 768x512 |
The 1920x1088 two-stage examples require about 114 GB of peak GPU memory.
80 GB GPUs do not have enough safety margin for the canonical two-stage
@@ -133,6 +134,33 @@ SP attention head counts must be divisible by `ulysses_degree`. Use each
phase's `H, W` (half/full resolution for two-stage); non-divisible shapes fail.
Each GPU must still fit the resident weights. Warm up with the production shape.
### 1x Intel Arc Pro B70 32 GB (XPU)
Offline Full/SFT one-stage T2V at 768x512, 17 frames. Select the legacy
convolutional VAE with `ltx2_use_conv_vae`; the default DiffVAE decoder loads a
NATTEN kernel, which has no XPU build. FP8 with layerwise offload streams both
the transformer and the Gemma-4 text encoder.
Environment: torch 2.14.0+xpu, vLLM 0.29.0, vLLM-Omni `main` at `d8d162d8`.
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model Lightricks/LTX-2.5-Diffusers \
--model-class-name LTX2Pipeline \
--stage-overrides '{"0":{"extras":{"ltx2_use_conv_vae":true}}}' \
--prompt "A serene lakeside sunrise with mist over the water." \
--num-frames 17 \
--quantization fp8 \
--enable-layerwise-offload \
--vae-use-tiling \
--vae-use-slicing \
--enforce-eager \
--output ltx25_output.mp4
```
Peaks at 16.3 GiB in 163 s. Two-stage, distilled, and I2V paths were not
exercised on this hardware.
## Offline inference
Choose values from the pipeline table. For example, the distilled two-stage
recipes/LTX/LTX-2.md
+57
View File
@@ -165,6 +165,63 @@ curl -X POST http://localhost:8000/v1/videos/sync \
Use `image_reference` for a URL or JSON-safe image reference. Do not provide it
together with `input_reference`.
## Hardware Support
## XPU
### 1x Intel Arc Pro B70 (32 GB)
Offline one-stage T2V at 768x512, 17 frames. FP8 with layerwise offload streams
both the transformer and the Gemma text encoder, which otherwise stays resident
beside it.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.14.0+xpu
- vLLM: 0.29.0
- vLLM-Omni: `main` at `d8d162d8`
#### Command
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model diffusers/LTX-2.3-Diffusers \
--prompt "A serene lakeside sunrise with mist over the water." \
--num-frames 17 \
--quantization fp8 \
--enable-layerwise-offload \
--vae-use-tiling \
--vae-use-slicing \
--enforce-eager \
--output ltx23_output.mp4
```
The same flags cover every one-stage checkpoint; only `--model` changes. The
distilled checkpoints additionally need their class and an explicit size,
because their preset defaults to 1536x1024:
```bash
--model diffusers/LTX-2.3-Distilled-Diffusers \
--model-class-name LTX2DistilledOneStagePipeline \
--height 512 --width 768
```
| Checkpoint | Peak | Time |
| --- | ---: | ---: |
| `Lightricks/LTX-2` | 14.8 GiB | 160 s |
| `diffusers/LTX-2.3-Diffusers` | 16.4 GiB | 167 s |
| `dg845/LTX-2.3-Diffusers` | 16.4 GiB | 155 s |
| `diffusers/LTX-2.3-Distilled-Diffusers` | 14.9 GiB | 128 s |
#### Verification
Each command writes its `--output` MP4. Confirm it decodes and that the sampled
frames match the prompt.
## Guidance
One-stage and ordinary Stage 1 support independent video/audio CFG,
recipes/Qwen/Qwen-Image.md
+63 -3
View File
@@ -7,7 +7,8 @@
- Vendor: Qwen
- Model: `Qwen/Qwen-Image`
- Task: Text-to-image generation
- Mode: Online serving with optional step-wise continuous batching
- Mode: Online serving with optional step-wise continuous batching; offline
inference
- Maintainer: Community
## When to use this recipe
@@ -25,11 +26,14 @@ the benchmark assets already bundled in this repository.
[`examples/online_serving/text_to_image/README.md`](../../examples/online_serving/text_to_image/README.md)
- Related benchmark:
[`benchmarks/diffusion/diffusion_benchmark_serving.py`](../../benchmarks/diffusion/diffusion_benchmark_serving.py)
- Offline inference example used by the XPU section:
[`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py)
## Hardware Support
This recipe currently documents one CUDA GPU serving configuration. Extend it
with more hardware sections as community validation lands.
This recipe documents CUDA GPU serving configurations and one Intel XPU
offline-inference configuration. Extend it with more hardware sections as
community validation lands.
## GPU
@@ -235,3 +239,59 @@ CUTLASS was the fastest validated backend for this checkpoint.
- The benchmark table intentionally reports Qwen-Image-2512 as
concurrency-1 request-level data; do not compare it directly with an online
HTTP concurrency benchmark.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
Offline text-to-image at 1024x1024, with layerwise offload streaming the
transformer blocks so the 20B model fits one card.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model Qwen/Qwen-Image \
--prompt "a cup of coffee on the table" \
--num-inference-steps 50 \
--enable-layerwise-offload \
--vae-use-tiling \
--vae-use-slicing \
--enforce-eager \
--output qwen_image_output.png
```
Given spare cards, `--enable-cpu-offload` shards the transformer instead of
streaming it. It shortens the denoise loop but not the run, because the
per-request model swap costs more than the loop saves.
```bash
--tensor-parallel-size 4 \
--enable-cpu-offload
```
| Offload | Cards | Peak VRAM | Time |
| --- | ---: | ---: | ---: |
| `--enable-layerwise-offload` | 1 | 16.6 GiB | 50 s |
| `--enable-cpu-offload` | 2 | 26.4 GiB | 85 s |
| `--enable-cpu-offload` | 4 | 19.8 GiB | 70 s |
| `--enable-cpu-offload` | 8 | 16.9 GiB | 87 s |
#### Verification
Confirm `qwen_image_output.png` is written and looks coherent for the prompt.
#### Notes
- Memory usage: peak 16.6 GiB, about 55 s per image.
- Known limitations: only offline generation was qualified. Step-wise continuous
batching, ModelOpt quantization, and online serving are out of scope for this
profile.
recipes/Qwen/Qwen3-Omni.md
+51 -5
View File
@@ -53,7 +53,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/your_qwen3_omni_overrides.yaml
```
#### Runtime tuning
### Runtime tuning
Prefer CLI overrides for day-to-day tuning:
@@ -75,7 +75,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
}'
```
#### Stage-based launch (one stage per process)
### Stage-based launch (one stage per process)
Use three terminals (one per stage). Start with the default commands below, then
add `--max-num-seqs` only if you need explicit per-stage concurrency control.
@@ -135,7 +135,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
If you use custom deploy YAML, add `--deploy-config` to each stage command.
#### Verification
### Verification
After server startup, run a multimodal example client:
@@ -185,7 +185,7 @@ python examples/online_serving/qwen3_omni/openai_realtime_client.py \
--output-wav realtime_output.wav
```
#### Benchmark with `vllm bench`
### Benchmark with `vllm bench`
After the server is up, you can run online serving benchmarks with
`vllm bench serve --omni`.
@@ -240,8 +240,54 @@ vllm bench serve \
--percentile-metrics ttft,tpot,itl,e2el
```
#### Notes
### Notes
- `/v1/realtime` is unsupported while `async_chunk` is enabled.
- The default deploy uses `SharedMemoryConnector`; this is for single-host
stage wiring.
## XPU
### 5x Intel Arc Pro B70 (32 GB)
Offline end-to-end text query returning both text and speech. The 62 GiB of
BF16 thinker weights need four cards, so the `xpu:` block of
`vllm_omni/deploy/qwen3_omni_moe.yaml` runs the thinker at TP=4 on cards 0-3
and the talker and code2wav on card 4. It is the registry default here, so
`--deploy-config` is not needed.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
The raised timeouts absorb the first decode step, which JIT-compiles the
Triton MoE kernels.
```bash
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
python examples/offline_inference/qwen3_omni/end2end.py \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--query-type text \
--stage-init-timeout 1800 \
--init-timeout 1800 \
--output-wav qwen3omni_output
```
#### Verification
`qwen3omni_output/` holds one `.txt` answer and the matching `.wav`. Confirm
the transcript of the audio tracks the text response.
#### Notes
- Runtime: about 260 s for a single text query.
- The thinker MoE layers have no tuned Triton config for this shape and log
`Using default MoE config. Performance might be sub-optimal!`.
- Known limitations: only `--query-type text` was qualified. The audio, image,
and video query types and online serving are out of scope for this profile.
recipes/README.md
+10 -7
View File
@@ -45,6 +45,7 @@ recipes/
| [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB |
| [`Bagel/BAGEL-7B-MoT.md`](./Bagel/BAGEL-7B-MoT.md) | Text-to-image with shared online/offline examples | 1x A100 80GB / 2x CUDA GPUs |
| [`black-forest-labs/FLUX.1-schnell.md`](./black-forest-labs/FLUX.1-schnell.md) | Offline text-to-image generation with CPU offload | 1x NVIDIA GeForce RTX 5090 32GB |
| [`black-forest-labs/FLUX.2-klein.md`](./black-forest-labs/FLUX.2-klein.md) | Offline image editing (FLUX.2-klein-4B) | 1x Intel Arc Pro B70 32GB |
| [`Boogu/Boogu-Image.md`](./Boogu/Boogu-Image.md) | Text-to-image online serving (Boogu-Image-0.1-Base) | 1x A100/H100 40GB+ |
| [`BosonAI/Higgs-Audio-V3-TTS.md`](./BosonAI/Higgs-Audio-V3-TTS.md) | Online + offline multilingual TTS with voice cloning | 1x H100 80GB |
| [`BreezeBlue/Breeze-TTS-2.md`](./BreezeBlue/Breeze-TTS-2.md) | Online + offline TTS (plain / voice design / voice clone / voice direction, streaming PCM) | 1x NVIDIA L20 48GB |
@@ -62,9 +63,9 @@ recipes/
| [`MiniMaxAI/MiniMax-H3.md`](./MiniMaxAI/MiniMax-H3.md) | T2VA, FL2VA, image+audio Ref2VA, and multi-video Ref2VA serving | 1x GPU with CPU offload / 4x B300 |
| [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) |
| [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB |
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU |
| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified |
| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB / 1x Intel Arc Pro B70 32GB |
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU / 1x Intel Arc Pro B70 32GB |
| [`LTX/LTX-2.5.md`](./LTX/LTX-2.5.md) | LTX-2.5-Diffusers: Full/SFT one-stage and distilled two-stage T2V/I2V with synchronized audio | NVIDIA B300; cuDNN-qualified / 1x Intel Arc Pro B70 32GB |
| [`MammothModa2/MammothModa2.md`](./MammothModa2/MammothModa2.md) | Preview and Dev text-to-image (AR → DiT); Dev text/image understanding | Preview: 1x L40S 48GB / 1x ≥40GB GPU; Dev: 1x NVIDIA GPU with sufficient cache headroom |
| [`NVIDIA/SANA-Video-2B.md`](./NVIDIA/SANA-Video-2B.md) | Native and Diffusers-adapter 480p/720p T2V+I2V | 1x RTX 5090 32GB |
| [`meituan-longcat/LongCat-Video-Avatar-1.5.md`](./meituan-longcat/LongCat-Video-Avatar-1.5.md) | Audio-driven avatar video generation (AT2V / AI2V, single- and multi-speaker, AVC continuation) | 1x H100 80GB |
@@ -82,10 +83,10 @@ recipes/
| [`NVIDIA/Nemotron-Labs-Audex.md`](./NVIDIA/Nemotron-Labs-Audex.md) | TTS / text-to-audio / audio understanding / cascaded S2S (Audex 2B + 30B-A3B) | 1x H100 80GB |
| [`NVIDIA/NemotronLabs-VoiceChat.md`](./NVIDIA/NemotronLabs-VoiceChat.md) | Offline speech-to-speech voice chat (11B, frame-locked 12.5 Hz, 3-stage thinker/talker/code2wav) | 1x H100 80GB |
| [`NVIDIA/SANA-WM.md`](./NVIDIA/SANA-WM.md) | First-frame image-to-video serving with camera control | 1x 24GB+ CUDA GPU; ~40GB disk |
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 |
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4; offline text-to-image on XPU | 1x A100 80GB / 2x B200 / 1x Intel Arc Pro B70 32GB |
| [`Qwen/Qwen-Image-2512.md`](./Qwen/Qwen-Image-2512.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt FP8 / mixed FP8/NVFP4 | 1x A800 80GB / 2x B200 |
| [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB |
| [`Qwen/Qwen3-Omni.md`](./Qwen/Qwen3-Omni.md) | Online serving for multimodal chat | 1x A100 80GB |
| [`Qwen/Qwen3-Omni.md`](./Qwen/Qwen3-Omni.md) | Online serving for multimodal chat; offline end-to-end text-to-speech on XPU | 1x A100 80GB / 5x Intel Arc Pro B70 32GB |
| [`Qwen/Qwen3-TTS.md`](./Qwen/Qwen3-TTS.md) | Text-to-speech serving (CustomVoice / VoiceDesign / Base) | 1x H100/A100 80GB / 1x AMD MI300X |
| [`SenseNova/SenseNova-U1.md`](./SenseNova/SenseNova-U1.md) | Unified image generation and understanding | 1x or 2x H200 / 1x AMD MI300X |
| [`SenseNova/SenseNova-U1.5.md`](./SenseNova/SenseNova-U1.5.md) | Unified image generation and understanding, with an 8-step distilled LoRA | 1x A800 80GB |
@@ -93,13 +94,15 @@ recipes/
| [`Tencent/AuK-H100.md`](./Tencent/AuK-H100.md) | Offline instruction-driven TTS, speech editing, enhancement and separation (AuK base and Flash) | 1x H100 80GB |
| [`Tencent/Covo-Audio-Chat.md`](./Tencent/Covo-Audio-Chat.md) | Online serving for audio chat | 1x A100 80GB |
| [`Tencent/HunyuanImage-3.0-Instruct.md`](./Tencent/HunyuanImage-3.0-Instruct.md) | DiT-only text-to-image serving and benchmark, including ModelOpt mixed FP8/NVFP4 | 4x H100/H800 80GB / 2x B200 |
| [`Wan-AI/Wan2.2-T2V.md`](./Wan-AI/Wan2.2-T2V.md) | Text-to-video serving (Wan2.2 14B) with Skip-Softmax sparse attention | 1x datacenter Blackwell (B200/B300) |
| [`Tencent/HunyuanVideo-1.5.md`](./Tencent/HunyuanVideo-1.5.md) | Offline text-to-video (HunyuanVideo-1.5 Diffusers) | 1x Intel Arc Pro B70 32GB |
| [`Tongyi-MAI/Z-Image-Turbo.md`](./Tongyi-MAI/Z-Image-Turbo.md) | Offline few-step text-to-image (Z-Image-Turbo) | 1x Intel Arc Pro B70 32GB |
| [`Wan-AI/Wan2.2-T2V.md`](./Wan-AI/Wan2.2-T2V.md) | Text-to-video serving (Wan2.2 14B) with Skip-Softmax sparse attention | 8x H20/H100/A100 / 1x datacenter Blackwell (B200/B300) / 1x Intel Arc Pro B70 32GB |
| [`Wan-AI/Wan2.2-TI2V-5B-A100.md`](./Wan-AI/Wan2.2-TI2V-5B-A100.md) | Unified 720p text-to-video and image-to-video serving (Wan2.2 5B) | 1x A100-SXM4-80GB |
| [`Wan-AI/Wan2.2-I2V.md`](./Wan-AI/Wan2.2-I2V.md) | Image-to-video serving (Wan2.2 14B) | 8x Ascend NPU (A2/A3) |
| [`Wan-AI/Wan2.2-S2V.md`](./Wan-AI/Wan2.2-S2V.md) | Speech-to-video serving (Wan2.2 14B) | 2x A100/H100 80GB |
| [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) |
| [`XiaomiMiMo/MiMo-Audio.md`](./XiaomiMiMo/MiMo-Audio.md) | Offline + online omni audio (TTS, ASR, dialogue) | 1x RTX 5090 / 5090D 32GB |
| [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB |
| [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB / 1x Intel Arc Pro B70 32GB |
| [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB / 2x non-standard RTX 4090 48GB (49,140 MiB each) |
| [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB |
| [`JD/JoyAI-VL-Interaction.md`](./JD/JoyAI-VL-Interaction.md) | Real-time streaming video-language interaction (proactive speak/silence/delegate) | 1x GPU 24GB+ |
recipes/StabilityAI/Stable-Diffusion-3.5.md
+36
View File
@@ -137,3 +137,39 @@ print('saved output.png')
- Image quality is noticeably better than medium, with sharper details.
- `negative_prompt` requires `guidance_scale > 1` to take effect; SD3 defaults to 1.0 (no CFG).
- 2048×2048 fits within 48 GB but leaves little headroom (~45.5 GiB used out of 49 GiB).
## XPU
### 1x Intel Arc Pro B70 (32 GB) — stable-diffusion-3.5-medium
Offline text-to-image at 1024x1024 with BF16 weights fully resident.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model stabilityai/stable-diffusion-3.5-medium \
--prompt "a sunset over mountains, photorealistic" \
--num-inference-steps 28 \
--vae-use-tiling \
--output sd35_medium_output.png
```
#### Verification
Confirm `sd35_medium_output.png` is written and matches the prompt.
#### Notes
- Memory usage: peak 19.5 GiB against 29.8 GiB free.
- Known limitations: only `stable-diffusion-3.5-medium` offline generation was
qualified. `stable-diffusion-3.5-large` and online serving are out of scope
for this profile.
recipes/Tencent/HunyuanVideo-1.5.md
+72
View File
@@ -0,0 +1,72 @@
# HunyuanVideo 1.5
> Text-to-video with the HunyuanVideo-1.5 Diffusers checkpoints
## Summary
- Vendor: Tencent (community Diffusers ports)
- Model: `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v`
- Task: Text-to-video
- Mode: Offline inference
- Maintainer: Community
## When to use this recipe
Use this recipe to generate a short clip with HunyuanVideo-1.5 on a memory-
constrained card, trading output size for capacity.
## References
- Model card:
<https://huggingface.co/hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v>
- Shared runnable example:
[`examples/offline_inference/text_to_video/text_to_video.py`](../../examples/offline_inference/text_to_video/text_to_video.py)
- CPU offload guide:
[`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md)
## Hardware Support
This recipe documents one validated Intel XPU configuration. Extend it with
more hardware sections as community validation lands.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
BF16 weights with model-level CPU offload at a reduced 320x240, 17-frame
shape. The 480p and 720p T2V and I2V checkpoints all use the same command; only
`--model` changes.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v \
--prompt "A serene lakeside sunrise with mist over the water." \
--height 240 --width 320 --num-frames 17 \
--num-inference-steps 50 \
--guidance-scale 6.0 \
--enable-cpu-offload \
--vae-use-tiling \
--vae-use-slicing \
--output hunyuan_video_15_output.mp4
```
#### Verification
Confirm `hunyuan_video_15_output.mp4` decodes and that the sampled frames match
the prompt.
#### Notes
- Memory usage: 18.3 GiB loaded, 25.7 GiB peak, about 39 s per clip.
- Known limitations: the native 720p geometry does not fit a 32 GB card with
CPU offload. Online serving is out of scope for this profile.
recipes/Tongyi-MAI/Z-Image-Turbo.md
+68
View File
@@ -0,0 +1,68 @@
# Z-Image Turbo
> Few-step text-to-image with `Tongyi-MAI/Z-Image-Turbo`
## Summary
- Vendor: Tongyi-MAI
- Model: `Tongyi-MAI/Z-Image-Turbo`
- Task: Text-to-image
- Mode: Offline inference
- Maintainer: Community
## When to use this recipe
Use this recipe to generate 1024x1024 images with the distilled Z-Image Turbo
checkpoint through the shared offline text-to-image example.
## References
- Model card: <https://huggingface.co/Tongyi-MAI/Z-Image-Turbo>
- Shared runnable example:
[`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py)
- CPU offload guide:
[`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md)
## Hardware Support
This recipe documents one validated Intel XPU configuration. Extend it with
more hardware sections as community validation lands.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
BF16 weights with model-level CPU offload and VAE tiling/slicing, 25 steps.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model Tongyi-MAI/Z-Image-Turbo \
--prompt "a cup of coffee on the table" \
--num-inference-steps 25 \
--enable-cpu-offload \
--vae-use-tiling \
--vae-use-slicing \
--enforce-eager \
--output z_image_turbo_output.png
```
#### Verification
Confirm `z_image_turbo_output.png` is written as a 1024x1024 PNG matching the
prompt.
#### Notes
- Memory usage: peak 15.6 GiB, about 58 s per image.
- Known limitations: only offline generation was qualified. Online serving is
out of scope for this profile.
recipes/Wan-AI/Wan2.2-T2V.md
+73 -9
View File
@@ -85,14 +85,78 @@ For online serving client examples and request formats, see
#### Notes
- **Key flags:**
- `--omni` — enables vLLM-Omni diffusion serving.
- `--use-hsdp` — Hybrid Sharded Data Parallelism for the 14B DiT weights.
- `--usp <N>` — Unified (Ulysses) Sequence Parallelism degree.
- `--cfg-parallel-size <N>` — CFG parallelism; set to 2 for the official model,
- `--omni` — enables vLLM-Omni diffusion serving.
- `--use-hsdp` — Hybrid Sharded Data Parallelism for the 14B DiT weights.
- `--usp <N>` — Unified (Ulysses) Sequence Parallelism degree.
- `--cfg-parallel-size <N>` — CFG parallelism; set to 2 for the official model,
omit for distilled checkpoints.
- `--vae-patch-parallel-size 8` / `--vae-use-tiling` — parallel + tiled VAE
- `--vae-patch-parallel-size 8` / `--vae-use-tiling` — parallel + tiled VAE
decoding; disabling patch parallelism can significantly increase VAE latency.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
Offline text-to-video at a reduced 576x320, 17-frame shape. Layerwise offload
streams the two 14B expert transformers; VAE slicing bounds the decode
activations.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--prompt "A serene lakeside sunrise with mist over the water." \
--negative-prompt "blurry, noisy, artifacts, distorted, low quality, streaks, sparkles, blocky, pixelated" \
--height 320 --width 576 --num-frames 17 \
--num-inference-steps 60 \
--guidance-scale 4.0 \
--flow-shift 5.0 \
--boundary-ratio 0.875 \
--enable-layerwise-offload \
--vae-use-slicing \
--enforce-eager \
--seed 12345 \
--output wan22_t2v_output.mp4
```
Given spare cards, `--enable-cpu-offload` shards the two expert transformers
instead of streaming them, which nearly halves the denoise loop. Four is the
fastest count; eight is slower.
```bash
--tensor-parallel-size 4 \
--enable-cpu-offload
```
| Offload | Cards | Peak VRAM | Denoise |
| --- | ---: | ---: | ---: |
| `--enable-layerwise-offload` | 1 | 9.9 GiB | 160 s |
| `--enable-cpu-offload` | 2 | 18.9 GiB | 113 s |
| `--enable-cpu-offload` | 4 | 12.4 GiB | 86 s |
#### Verification
Confirm `wan22_t2v_output.mp4` decodes and that frame 0 is clean.
#### Notes
- Key flags: `--boundary-ratio 0.875` sets the high-noise to low-noise expert
switch. `--flow-shift 5.0` with `--guidance-scale 4.0` over 60 steps clears
the frame-0 VAE corruption seen at the stock 50-step, `flow_shift=12.0`
settings.
- Known limitations: only offline T2V at 576x320 was qualified. Larger shapes,
TI2V, HSDP, CFG parallelism, and online serving are out of scope for this
profile.
## Accelerating with TRTLLM_ATTN + Skip-Softmax (datacenter Blackwell)
On **datacenter Blackwell** (sm_100 / sm_103), the mask-free Wan pipeline
@@ -153,7 +217,7 @@ instead of `target_sparsity`):
### Skip-Softmax controls
| Key | Valid values | Meaning |
|---|---|---|
| --- | --- | --- |
| `target_sparsity` | finite, `[0, 1]` | Operating point on the checkpoint's calibrated curve (`a·exp(b·s)`). |
| `threshold` | finite, `≥ 0` | Direct skip threshold, no calibration needed. Mutually exclusive with `target_sparsity`. |
| `disabled_until_timestep` | finite, `[0, 1]` | Keeps early, high-noise steps dense; skip turns on once the normalized timestep `t ≤ D`. |
@@ -169,7 +233,7 @@ Measured on B300 / SM103, 1280×720 / 81f / 50 steps, torch.compile, official
ModelOpt calibration (speedup and divergence relative to dense):
| config | speedup | LPIPS (divergence vs dense) |
|---|---|---|
| --- | --- | --- |
| dense | 1.000x | - |
| `s=0.65, D=0.86` | 1.062x | 0.041 |
| `s=0.75, D=1.00` | 1.181x | 0.377 |
@@ -181,6 +245,6 @@ ModelOpt calibration (speedup and divergence relative to dense):
`flashinfer`. Elsewhere, or without FlashInfer, selecting `TRTLLM_ATTN` raises
rather than silently degrading.
- **Known limitations:**
- Skip-Softmax is not supported under ring sequence parallelism; use Ulysses SP
- Skip-Softmax is not supported under ring sequence parallelism; use Ulysses SP
(`--usp`), or run Skip-Softmax without ring.
- `TRTLLM_ATTN` is BF16-only (no attention-level quantization).
- `TRTLLM_ATTN` is BF16-only (no attention-level quantization).
recipes/black-forest-labs/FLUX.2-klein.md
+69
View File
@@ -0,0 +1,69 @@
# FLUX.2-klein
> Text-guided image editing with `black-forest-labs/FLUX.2-klein-4B`
## Summary
- Vendor: Black Forest Labs
- Model: `black-forest-labs/FLUX.2-klein-4B`
- Task: Image editing (image-to-image)
- Mode: Offline inference
- Maintainer: Community
## When to use this recipe
Use this recipe to run single-image edits with FLUX.2-klein-4B through the
shared offline image-edit example.
## References
- Model card: <https://huggingface.co/black-forest-labs/FLUX.2-klein-4B>
- Shared runnable example:
[`examples/offline_inference/image_to_image/image_edit.py`](../../examples/offline_inference/image_to_image/image_edit.py)
- CPU offload guide:
[`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md)
## Hardware Support
This recipe documents one validated Intel XPU configuration. Extend it with
more hardware sections as community validation lands.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
BF16 weights with model-level CPU offload. `--image` takes any RGB image.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/image_to_image/image_edit.py \
--model black-forest-labs/FLUX.2-klein-4B \
--image test_input.png \
--prompt "Add a sunset sky with orange and purple clouds" \
--num-inference-steps 28 \
--guidance-scale 3.5 \
--enable-cpu-offload \
--enforce-eager \
--seed 42 \
--output image_edit_output.png
```
#### Verification
Confirm `image_edit_output.png` follows the prompt while preserving the rest
of the input image.
#### Notes
- Memory usage: 7.7 GiB loaded in 8.4 s, about 18 s per edit.
- Known limitations: only offline editing of the 4B checkpoint was qualified.
`FLUX.2-klein-9B` and online serving are out of scope for this profile.
recipes/krea/Krea-2.md
+55
View File
@@ -141,3 +141,58 @@ confirm a valid 1024×1024 PNG is written.
no-CFG, True for Raw CFG).
- Known limitations: TP / SP / CFG-Parallel are not yet wired for Krea 2. HSDP,
layerwise CPU offload, and VAE-patch-parallel (decode) are supported.
## XPU
### 1x Intel Arc Pro B70 (32 GB)
Offline text-to-image for the distilled Turbo checkpoint at 1024x1024, with
layerwise offload streaming the transformer blocks.
#### Environment
- OS: Linux
- Python: 3.10+
- torch: 2.13.0+xpu
- vLLM: 0.29.0 (`98dff2a8`)
- vLLM-Omni: `main` at `4c7a98c2`
#### Command
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model krea/Krea-2-Turbo \
--prompt "a photo of a sunset over mountains" \
--height 1024 --width 1024 \
--num-inference-steps 8 \
--enable-layerwise-offload \
--vae-use-tiling \
--enforce-eager \
--output krea2_turbo_output.png
```
Given spare cards, `--enable-cpu-offload` replaces layerwise streaming. At 8
steps the swap dominates the run, and cards past the second shard nothing
further.
```bash
--tensor-parallel-size 2 \
--enable-cpu-offload
```
| Offload | Cards | Peak VRAM | Time |
| --- | ---: | ---: | ---: |
| `--enable-layerwise-offload` | 1 | 12.3 GiB | 27 s |
| `--enable-cpu-offload` | 2 | 25.6 GiB | 59 s |
| `--enable-cpu-offload` | 4 | 25.6 GiB | 59 s |
#### Verification
Confirm `krea2_turbo_output.png` is written as a 1024x1024 PNG matching the
prompt.
#### Notes
- Memory usage: 7.8 GiB loaded, 12.3 GiB peak, about 26 s per image.
- Known limitations: only offline Turbo generation was qualified. Krea-2-Raw,
LoRA, Cache-DiT, and online serving are out of scope for this profile.