Compare commits

..
Author SHA1 Message Date
Hongsheng Liu b112c7fdee docs: restore BitsAndBytes navigation
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-12 00:10:38 +08:00
Hongsheng Liu e63ae881e7 docs: remove GGUF from active navigation
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-11 17:22:19 +08:00
Hongsheng Liu 2787f7f5f6 docs: clarify attention design coverage
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-11 17:07:49 +08:00
Hongsheng Liu 3cc05bf1c1 docs: address feature taxonomy review
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-11 16:55:39 +08:00
Hongsheng Liu 5a33440362 docs: align User Guide feature taxonomy
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-11 16:16:46 +08:00
12 changed files with 211 additions and 240 deletions

No files matched your search

+31 -25
View File
@@ -40,30 +40,12 @@ nav:
- Models: - Models:
- models/supported_models.md - models/supported_models.md
- Features: - Features:
- Overview: features/README.md
- Runtime and Stage Execution:
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
- Sleep Mode: features/sleep_mode.md - Sleep Mode: features/sleep_mode.md
- Diffusion Features:
- Overview: user_guide/diffusion_features.md
- Feature Compatibility: user_guide/feature_compatibility.md
- Regional Compilation: user_guide/diffusion/regional_compilation.md
- Cache Acceleration:
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
- Attention Backends: user_guide/diffusion/attention_backends.md
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
- Parallelism:
- Overview: user_guide/diffusion/parallelism/overview.md
- CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md
- Expert Parallel: user_guide/diffusion/parallelism/expert_parallel.md
- Hybrid Sharded Data Parallel: user_guide/diffusion/parallelism/hsdp.md
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
- CPU Offloading: user_guide/diffusion/cpu_offload.md
- LoRA: user_guide/diffusion/lora.md
- Custom Pipeline: features/custom_pipeline.md
- Execution Modes: user_guide/diffusion/execution_modes.md
- Quantization: - Quantization:
- Quantization: user_guide/quantization/overview.md - Overview: user_guide/quantization/overview.md
- Online Quantization: user_guide/quantization/online.md - Online Quantization: user_guide/quantization/online.md
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
- FP8 W8A8: user_guide/quantization/fp8.md - FP8 W8A8: user_guide/quantization/fp8.md
@@ -72,9 +54,33 @@ nav:
- MXFP8 W8A8: user_guide/quantization/mxfp8.md - MXFP8 W8A8: user_guide/quantization/mxfp8.md
- MXFP4 W4A4: user_guide/quantization/mxfp4.md - MXFP4 W4A4: user_guide/quantization/mxfp4.md
- ModelOpt: user_guide/quantization/modelopt.md - ModelOpt: user_guide/quantization/modelopt.md
- GGUF: user_guide/quantization/gguf.md
- AutoRound: user_guide/quantization/autoround.md - AutoRound: user_guide/quantization/autoround.md
- msModelSlim: user_guide/quantization/msmodelslim.md - msModelSlim: user_guide/quantization/msmodelslim.md
- Diffusion Acceleration:
- Overview and Compatibility:
- Overview: user_guide/diffusion_features.md
- Feature Compatibility: user_guide/feature_compatibility.md
- CPU Offloading: user_guide/diffusion/cpu_offload.md
- Cache Acceleration:
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
- Parallelism:
- Overview: user_guide/diffusion/parallelism/overview.md
- CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md
- Expert Parallel: user_guide/diffusion/parallelism/expert_parallel.md
- Hybrid Sharded Data Parallel: user_guide/diffusion/parallelism/hsdp.md
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
- Attention Backends: user_guide/diffusion/attention_backends.md
- Regional Compilation: user_guide/diffusion/regional_compilation.md
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
- Startup and Loading: user_guide/diffusion/startup_and_loading.md
- LoRA: user_guide/diffusion/lora.md
- Experimental:
- Session State Manager: features/session_state_manager.md
- Integrations:
- Custom Diffusion Pipeline: features/custom_pipeline.md
- ComfyUI: features/comfyui.md - ComfyUI: features/comfyui.md
- Fault Tolerance: - Fault Tolerance:
- Failure Modes: user_guide/fault_injection_reliability_matrix.md - Failure Modes: user_guide/fault_injection_reliability_matrix.md
@@ -108,6 +114,7 @@ nav:
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md - Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md - Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md - Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
- Quantization: design/feature/quantization.md
- Diffusion Acceleration: - Diffusion Acceleration:
- Parallelism: - Parallelism:
- CFG-Parallel: design/feature/cfg_parallel.md - CFG-Parallel: design/feature/cfg_parallel.md
@@ -117,9 +124,8 @@ nav:
- Sequence Parallel: design/feature/sequence_parallel.md - Sequence Parallel: design/feature/sequence_parallel.md
- Tensor Parallel: design/feature/tensor_parallel.md - Tensor Parallel: design/feature/tensor_parallel.md
- VAE Patch Parallelism: design/feature/vae_parallel.md - VAE Patch Parallelism: design/feature/vae_parallel.md
- Attention Backends: - Attention Optimization:
- Skip-Softmax: design/feature/skip_softmax.md - Skip-Softmax: design/feature/skip_softmax.md
- Quantization: design/feature/quantization.md
- Cache-DiT: design/feature/cache_dit.md - Cache-DiT: design/feature/cache_dit.md
- TeaCache: design/feature/teacache.md - TeaCache: design/feature/teacache.md
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md - Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
+13 -5
View File
@@ -11,6 +11,10 @@ in the active navigation.
## Feature Design Documents ## Feature Design Documents
For user-facing configuration and current compatibility, see the
[Features overview](../features/README.md). A design document defines an
implementation contract; it is not, by itself, a general support claim.
### Runtime and stage execution ### Runtime and stage execution
- [Disaggregated Inference](feature/disaggregated_inference.md) - [Disaggregated Inference](feature/disaggregated_inference.md)
@@ -30,6 +34,10 @@ in the active navigation.
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md) - [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md) - [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
### Quantization
- [Quantization](feature/quantization.md)
### Diffusion acceleration ### Diffusion acceleration
#### Parallelism #### Parallelism
@@ -42,14 +50,14 @@ in the active navigation.
- [Tensor Parallel](feature/tensor_parallel.md) - [Tensor Parallel](feature/tensor_parallel.md)
- [VAE Patch Parallelism](feature/vae_parallel.md) - [VAE Patch Parallelism](feature/vae_parallel.md)
#### Attention Backends #### Attention optimization
The [Diffusion Attention Backends](../user_guide/diffusion/attention_backends.md)
guide lists all selectable backends and their platform defaults. Skip-Softmax
is the backend-related optimization with a standalone feature-design contract:
- [Skip-Softmax](feature/skip_softmax.md) - [Skip-Softmax](feature/skip_softmax.md)
#### Quantization
- [Quantization](feature/quantization.md)
- [Cache-DiT](feature/cache_dit.md) - [Cache-DiT](feature/cache_dit.md)
- [TeaCache](feature/teacache.md) - [TeaCache](feature/teacache.md)
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md) - [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
+67
View File
@@ -0,0 +1,67 @@
# Features
Use this section to choose and configure vLLM-Omni runtime and optimization
features. The navigation follows the active
[Feature Design](../design/index.md#feature-design-documents) taxonomy whenever
an implementation contract has a user-facing workflow.
!!! note "User guides, design documents, and recipes"
User guides explain how to enable a feature and where it is supported.
Design documents define internal contracts and do not, by themselves,
imply general or production support. For model-specific launch flags and
hardware requirements, use the linked examples and validated recipes from
[Supported Models](../models/supported_models.md).
## Runtime and Stage Execution
| Goal | User guide | Related design contract |
| --- | --- | --- |
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
Some runtime designs are deliberately not promoted as standalone User Guide
features yet:
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
[OmniConnector implementations](../design/index.md#communication) describe
topology and transport contracts. Practical deployment configuration remains
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
- [Async Chunk](../design/feature/async_chunk.md) and
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
are model- and pipeline-dependent. Use the selected model's deploy
configuration and recipe for the supported settings.
- [Automatic Prefix Caching](../design/feature/prefix_caching.md) remains a
design-level contract until its user-facing configuration and compatibility
surface is consolidated.
## Quantization
Quantization is a cross-model feature rather than a diffusion-only
optimization. The unified [`quantization_config` guide](../user_guide/quantization/overview.md)
covers diffusion-only models, multi-stage omni/TTS models, and multi-stage
diffusion models. Its [design contract](../design/feature/quantization.md)
defines the shared configuration and backend extension points.
## Diffusion Acceleration
| Goal | User guide | Related design contract |
| --- | --- | --- |
| Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) |
| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [Distributed Layerwise Offload](../design/feature/distributed_layerwise_offload.md) |
| Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) |
| Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) |
| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Skip-Softmax](../design/feature/skip_softmax.md) |
| Compile repeated diffusion regions | [Regional Compilation](../user_guide/diffusion/regional_compilation.md) | User-facing optimization guide |
| Add generated video frames | [Frame Interpolation](../user_guide/diffusion/frame_interpolation.md) | User-facing extension guide |
| Reduce diffusion model startup time | [Startup and Loading](../user_guide/diffusion/startup_and_loading.md) | User-facing loading guide |
| Apply diffusion adapters | [LoRA](../user_guide/diffusion/lora.md) | User-facing extension guide |
The [Pipeline Parallelism guide](../user_guide/diffusion/parallelism/pipeline_parallel.md)
remains available by direct link, but it is not promoted in the primary User
Guide navigation until its user-facing support and placement are settled.
## Experimental
[Session State Manager](session_state_manager.md) is opt-in and experimental.
Its APIs and compatibility may change without notice.
@@ -0,0 +1,60 @@
# Diffusion Startup and Loading
Large diffusion models can take several minutes to load at startup. vLLM-Omni
loads safetensors shards in parallel to reduce this initialization time.
Multi-thread weight loading is enabled by default with four threads. No
configuration is needed for the default behavior.
## Configuration
| Parameter | CLI flag | Default | Description |
| --- | --- | --- | --- |
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` | Pass the flag to disable multi-thread loading |
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of parallel weight-loading threads |
!!! tip
The default balances startup speed and disk I/O contention. Fast NVMe
storage may benefit from more threads, while network storage or hard disks
may not.
## Online Serving
```bash
# Default: multi-thread loading with four threads
vllm serve Qwen/Qwen-Image --omni --port 8091
# Increase the thread count
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni \
--num-weight-load-threads 8
# Disable multi-thread loading
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
```
## Offline Inference
```python
from vllm_omni import Omni
# Default: multi-thread loading with four threads
omni = Omni(model="Qwen/Qwen-Image")
# Increase the thread count
omni = Omni(
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
num_weight_load_threads=8,
)
```
## Reference Benchmarks
The following measurements were collected on NVIDIA H800 hardware. Treat them
as reference results rather than a guarantee for other storage or hardware
configurations.
| Model | Sequential loading | Multi-thread loading | Speedup |
| --- | ---: | ---: | ---: |
| **Qwen/Qwen-Image** (53.7 GiB) | 168 s | 27 s | **6.2x** |
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283 s | 56 s | **5.1x** |
+37 -85
View File
@@ -49,7 +49,7 @@ Parallelism methods distribute computation across GPUs without quality loss (mat
| Method | Description | Best For | | Method | Description | Best For |
|--------|-------------|----------| |--------|-------------|----------|
| **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds | | **[Multi-Thread Weight Loading](diffusion/startup_and_loading.md)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds |
**Note:** Some acceleration methods can be combined together for optimized performance. See [Feature Compatibility Table](#feature-compatibility) and [Feature Compatibility Tutorial](feature_compatibility.md) for detailed configuration examples. **Note:** Some acceleration methods can be combined together for optimized performance. See [Feature Compatibility Table](#feature-compatibility) and [Feature Compatibility Tutorial](feature_compatibility.md) for detailed configuration examples.
@@ -100,7 +100,9 @@ The following tables show which models support each feature:
- **🔀SP (Ulysses & Ring)**: Includes both Ulysses-SP and Ring-Attention methods - **🔀SP (Ulysses & Ring)**: Includes both Ulysses-SP and Ring-Attention methods
- ✅ = Fully supported - ✅ = Fully supported
- ✅* = Supported with the constraint listed below the table
- ❌ = Not supported - ❌ = Not supported
- ❓ = Not verified; not recommended
> Notes: > Notes:
@@ -119,7 +121,7 @@ The following tables show which models support each feature:
| **FLUX.2-dev** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | | **FLUX.2-dev** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **GLM-Image** | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | | **GLM-Image** | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | | **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅* |
| **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ | | **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | | **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | | **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
@@ -145,6 +147,7 @@ The following tables show which models support each feature:
> 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary. > 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload. > 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`. > 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
### VideoGen ### VideoGen
@@ -155,13 +158,16 @@ The following tables show which models support each feature:
| **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ | | **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ | | **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LTX-2.3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ | | **LTX-2.3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | | **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ | | **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | | **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ | | **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ | | **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
> **Step execution note:** Helios supports single-request step execution only;
> use `max_num_seqs=1`.
**Frame Interpolation Support** **Frame Interpolation Support**
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines - **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines
@@ -190,7 +196,7 @@ The following tables show which models support each feature:
| **🔀Ring-Attn** | ✅ | ✅ | ✅ | | | | | | | | | | | | | **🔀Ring-Attn** | ✅ | ✅ | ✅ | | | | | | | | | | | |
| **🔀CFG-Parallel** | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | | | **🔀CFG-Parallel** | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |
| **🔀Tensor Parallel** | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | | **🔀Tensor Parallel** | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
| **🔀HSDP** | ❓ | ❓ | ❓ | ❓ | ❓ | ❌ | | | | | | | | | | **🔀HSDP** | ❓ | ❓ | ✅ | ❓ | ✅ | ❌ | | | | | | | | |
| **🔀Expert Parallel** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | | | | | | | | **🔀Expert Parallel** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | | | | | | |
| **💾CPU Offloading (Layerwise)** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | | | | | | | **💾CPU Offloading (Layerwise)** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | | | | | |
| **💾CPU Offloading (Module-wise)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ❓ | ❌ | | | | | | | **💾CPU Offloading (Module-wise)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ❓ | ❌ | | | | | |
@@ -201,97 +207,43 @@ The following tables show which models support each feature:
!!! info !!! info
1. Tensor Parallel and HSDP are not compatible. 1. HSDP can be combined with Ulysses-SP or CFG-Parallel. Tensor Parallel
and HSDP are not compatible; other HSDP combinations in the table
remain unverified.
2. TeaCache and Cache-DiT are not compatible. 2. TeaCache and Cache-DiT are not compatible.
3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible. 3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
4. CPU Offloading (Layerwise) supports single-card for now. 4. The CPU Offloading (Layerwise) row describes local layerwise offload.
5. Using FP8-Quant as an example of qunatization methods. Multi-device Distributed Layerwise Offload has a separate topology and
compatibility matrix in the [CPU Offloading guide](diffusion/cpu_offload.md#distributed-layerwise-offloading).
5. The compatibility matrix uses FP8 as the representative quantization method.
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches). 6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
## Multi-Thread Weight Loading ## Multi-Thread Weight Loading
Large diffusion models can take several minutes to load weights at startup (e.g., ~3 min for Qwen-Image, ~5 min for Wan2.2 I2V 14B). Multi-thread weight loading speeds up this process by loading safetensors shards in parallel using a thread pool instead of sequentially. The loading guide now lives at [Diffusion Startup and
Loading](diffusion/startup_and_loading.md). This heading remains so existing
This optimization is **enabled by default** with 4 threads. No configuration is needed for the default behavior. links to this section continue to work.
### Configuration
| Parameter | CLI Flag | Default | Description |
|-----------|----------|---------|-------------|
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` (enabled) | Pass the flag to disable multi-thread loading |
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of threads for parallel weight loading |
!!! tip
The default of 4 threads balances speed and disk I/O contention. On fast NVMe storage you may benefit from more threads (e.g., 8). On HDD or network storage, the default of 4 avoids saturating I/O bandwidth.
### Online Serving
```bash
# Default (multi-thread enabled, 4 threads)
vllm serve Qwen/Qwen-Image --omni --port 8091
# Custom thread count
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --num-weight-load-threads 8
# Disable multi-thread loading
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
```
### Offline Inference
```python
from vllm_omni import Omni
# Default (multi-thread enabled, 4 threads)
omni = Omni(model="Qwen/Qwen-Image")
# Custom thread count
omni = Omni(
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
num_weight_load_threads=8,
)
```
### Benchmarks
Measured on NVIDIA H800:
| Model | Before | After | Speedup |
|-------|--------|-------|---------|
| **Qwen/Qwen-Image** (53.7 GiB) | 168s | 27s | **6.2x** |
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283s | 56s | **5.1x** |
## Learn More ## Learn More
**Cache Acceleration:** The Diffusion Acceleration navigation groups the remaining guides as follows:
- **[TeaCache Configuration Guide](diffusion/cache_acceleration/teacache.md)** - Parameter tuning, performance tips, troubleshooting | Area | Guide |
- **[Cache-DiT Advanced Guide](diffusion/cache_acceleration/cache_dit.md)** - DBCache, TaylorSeer, SCM techniques and optimization | --- | --- |
| Compatibility | [Feature Compatibility](feature_compatibility.md) |
| CPU offloading | [CPU Offloading](diffusion/cpu_offload.md) |
| Cache acceleration | [TeaCache](diffusion/cache_acceleration/teacache.md), [Cache-DiT](diffusion/cache_acceleration/cache_dit.md) |
| Parallelism | [Parallelism Overview](diffusion/parallelism/overview.md) |
| Attention | [Attention Backends](diffusion/attention_backends.md) |
| Compilation | [Regional Compilation](diffusion/regional_compilation.md) |
| Video extension | [Frame Interpolation](diffusion/frame_interpolation.md) |
| Startup | [Startup and Loading](diffusion/startup_and_loading.md) |
| Adapters | [LoRA](diffusion/lora.md) |
**Parallelism Methods:** Related cross-model and runtime features are documented separately:
- **[Parallelism Overview](diffusion/parallelism/overview.md)** - Tensor Parallelism, Sequence Parallelism, CFG Parallelism, Pipeline Parallelism, HSDP, and Expert Parallelism - [Quantization](quantization/overview.md) covers diffusion-only models,
multi-stage omni/TTS models, and multi-stage diffusion models.
**Memory Optimization:** - [Execution Modes and Streaming](diffusion/execution_modes.md) covers the
diffusion runtime, including batching, step execution, and streaming output.
- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
**Extensions:**
- **[LoRA Inference Guide](diffusion/lora.md)** - Low-Rank Adaptation for style customization and fine-tuning
- **[Frame Interpolation Guide](diffusion/frame_interpolation.md)** - Worker-side post-generation video frame interpolation for smoother motion
**Execution Modes:**
- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output
**Startup Optimization:**
- **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** - Speed up model startup by loading safetensors shards in parallel
**Advanced Topics:**
- **[Feature Compatibility](feature_compatibility.md)** - How to combine multiple features for maximum performance
-18
View File
@@ -2206,24 +2206,6 @@ class TestBaseConfigInheritance:
class TestPlatformOverrides: class TestPlatformOverrides:
"""Test platform-specific deploy config overrides.""" """Test platform-specific deploy config overrides."""
def test_qwen3_omni_musa_clears_parent_quantization_for_audio_stages(self):
pipeline = resolve_pipeline_config(
"qwen3_omni_moe",
Q3_OMNI_ALL_STAGES_HF_CONFIG,
)
assert isinstance(pipeline, PipelineConfig)
deploy_path = Path(get_deploy_config_path("qwen3_omni_moe.yaml"))
base_stages = merge_pipeline_deploy(pipeline, load_deploy_config(deploy_path))
assert "hf_overrides" not in base_stages[1].yaml_engine_args
assert "hf_overrides" not in base_stages[2].yaml_engine_args
musa = _apply_platform_overrides(load_deploy_config(deploy_path), platform="musa")
musa_stages = merge_pipeline_deploy(pipeline, musa)
expected = {"quantization_config": None}
assert musa_stages[1].yaml_engine_args["hf_overrides"] == expected
assert musa_stages[2].yaml_engine_args["hf_overrides"] == expected
def test_qwen3_tts_rocm_disables_code2wav_outer_cudagraph(self): def test_qwen3_tts_rocm_disables_code2wav_outer_cudagraph(self):
deploy_path = Path(get_deploy_config_path("qwen3_tts.yaml")) deploy_path = Path(get_deploy_config_path("qwen3_tts.yaml"))
+1 -76
View File
@@ -3,88 +3,13 @@ from types import SimpleNamespace
import pytest import pytest
import torch import torch
from vllm.v1.cudagraph_dispatcher import CUDAGraphMode
from vllm_omni.worker.gpu_model_runner import ( from vllm_omni.worker.gpu_model_runner import OmniGPUModelRunner, _filter_mrope_kwargs_for_model
OmniGPUModelRunner,
_filter_mrope_kwargs_for_model,
)
from vllm_omni.worker.omni_connector_model_runner_mixin import OmniConnectorModelRunnerMixin from vllm_omni.worker.omni_connector_model_runner_mixin import OmniConnectorModelRunnerMixin
pytestmark = [pytest.mark.core_model, pytest.mark.cpu] pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _runner_for_talker_graph_init(
*,
talker_mtp_graph_safe: bool | None,
has_separate_talker: bool = True,
model_stage: str = "talker",
) -> OmniGPUModelRunner:
runner = object.__new__(OmniGPUModelRunner)
runner.model = SimpleNamespace(
talker=object() if has_separate_talker else None,
talker_mtp=object(),
model_stage=model_stage,
)
if talker_mtp_graph_safe is not None:
runner.model.talker_mtp_graph_safe = talker_mtp_graph_safe
runner.model_config = SimpleNamespace(hf_text_config=SimpleNamespace(hidden_size=4))
runner.compilation_config = SimpleNamespace(
cudagraph_mode=CUDAGraphMode.FULL,
max_cudagraph_capture_size=1,
)
runner.vllm_config = object()
runner.max_num_reqs = 1
runner.dtype = torch.float32
runner._make_buffer = lambda *args, **kwargs: SimpleNamespace(args=args, kwargs=kwargs)
return runner
def test_talker_mtp_skips_graph_when_model_declares_unsafe(monkeypatch):
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=False)
talker_mtp = runner.model.talker_mtp
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: pytest.fail("graph wrapper must not be selected"),
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is talker_mtp
@pytest.mark.parametrize("model_stage", ["thinker", "code2wav"])
def test_non_talker_stage_does_not_use_talker_mtp_graph(monkeypatch, model_stage: str):
runner = _runner_for_talker_graph_init(
talker_mtp_graph_safe=None,
has_separate_talker=False,
model_stage=model_stage,
)
talker_mtp = runner.model.talker_mtp
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: pytest.fail("graph wrapper must not be selected"),
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is talker_mtp
@pytest.mark.parametrize("talker_mtp_graph_safe", [None, True])
def test_talker_mtp_uses_graph_for_legacy_or_explicit_safe_model(monkeypatch, talker_mtp_graph_safe):
runner = _runner_for_talker_graph_init(talker_mtp_graph_safe=talker_mtp_graph_safe)
wrapped = object()
monkeypatch.setattr(
"vllm_omni.worker.gpu_model_runner.current_omni_platform.get_graph_wrapper_cls",
lambda: lambda *args, **kwargs: wrapped,
)
OmniGPUModelRunner._init_talker_mtp(runner)
assert runner.talker_mtp is wrapped
class DummyBuffer: class DummyBuffer:
"""A minimal buffer wrapper that exposes the `.gpu` attribute.""" """A minimal buffer wrapper that exposes the `.gpu` attribute."""
-12
View File
@@ -69,18 +69,6 @@ stages:
repetition_penalty: 1.1 repetition_penalty: 1.1
platforms: platforms:
# The supported Qwen3-Omni ModelOpt checkpoint quantizes the thinker only. Clear the
# checkpoint-level metadata for the BF16 audio stages on the verified MUSA
# profile before vLLM performs its normal quantization auto-detection.
musa:
stages:
- stage_id: 1
hf_overrides:
quantization_config: null
- stage_id: 2
hf_overrides:
quantization_config: null
npu: npu:
stages: stages:
- stage_id: 0 - stage_id: 0
@@ -152,9 +152,6 @@ class Qwen3OmniMoeForConditionalGeneration(
dtype=torch.long, dtype=torch.long,
) )
elif self.model_stage == "talker": elif self.model_stage == "talker":
# The outer wrapper exposes talker_mtp for every stage, but only
# the talker stage owns the module that the method invokes.
self.talker_mtp_graph_safe = current_omni_platform.supports_talker_mtp_graph_capture()
multimodal_config.skip_mm_profiling = True multimodal_config.skip_mm_profiling = True
self.has_preprocess = True self.has_preprocess = True
self.has_postprocess = True self.has_postprocess = True
-5
View File
@@ -155,11 +155,6 @@ class OmniPlatform(Platform):
"""Check if the platform supports torch.compile with inductor backend.""" """Check if the platform supports torch.compile with inductor backend."""
raise NotImplementedError raise NotImplementedError
@classmethod
def supports_talker_mtp_graph_capture(cls) -> bool:
"""Whether a model may capture its dedicated talker MTP graph."""
return True
@classmethod @classmethod
def has_flash_attn_package(cls) -> bool: def has_flash_attn_package(cls) -> bool:
"""Check if a Flash Attention package is available and usable on this platform.""" """Check if a Flash Attention package is available and usable on this platform."""
-5
View File
@@ -115,11 +115,6 @@ class MUSAOmniPlatform(OmniPlatform, MUSAPlatformBase):
"""MUSA supports torch.compile with inductor backend.""" """MUSA supports torch.compile with inductor backend."""
return True return True
@classmethod
def supports_talker_mtp_graph_capture(cls) -> bool:
"""MUSA keeps Qwen3 talker MTP outside its dedicated FULL graph."""
return False
@classmethod @classmethod
def supports_float64(cls) -> bool: def supports_float64(cls) -> bool:
"""MUSA does not support float64 yet.""" """MUSA does not support float64 yet."""
+2 -6
View File
@@ -206,12 +206,8 @@ class OmniGPUModelRunner(GPUModelRunner):
cudagraph_mode = self.compilation_config.cudagraph_mode cudagraph_mode = self.compilation_config.cudagraph_mode
assert cudagraph_mode is not None assert cudagraph_mode is not None
has_separate_talker = getattr(self.model, "talker", None) is not None has_separate_talker = getattr(self.model, "talker", None) is not None
talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", None) talker_mtp_graph_safe = getattr(self.model, "talker_mtp_graph_safe", False)
# Preserve the legacy separate-talker default only when the model has if cudagraph_mode.has_full_cudagraphs() and (has_separate_talker or talker_mtp_graph_safe):
# not declared graph safety. An explicit False must disable wrapping.
if talker_mtp_graph_safe is None:
talker_mtp_graph_safe = has_separate_talker
if cudagraph_mode.has_full_cudagraphs() and talker_mtp_graph_safe:
graph_wrapper_cls = current_omni_platform.get_graph_wrapper_cls() graph_wrapper_cls = current_omni_platform.get_graph_wrapper_cls()
self.talker_mtp = graph_wrapper_cls(talker_mtp, self.vllm_config, runtime_mode=CUDAGraphMode.FULL) self.talker_mtp = graph_wrapper_cls(talker_mtp, self.vllm_config, runtime_mode=CUDAGraphMode.FULL)
# TTS exposes mtp_hidden_size; Omni uses hf_text_config.hidden_size. # TTS exposes mtp_hidden_size; Omni uses hf_text_config.hidden_size.