diff --git a/docs/.nav.yml b/docs/.nav.yml index 0da0c779c..d6789a6cf 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -40,16 +40,31 @@ nav: - Models: - models/supported_models.md - Features: - - Sleep Mode: features/sleep_mode.md - - Diffusion Features: - - Overview: user_guide/diffusion_features.md - - Feature Compatibility: user_guide/feature_compatibility.md - - Regional Compilation: user_guide/diffusion/regional_compilation.md + - Overview: features/README.md + - Runtime and Stage Execution: + - Execution Modes and Streaming: user_guide/diffusion/execution_modes.md + - Sleep Mode: features/sleep_mode.md + - Diffusion Acceleration: + - Overview and Compatibility: + - Overview: user_guide/diffusion_features.md + - Feature Compatibility: user_guide/feature_compatibility.md + - Quantization: + - Overview: user_guide/quantization/overview.md + - Online Quantization: user_guide/quantization/online.md + - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md + - FP8 W8A8: user_guide/quantization/fp8.md + - Int8 W8A8: user_guide/quantization/int8.md + - BitsAndBytes W4: user_guide/quantization/bitsandbytes.md + - MXFP8 W8A8: user_guide/quantization/mxfp8.md + - MXFP4 W4A4: user_guide/quantization/mxfp4.md + - ModelOpt: user_guide/quantization/modelopt.md + - GGUF: user_guide/quantization/gguf.md + - AutoRound: user_guide/quantization/autoround.md + - msModelSlim: user_guide/quantization/msmodelslim.md + - CPU Offloading: user_guide/diffusion/cpu_offload.md - Cache Acceleration: - TeaCache: user_guide/diffusion/cache_acceleration/teacache.md - Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md - - Attention Backends: user_guide/diffusion/attention_backends.md - - Frame Interpolation: user_guide/diffusion/frame_interpolation.md - Parallelism: - Overview: user_guide/diffusion/parallelism/overview.md - CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md @@ -58,23 +73,15 @@ nav: - Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md - Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md - VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md - - CPU Offloading: user_guide/diffusion/cpu_offload.md + - Attention Backends: user_guide/diffusion/attention_backends.md + - Regional Compilation: user_guide/diffusion/regional_compilation.md + - Frame Interpolation: user_guide/diffusion/frame_interpolation.md + - Startup and Loading: user_guide/diffusion/startup_and_loading.md - LoRA: user_guide/diffusion/lora.md - - Custom Pipeline: features/custom_pipeline.md - - Execution Modes: user_guide/diffusion/execution_modes.md - - Quantization: - - Quantization: user_guide/quantization/overview.md - - Online Quantization: user_guide/quantization/online.md - - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md - - FP8 W8A8: user_guide/quantization/fp8.md - - Int8 W8A8: user_guide/quantization/int8.md - - BitsAndBytes W4: user_guide/quantization/bitsandbytes.md - - MXFP8 W8A8: user_guide/quantization/mxfp8.md - - MXFP4 W4A4: user_guide/quantization/mxfp4.md - - ModelOpt: user_guide/quantization/modelopt.md - - GGUF: user_guide/quantization/gguf.md - - AutoRound: user_guide/quantization/autoround.md - - msModelSlim: user_guide/quantization/msmodelslim.md + - Experimental: + - Session State Manager: features/session_state_manager.md + - Integrations: + - Custom Diffusion Pipeline: features/custom_pipeline.md - ComfyUI: features/comfyui.md - Fault Tolerance: - Failure Modes: user_guide/fault_injection_reliability_matrix.md diff --git a/docs/design/index.md b/docs/design/index.md index c98af378d..14dd3edb5 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -11,6 +11,10 @@ in the active navigation. ## Feature Design Documents +For user-facing configuration and current compatibility, see the +[Features overview](../features/README.md). A design document defines an +implementation contract; it is not, by itself, a general support claim. + ### Runtime and stage execution - [Disaggregated Inference](feature/disaggregated_inference.md) diff --git a/docs/features/README.md b/docs/features/README.md new file mode 100644 index 000000000..22e9b9582 --- /dev/null +++ b/docs/features/README.md @@ -0,0 +1,60 @@ +# Features + +Use this section to choose and configure vLLM-Omni runtime and optimization +features. The navigation follows the active +[Feature Design](../design/index.md#feature-design-documents) taxonomy whenever +an implementation contract has a user-facing workflow. + +!!! note "User guides, design documents, and recipes" + + User guides explain how to enable a feature and where it is supported. + Design documents define internal contracts and do not, by themselves, + imply general or production support. For model-specific launch flags and + hardware requirements, use the linked examples and validated recipes from + [Supported Models](../models/supported_models.md). + +## Runtime and Stage Execution + +| Goal | User guide | Related design contract | +| --- | --- | --- | +| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) | +| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide | + +Some runtime designs are deliberately not promoted as standalone User Guide +features yet: + +- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and + [OmniConnector implementations](../design/index.md#communication) describe + topology and transport contracts. Practical deployment configuration remains + under [Pipeline and deploy configurations](../configuration/stage_configs.md). +- [Async Chunk](../design/feature/async_chunk.md) and + [Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md) + are model- and pipeline-dependent. Use the selected model's deploy + configuration and recipe for the supported settings. +- [Automatic Prefix Caching](../design/feature/prefix_caching.md) remains a + design-level contract until its user-facing configuration and compatibility + surface is consolidated. + +## Diffusion Acceleration + +| Goal | User guide | Related design contract | +| --- | --- | --- | +| Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) | +| Reduce model memory with lower-precision weights | [Quantization](../user_guide/quantization/overview.md) | [Quantization](../design/feature/quantization.md) | +| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [Distributed Layerwise Offload](../design/feature/distributed_layerwise_offload.md) | +| Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) | +| Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) | +| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Skip-Softmax](../design/feature/skip_softmax.md) | +| Compile repeated diffusion regions | [Regional Compilation](../user_guide/diffusion/regional_compilation.md) | User-facing optimization guide | +| Add generated video frames | [Frame Interpolation](../user_guide/diffusion/frame_interpolation.md) | User-facing extension guide | +| Reduce diffusion model startup time | [Startup and Loading](../user_guide/diffusion/startup_and_loading.md) | User-facing loading guide | +| Apply diffusion adapters | [LoRA](../user_guide/diffusion/lora.md) | User-facing extension guide | + +The [Pipeline Parallelism guide](../user_guide/diffusion/parallelism/pipeline_parallel.md) +remains available by direct link, but it is not promoted in the primary User +Guide navigation until its user-facing support and placement are settled. + +## Experimental + +[Session State Manager](session_state_manager.md) is opt-in and experimental. +Its APIs and compatibility may change without notice. diff --git a/docs/user_guide/diffusion/startup_and_loading.md b/docs/user_guide/diffusion/startup_and_loading.md new file mode 100644 index 000000000..2b8d0d220 --- /dev/null +++ b/docs/user_guide/diffusion/startup_and_loading.md @@ -0,0 +1,60 @@ +# Diffusion Startup and Loading + +Large diffusion models can take several minutes to load at startup. vLLM-Omni +loads safetensors shards in parallel to reduce this initialization time. + +Multi-thread weight loading is enabled by default with four threads. No +configuration is needed for the default behavior. + +## Configuration + +| Parameter | CLI flag | Default | Description | +| --- | --- | --- | --- | +| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` | Pass the flag to disable multi-thread loading | +| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of parallel weight-loading threads | + +!!! tip + + The default balances startup speed and disk I/O contention. Fast NVMe + storage may benefit from more threads, while network storage or hard disks + may not. + +## Online Serving + +```bash +# Default: multi-thread loading with four threads +vllm serve Qwen/Qwen-Image --omni --port 8091 + +# Increase the thread count +vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni \ + --num-weight-load-threads 8 + +# Disable multi-thread loading +vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load +``` + +## Offline Inference + +```python +from vllm_omni import Omni + +# Default: multi-thread loading with four threads +omni = Omni(model="Qwen/Qwen-Image") + +# Increase the thread count +omni = Omni( + model="Wan-AI/Wan2.2-I2V-A14B-Diffusers", + num_weight_load_threads=8, +) +``` + +## Reference Benchmarks + +The following measurements were collected on NVIDIA H800 hardware. Treat them +as reference results rather than a guarantee for other storage or hardware +configurations. + +| Model | Sequential loading | Multi-thread loading | Speedup | +| --- | ---: | ---: | ---: | +| **Qwen/Qwen-Image** (53.7 GiB) | 168 s | 27 s | **6.2x** | +| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283 s | 56 s | **5.1x** | diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 6d76cb690..c810d582f 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -49,7 +49,7 @@ Parallelism methods distribute computation across GPUs without quality loss (mat | Method | Description | Best For | |--------|-------------|----------| -| **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds | +| **[Multi-Thread Weight Loading](diffusion/startup_and_loading.md)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds | **Note:** Some acceleration methods can be combined together for optimized performance. See [Feature Compatibility Table](#feature-compatibility) and [Feature Compatibility Tutorial](feature_compatibility.md) for detailed configuration examples. @@ -100,7 +100,9 @@ The following tables show which models support each feature: - **πŸ”€SP (Ulysses & Ring)**: Includes both Ulysses-SP and Ring-Attention methods - βœ… = Fully supported +- βœ…* = Supported with the constraint listed below the table - ❌ = Not supported +- ❓ = Not verified; not recommended > Notes: @@ -119,7 +121,7 @@ The following tables show which models support each feature: | **FLUX.2-dev** | βœ… | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… | ❌ | ❌ | | **GLM-Image** | ❌ | ❌ | ❌ | βœ… | βœ… | ❌ | βœ… | ❌ | ❌ | ❌ | ❌ | | **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | βœ… | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| **HunyuanImage3** | ❌ | βœ… | ❌ | ❌ | βœ… | ❌ | ❌ | ❌ | ❌ | βœ… | ❌ | +| **HunyuanImage3** | ❌ | βœ… | ❌ | ❌ | βœ… | ❌ | ❌ | ❌ | ❌ | βœ… | βœ…* | | **Krea 2** | ❌ | βœ… | ❌ | ❌ | ❌ | ❌ | βœ… | βœ… | βœ… (decode) | ❌ | ❌ | | **LongCat-Image** | βœ… | βœ… | βœ… | βœ… | βœ… | ❌ | ❌ | βœ… | ❌ | ❌ | ❌ | | **LongCat-Image-Edit** | βœ… | βœ… | βœ… | βœ… | βœ… | ❌ | ❌ | βœ… | ❌ | ❌ | ❌ | @@ -145,6 +147,7 @@ The following tables show which models support each feature: > 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary. > 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload. > 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_stepsβ‰ˆ8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`. +> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution). ### VideoGen @@ -155,13 +158,16 @@ The following tables show which models support each feature: | **Wan2.1-VACE** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… (decode) | ❌ | ❌ | | **LTX-2** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… (decode) | ❌ | ❌ | | **LTX-2.3** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… (decode) | ❌ | ❌ | -| **Helios** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | ❌ | ❌ | ❌ | +| **Helios** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | ❌ | ❌ | βœ…* | | **HunyuanVideo-1.5 T2V I2V** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… (encode/decode) | βœ… | ❌ | | **DreamID-Omni** | ❌ | ❌ | ❌ | βœ… | ❌ | ❌ | βœ… | βœ… | ❌ | ❌ | ❌ | | **Cosmos3** | ❌ | βœ… | βœ… | βœ… | βœ… | ❌ | βœ… | βœ… | βœ… (encode/decode) | βœ… | ❌ | | **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **MiniMax-H3** | βœ… (FL2VA) | βœ… | βœ… | ❌ | βœ… (DiT/TE) | ❌ | βœ… | βœ… | βœ… (tile) | βœ… (DiT) | ❌ | +> **Step execution note:** Helios supports single-request step execution only; +> use `max_num_seqs=1`. + **Frame Interpolation Support** - **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines @@ -190,7 +196,7 @@ The following tables show which models support each feature: | **πŸ”€Ring-Attn** | βœ… | βœ… | βœ… | | | | | | | | | | | | | **πŸ”€CFG-Parallel** | βœ… | βœ… | βœ… | βœ… | | | | | | | | | | | | **πŸ”€Tensor Parallel** | βœ… | βœ… | βœ… | βœ… | βœ… | | | | | | | | | | -| **πŸ”€HSDP** | ❓ | ❓ | ❓ | ❓ | ❓ | ❌ | | | | | | | | | +| **πŸ”€HSDP** | ❓ | ❓ | βœ… | ❓ | βœ… | ❌ | | | | | | | | | | **πŸ”€Expert Parallel** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | | | | | | | | **πŸ’ΎCPU Offloading (Layerwise)** | βœ… | βœ… | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | | | | | | | **πŸ’ΎCPU Offloading (Module-wise)** | βœ… | βœ… | βœ… | βœ… | βœ… | βœ… | ❓ | ❓ | ❌ | | | | | | @@ -201,66 +207,23 @@ The following tables show which models support each feature: !!! info - 1. Tensor Parallel and HSDP are not compatible. + 1. HSDP can be combined with Ulysses-SP or CFG-Parallel. Tensor Parallel + and HSDP are not compatible; other HSDP combinations in the table + remain unverified. 2. TeaCache and Cache-DiT are not compatible. 3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible. - 4. CPU Offloading (Layerwise) supports single-card for now. - 5. Using FP8-Quant as an example of qunatization methods. + 4. The CPU Offloading (Layerwise) row describes local layerwise offload. + Multi-device Distributed Layerwise Offload has a separate topology and + compatibility matrix in the [CPU Offloading guide](diffusion/cpu_offload.md#distributed-layerwise-offloading). + 5. The compatibility matrix uses FP8 as the representative quantization method. 6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches). ## Multi-Thread Weight Loading -Large diffusion models can take several minutes to load weights at startup (e.g., ~3 min for Qwen-Image, ~5 min for Wan2.2 I2V 14B). Multi-thread weight loading speeds up this process by loading safetensors shards in parallel using a thread pool instead of sequentially. - -This optimization is **enabled by default** with 4 threads. No configuration is needed for the default behavior. - -### Configuration - -| Parameter | CLI Flag | Default | Description | -|-----------|----------|---------|-------------| -| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` (enabled) | Pass the flag to disable multi-thread loading | -| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of threads for parallel weight loading | - -!!! tip - The default of 4 threads balances speed and disk I/O contention. On fast NVMe storage you may benefit from more threads (e.g., 8). On HDD or network storage, the default of 4 avoids saturating I/O bandwidth. - -### Online Serving - -```bash -# Default (multi-thread enabled, 4 threads) -vllm serve Qwen/Qwen-Image --omni --port 8091 - -# Custom thread count -vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --num-weight-load-threads 8 - -# Disable multi-thread loading -vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load -``` - -### Offline Inference - -```python -from vllm_omni import Omni - -# Default (multi-thread enabled, 4 threads) -omni = Omni(model="Qwen/Qwen-Image") - -# Custom thread count -omni = Omni( - model="Wan-AI/Wan2.2-I2V-A14B-Diffusers", - num_weight_load_threads=8, -) -``` - -### Benchmarks - -Measured on NVIDIA H800: - -| Model | Before | After | Speedup | -|-------|--------|-------|---------| -| **Qwen/Qwen-Image** (53.7 GiB) | 168s | 27s | **6.2x** | -| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283s | 56s | **5.1x** | +The loading guide now lives at [Diffusion Startup and +Loading](diffusion/startup_and_loading.md). This heading remains so existing +links to this section continue to work. ## Learn More