Merge branch 'main' into codex-vae-spatial-decode-auto
This commit is contained in:
186 files changed
+10814
-1165
No files matched your search
+32
-26
@@ -40,30 +40,12 @@ nav:
|
||||
- Models:
|
||||
- models/supported_models.md
|
||||
- Features:
|
||||
- Sleep Mode: features/sleep_mode.md
|
||||
- Diffusion Features:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- Regional Compilation: user_guide/diffusion/regional_compilation.md
|
||||
- Cache Acceleration:
|
||||
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
|
||||
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
|
||||
- Attention Backends: user_guide/diffusion/attention_backends.md
|
||||
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
|
||||
- Parallelism:
|
||||
- Overview: user_guide/diffusion/parallelism/overview.md
|
||||
- CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md
|
||||
- Expert Parallel: user_guide/diffusion/parallelism/expert_parallel.md
|
||||
- Hybrid Sharded Data Parallel: user_guide/diffusion/parallelism/hsdp.md
|
||||
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
||||
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
||||
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||
- LoRA: user_guide/diffusion/lora.md
|
||||
- Custom Pipeline: features/custom_pipeline.md
|
||||
- Execution Modes: user_guide/diffusion/execution_modes.md
|
||||
- Overview: features/README.md
|
||||
- Runtime and Stage Execution:
|
||||
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
|
||||
- Sleep Mode: features/sleep_mode.md
|
||||
- Quantization:
|
||||
- Quantization: user_guide/quantization/overview.md
|
||||
- Overview: user_guide/quantization/overview.md
|
||||
- Online Quantization: user_guide/quantization/online.md
|
||||
- Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
|
||||
- FP8 W8A8: user_guide/quantization/fp8.md
|
||||
@@ -72,9 +54,33 @@ nav:
|
||||
- MXFP8 W8A8: user_guide/quantization/mxfp8.md
|
||||
- MXFP4 W4A4: user_guide/quantization/mxfp4.md
|
||||
- ModelOpt: user_guide/quantization/modelopt.md
|
||||
- GGUF: user_guide/quantization/gguf.md
|
||||
- AutoRound: user_guide/quantization/autoround.md
|
||||
- msModelSlim: user_guide/quantization/msmodelslim.md
|
||||
- Diffusion Acceleration:
|
||||
- Overview and Compatibility:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload.md
|
||||
- Cache Acceleration:
|
||||
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
|
||||
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
|
||||
- Parallelism:
|
||||
- Overview: user_guide/diffusion/parallelism/overview.md
|
||||
- CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md
|
||||
- Expert Parallel: user_guide/diffusion/parallelism/expert_parallel.md
|
||||
- Hybrid Sharded Data Parallel: user_guide/diffusion/parallelism/hsdp.md
|
||||
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
|
||||
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
|
||||
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
|
||||
- Attention Backends: user_guide/diffusion/attention_backends.md
|
||||
- Regional Compilation: user_guide/diffusion/regional_compilation.md
|
||||
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
|
||||
- Startup and Loading: user_guide/diffusion/startup_and_loading.md
|
||||
- LoRA: user_guide/diffusion/lora.md
|
||||
- Experimental:
|
||||
- Session State Manager: features/session_state_manager.md
|
||||
- Integrations:
|
||||
- Custom Diffusion Pipeline: features/custom_pipeline.md
|
||||
- ComfyUI: features/comfyui.md
|
||||
- Fault Tolerance:
|
||||
- Failure Modes: user_guide/fault_injection_reliability_matrix.md
|
||||
@@ -108,6 +114,7 @@ nav:
|
||||
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||
- Quantization: design/feature/quantization.md
|
||||
- Diffusion Acceleration:
|
||||
- Parallelism:
|
||||
- CFG-Parallel: design/feature/cfg_parallel.md
|
||||
@@ -117,9 +124,8 @@ nav:
|
||||
- Sequence Parallel: design/feature/sequence_parallel.md
|
||||
- Tensor Parallel: design/feature/tensor_parallel.md
|
||||
- VAE Patch Parallelism: design/feature/vae_parallel.md
|
||||
- Attention Backends:
|
||||
- Attention Optimization:
|
||||
- Skip-Softmax: design/feature/skip_softmax.md
|
||||
- Quantization: design/feature/quantization.md
|
||||
- Cache-DiT: design/feature/cache_dit.md
|
||||
- TeaCache: design/feature/teacache.md
|
||||
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||
|
||||
+13
-5
@@ -11,6 +11,10 @@ in the active navigation.
|
||||
|
||||
## Feature Design Documents
|
||||
|
||||
For user-facing configuration and current compatibility, see the
|
||||
[Features overview](../features/README.md). A design document defines an
|
||||
implementation contract; it is not, by itself, a general support claim.
|
||||
|
||||
### Runtime and stage execution
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
@@ -30,6 +34,10 @@ in the active navigation.
|
||||
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
### Quantization
|
||||
|
||||
- [Quantization](feature/quantization.md)
|
||||
|
||||
### Diffusion acceleration
|
||||
|
||||
#### Parallelism
|
||||
@@ -42,14 +50,14 @@ in the active navigation.
|
||||
- [Tensor Parallel](feature/tensor_parallel.md)
|
||||
- [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||
|
||||
#### Attention Backends
|
||||
#### Attention optimization
|
||||
|
||||
The [Diffusion Attention Backends](../user_guide/diffusion/attention_backends.md)
|
||||
guide lists all selectable backends and their platform defaults. Skip-Softmax
|
||||
is the backend-related optimization with a standalone feature-design contract:
|
||||
|
||||
- [Skip-Softmax](feature/skip_softmax.md)
|
||||
|
||||
#### Quantization
|
||||
|
||||
- [Quantization](feature/quantization.md)
|
||||
|
||||
- [Cache-DiT](feature/cache_dit.md)
|
||||
- [TeaCache](feature/teacache.md)
|
||||
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Features
|
||||
|
||||
Use this section to choose and configure vLLM-Omni runtime and optimization
|
||||
features. The navigation follows the active
|
||||
[Feature Design](../design/index.md#feature-design-documents) taxonomy whenever
|
||||
an implementation contract has a user-facing workflow.
|
||||
|
||||
!!! note "User guides, design documents, and recipes"
|
||||
|
||||
User guides explain how to enable a feature and where it is supported.
|
||||
Design documents define internal contracts and do not, by themselves,
|
||||
imply general or production support. For model-specific launch flags and
|
||||
hardware requirements, use the linked examples and validated recipes from
|
||||
[Supported Models](../models/supported_models.md).
|
||||
|
||||
## Runtime and Stage Execution
|
||||
|
||||
| Goal | User guide | Related design contract |
|
||||
| --- | --- | --- |
|
||||
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
|
||||
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
|
||||
|
||||
Some runtime designs are deliberately not promoted as standalone User Guide
|
||||
features yet:
|
||||
|
||||
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
|
||||
[OmniConnector implementations](../design/index.md#communication) describe
|
||||
topology and transport contracts. Practical deployment configuration remains
|
||||
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
|
||||
- [Async Chunk](../design/feature/async_chunk.md) and
|
||||
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
|
||||
are model- and pipeline-dependent. Use the selected model's deploy
|
||||
configuration and recipe for the supported settings.
|
||||
- [Automatic Prefix Caching](../design/feature/prefix_caching.md) remains a
|
||||
design-level contract until its user-facing configuration and compatibility
|
||||
surface is consolidated.
|
||||
|
||||
## Quantization
|
||||
|
||||
Quantization is a cross-model feature rather than a diffusion-only
|
||||
optimization. The unified [`quantization_config` guide](../user_guide/quantization/overview.md)
|
||||
covers diffusion-only models, multi-stage omni/TTS models, and multi-stage
|
||||
diffusion models. Its [design contract](../design/feature/quantization.md)
|
||||
defines the shared configuration and backend extension points.
|
||||
|
||||
## Diffusion Acceleration
|
||||
|
||||
| Goal | User guide | Related design contract |
|
||||
| --- | --- | --- |
|
||||
| Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) |
|
||||
| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [Distributed Layerwise Offload](../design/feature/distributed_layerwise_offload.md) |
|
||||
| Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) |
|
||||
| Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) |
|
||||
| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Skip-Softmax](../design/feature/skip_softmax.md) |
|
||||
| Compile repeated diffusion regions | [Regional Compilation](../user_guide/diffusion/regional_compilation.md) | User-facing optimization guide |
|
||||
| Add generated video frames | [Frame Interpolation](../user_guide/diffusion/frame_interpolation.md) | User-facing extension guide |
|
||||
| Reduce diffusion model startup time | [Startup and Loading](../user_guide/diffusion/startup_and_loading.md) | User-facing loading guide |
|
||||
| Apply diffusion adapters | [LoRA](../user_guide/diffusion/lora.md) | User-facing extension guide |
|
||||
|
||||
The [Pipeline Parallelism guide](../user_guide/diffusion/parallelism/pipeline_parallel.md)
|
||||
remains available by direct link, but it is not promoted in the primary User
|
||||
Guide navigation until its user-facing support and placement are settled.
|
||||
|
||||
## Experimental
|
||||
|
||||
[Session State Manager](session_state_manager.md) is opt-in and experimental.
|
||||
Its APIs and compatibility may change without notice.
|
||||
@@ -80,6 +80,7 @@ th {
|
||||
| `HiggsAudioV2ForConditionalGeneration` | Higgs-Audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✅︎ | | | | — |
|
||||
| `HiggsMultimodalQwen3ForConditionalGeneration` | Higgs-Audio v3 (TTS) | `bosonai/higgs-audio-v3-tts-4b` | ✅︎ | | | | [Published](https://recipes.vllm.ai/bosonai/higgs-audio-v3-tts-4b) |
|
||||
| `IndexTTS2TalkerForConditionalGeneration` | IndexTTS-2 | `IndexTeam/IndexTTS-2` | ✅︎ | | | | — |
|
||||
| `IndexTTS25TalkerForConditionalGeneration` | IndexTTS-2.5 | native `checkpoints/` bundle | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/IndexTeam/IndexTTS-2_5.md) |
|
||||
| `NextStep11Pipeline` | NextStep-1.1 | `stepfun-ai/NextStep-1.1` | ✅︎ | ✅︎ | | ✅︎ | — |
|
||||
| `MiMoAudioModel` | MiMo-Audio-7B-Instruct | `XiaomiMiMo/MiMo-Audio-7B-Instruct` | ✅︎ | ✅︎ | | | — |
|
||||
| `MiMoV2ASRForCausalLM` | MiMo-V2.5-ASR | `XiaomiMiMo/MiMo-V2.5-ASR` | ✅︎ | ✅︎ | | | — |
|
||||
|
||||
@@ -1,14 +1,18 @@
|
||||
# LoRA (Low-Rank Adaptation) Guide
|
||||
|
||||
LoRA (Low-Rank Adaptation) enables fine-tuning diffusion models by adding trainable low-rank matrices to existing model weights. vLLM-Omni currently supports PEFT-style LoRA adapters, allowing you to customize model behavior without modifying the base model weights.
|
||||
LoRA (Low-Rank Adaptation) enables fine-tuning diffusion models by adding trainable low-rank matrices to existing model weights. vLLM-Omni supports two LoRA backends: **PEFT** for PEFT-style adapters, and **Distill** usually for few-steps inference. **PEFT** backend allows you to customize model behavior without modifying the base model weights. **Distill** backend fuses LoRA weights into the base model at initialization.
|
||||
|
||||
## Overview
|
||||
|
||||
LoRA adapters are lightweight, model-specific fine-tuning weights that can be dynamically loaded and applied to diffusion models. vLLM-Omni uses a unified LoRA handling mechanism similar to vLLM with LRU cache management.
|
||||
LoRA adapters are lightweight, model-specific fine-tuning weights that can be applied to diffusion models in two ways:
|
||||
|
||||
- **PEFT backend** (`--lora-backend peft`, default): Loads a PEFT-format adapter folder via `DiffusionLoRAManager`. Adapters are cached (LRU) and activated per request via `LoRARequest`. It uses a unified LoRA handling mechanisms similar to vLLM with LRU cache management.
|
||||
- **Distill backend** (`--lora-backend distill`): Calls `pipeline.load_lora_weights` once at initialization to fuse one or more concrete checkpoint files directly into the base weights. Typically used for distilled few-step LoRAs (e.g. Lightning, LightX2V).
|
||||
|
||||
## LoRA Adapter Format
|
||||
|
||||
LoRA adapters must be in **PEFT (Parameter-Efficient Fine-Tuning)** format. A typical LoRA adapter directory structure:
|
||||
### PEFT (Parameter-Efficient Fine-Tuning) format (default)
|
||||
A typical PEFT-format LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
@@ -25,19 +29,20 @@ The `adapter_config.json` file contains metadata about the LoRA adapter, includi
|
||||
|
||||
### Offline Inference
|
||||
|
||||
#### Pre-loaded LoRA
|
||||
#### PEFT backend: pre-loaded LoRA
|
||||
|
||||
Load a LoRA adapter at initialization. This adapter is pre-loaded into the cache and can be activated by requests:
|
||||
Load a PEFT-format LoRA adapter at initialization. The adapter is pre-loaded into the cache and can be activated per request:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
|
||||
lora_path="/path/to/lora_adapter"
|
||||
lora_path = "/path/to/lora_adapter"
|
||||
|
||||
omni = Omni(
|
||||
model="stabilityai/stable-diffusion-3.5-medium",
|
||||
lora_path=lora_path
|
||||
lora_path=lora_path,
|
||||
lora_backend="peft", # default, can be omitted
|
||||
)
|
||||
|
||||
lora_request = LoRARequest(
|
||||
@@ -53,6 +58,98 @@ outputs = omni.generate(
|
||||
)
|
||||
```
|
||||
|
||||
#### Distill backend: fuse distilled LoRA at init
|
||||
|
||||
For distilled few-step LoRAs, pass `lora_backend="distill"` together with one or more concrete `.safetensors` files. The weights are fused into the base model once at init; subsequent `generate()` calls do not need a `LoRARequest`.
|
||||
|
||||
##### Supported pipelines
|
||||
|
||||
For Qwen-Image and Wan pipelines, the distill backend calls `pipeline.load_lora_weights(...)` during worker initialization.
|
||||
|
||||
| Pipeline | Supported distilled LoRA repo | Notes |
|
||||
|----------|--------------------------------|-------|
|
||||
| `QwenImagePipeline` | `lightx2v/Qwen-Image-2512-Lightning` | Used with Qwen-Image-2512 Lightning-style few-step inference. |
|
||||
| `Wan22Pipeline` | `lightx2v/Wan2.1-Distill-Loras`, `lightx2v/Wan2.2-Distill-Loras` | Wan2.1 uses one LoRA file. For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
|
||||
| `Wan22I2VPipeline` | `lightx2v/Wan2.2-Distill-Loras` | For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
|
||||
|
||||
Other diffusion pipelines are not currently listed as supporting distilled LoRA. Use the PEFT backend for request-time adapters, or bake converted weights into a local Diffusers directory before serving.
|
||||
|
||||
Single-file example (Qwen-Image-Lightning):
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="Qwen/Qwen-Image-2512",
|
||||
lora_path="/path/to/Qwen-Image-2512-Lightning.safetensors",
|
||||
lora_backend="distill",
|
||||
)
|
||||
|
||||
outputs = omni.generate(prompt="A piece of cheesecake")
|
||||
```
|
||||
|
||||
Multi-file example (Wan2.2 MoE, high + low noise):
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||||
lora_path=[
|
||||
"/path/to/wan2.2_high_noise_lora.safetensors", # -> transformer
|
||||
"/path/to/wan2.2_low_noise_lora.safetensors", # -> transformer_2
|
||||
],
|
||||
lora_backend="distill",
|
||||
)
|
||||
```
|
||||
|
||||
The CLI examples under `examples/offline_inference/` accept the same flags, e.g.:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--lora-backend distill \
|
||||
--lora-path /path/to/high.safetensors /path/to/low.safetensors \
|
||||
--prompt "A cat playing with yarn"
|
||||
```
|
||||
|
||||
##### Online serving
|
||||
|
||||
Distilled LoRAs can also be fused when an online diffusion server starts. The
|
||||
adapter remains active for the lifetime of that server, so requests do not pass
|
||||
a per-request `lora` field.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen-Image-2512 \
|
||||
--omni \
|
||||
--port 8091 \
|
||||
--lora-backend distill \
|
||||
--lora-path /path/to/Qwen-Image-2512-Lightning-4steps.safetensors
|
||||
```
|
||||
|
||||
Then send a normal generation request with the sampling settings expected by
|
||||
the distilled checkpoint:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [
|
||||
{"role": "user", "content": "A Chinese female college student, around 20 years old, with a very short haircut that conveys a gentle, artistic vibe. Her hair naturally falls to partially cover her cheeks, projecting a tomboyish yet charming demeanor. She has cool-toned fair skin and delicate features, with a slightly shy yet subtly confident expression—her mouth crooked in a playful, youthful smirk. She wears an off-shoulder top, revealing one shoulder, with a well-proportioned figure. The image is framed as a close-up selfie: she dominates the foreground, while the background clearly shows her dormitory—a neatly made bed with white linens on the top bunk, a tidy study desk with organized stationery, and wooden cabinets and drawers. The photo is captured on a smartphone under soft, even ambient lighting, with natural tones, high clarity, and a bright, lively atmosphere full of youthful, everyday energy."}
|
||||
],
|
||||
"extra_body": {
|
||||
"height": 1024,
|
||||
"width": 1024,
|
||||
"num_inference_steps": 4,
|
||||
"true_cfg_scale": 1.0,
|
||||
"seed": 42
|
||||
}
|
||||
}' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > qwen_image_distill_lora_output.png
|
||||
```
|
||||
|
||||
For Wan2.2 MoE serving, pass the high-noise checkpoint first and the low-noise
|
||||
checkpoint second to `--lora-path`.
|
||||
|
||||
!!! note "Server-side Path Requirement"
|
||||
The LoRA adapter path (`local_path`) must be readable on the **server** machine. If your client and server are on different machines, ensure the LoRA adapter is accessible via a shared mount or copied to the server.
|
||||
|
||||
@@ -141,6 +238,7 @@ Notes:
|
||||
|
||||
- This route avoids runtime LoRA loading changes in vLLM-Omni when you choose to bake converted weights into a local Diffusers directory.
|
||||
- Output quality and speed depend on the replacement checkpoints and sampling params you choose.
|
||||
- If you only need to fuse distilled LoRAs into a Wan2.2 checkpoint at load time (without the full LightX2V convert + assemble pipeline), you can instead pass them directly via `--lora-backend distill --lora-path <high>.safetensors <low>.safetensors`. See the [Distill backend](#distill-backend-fuse-distilled-lora-at-init) section above.
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Diffusion Startup and Loading
|
||||
|
||||
Large diffusion models can take several minutes to load at startup. vLLM-Omni
|
||||
loads safetensors shards in parallel to reduce this initialization time.
|
||||
|
||||
Multi-thread weight loading is enabled by default with four threads. No
|
||||
configuration is needed for the default behavior.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | CLI flag | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` | Pass the flag to disable multi-thread loading |
|
||||
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of parallel weight-loading threads |
|
||||
|
||||
!!! tip
|
||||
|
||||
The default balances startup speed and disk I/O contention. Fast NVMe
|
||||
storage may benefit from more threads, while network storage or hard disks
|
||||
may not.
|
||||
|
||||
## Online Serving
|
||||
|
||||
```bash
|
||||
# Default: multi-thread loading with four threads
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091
|
||||
|
||||
# Increase the thread count
|
||||
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni \
|
||||
--num-weight-load-threads 8
|
||||
|
||||
# Disable multi-thread loading
|
||||
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
|
||||
```
|
||||
|
||||
## Offline Inference
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
# Default: multi-thread loading with four threads
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
|
||||
# Increase the thread count
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
num_weight_load_threads=8,
|
||||
)
|
||||
```
|
||||
|
||||
## Reference Benchmarks
|
||||
|
||||
The following measurements were collected on NVIDIA H800 hardware. Treat them
|
||||
as reference results rather than a guarantee for other storage or hardware
|
||||
configurations.
|
||||
|
||||
| Model | Sequential loading | Multi-thread loading | Speedup |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Qwen/Qwen-Image** (53.7 GiB) | 168 s | 27 s | **6.2x** |
|
||||
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283 s | 56 s | **5.1x** |
|
||||
@@ -49,7 +49,7 @@ Parallelism methods distribute computation across GPUs without quality loss (mat
|
||||
|
||||
| Method | Description | Best For |
|
||||
|--------|-------------|----------|
|
||||
| **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds |
|
||||
| **[Multi-Thread Weight Loading](diffusion/startup_and_loading.md)** | Loads safetensors shards in parallel using a thread pool | All diffusion models; reduces startup from minutes to seconds |
|
||||
|
||||
**Note:** Some acceleration methods can be combined together for optimized performance. See [Feature Compatibility Table](#feature-compatibility) and [Feature Compatibility Tutorial](feature_compatibility.md) for detailed configuration examples.
|
||||
|
||||
@@ -100,7 +100,9 @@ The following tables show which models support each feature:
|
||||
|
||||
- **🔀SP (Ulysses & Ring)**: Includes both Ulysses-SP and Ring-Attention methods
|
||||
- ✅ = Fully supported
|
||||
- ✅* = Supported with the constraint listed below the table
|
||||
- ❌ = Not supported
|
||||
- ❓ = Not verified; not recommended
|
||||
|
||||
> Notes:
|
||||
|
||||
@@ -119,7 +121,7 @@ The following tables show which models support each feature:
|
||||
| **FLUX.2-dev** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **GLM-Image** | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅* |
|
||||
| **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
@@ -145,6 +147,7 @@ The following tables show which models support each feature:
|
||||
> 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
|
||||
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
|
||||
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
|
||||
> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
|
||||
|
||||
### VideoGen
|
||||
|
||||
@@ -155,13 +158,16 @@ The following tables show which models support each feature:
|
||||
| **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **LTX-2.3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
|
||||
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
|
||||
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
|
||||
|
||||
> **Step execution note:** Helios supports single-request step execution only;
|
||||
> use `max_num_seqs=1`.
|
||||
|
||||
**Frame Interpolation Support**
|
||||
|
||||
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines
|
||||
@@ -190,7 +196,7 @@ The following tables show which models support each feature:
|
||||
| **🔀Ring-Attn** | ✅ | ✅ | ✅ | | | | | | | | | | | |
|
||||
| **🔀CFG-Parallel** | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |
|
||||
| **🔀Tensor Parallel** | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | |
|
||||
| **🔀HSDP** | ❓ | ❓ | ❓ | ❓ | ❓ | ❌ | | | | | | | | |
|
||||
| **🔀HSDP** | ❓ | ❓ | ✅ | ❓ | ✅ | ❌ | | | | | | | | |
|
||||
| **🔀Expert Parallel** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | | | | | | |
|
||||
| **💾CPU Offloading (Layerwise)** | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | | | | | |
|
||||
| **💾CPU Offloading (Module-wise)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ❓ | ❌ | | | | | |
|
||||
@@ -201,97 +207,43 @@ The following tables show which models support each feature:
|
||||
|
||||
!!! info
|
||||
|
||||
1. Tensor Parallel and HSDP are not compatible.
|
||||
1. HSDP can be combined with Ulysses-SP or CFG-Parallel. Tensor Parallel
|
||||
and HSDP are not compatible; other HSDP combinations in the table
|
||||
remain unverified.
|
||||
2. TeaCache and Cache-DiT are not compatible.
|
||||
3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
|
||||
4. CPU Offloading (Layerwise) supports single-card for now.
|
||||
5. Using FP8-Quant as an example of qunatization methods.
|
||||
4. The CPU Offloading (Layerwise) row describes local layerwise offload.
|
||||
Multi-device Distributed Layerwise Offload has a separate topology and
|
||||
compatibility matrix in the [CPU Offloading guide](diffusion/cpu_offload.md#distributed-layerwise-offloading).
|
||||
5. The compatibility matrix uses FP8 as the representative quantization method.
|
||||
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
|
||||
|
||||
|
||||
## Multi-Thread Weight Loading
|
||||
|
||||
Large diffusion models can take several minutes to load weights at startup (e.g., ~3 min for Qwen-Image, ~5 min for Wan2.2 I2V 14B). Multi-thread weight loading speeds up this process by loading safetensors shards in parallel using a thread pool instead of sequentially.
|
||||
|
||||
This optimization is **enabled by default** with 4 threads. No configuration is needed for the default behavior.
|
||||
|
||||
### Configuration
|
||||
|
||||
| Parameter | CLI Flag | Default | Description |
|
||||
|-----------|----------|---------|-------------|
|
||||
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` (enabled) | Pass the flag to disable multi-thread loading |
|
||||
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of threads for parallel weight loading |
|
||||
|
||||
!!! tip
|
||||
The default of 4 threads balances speed and disk I/O contention. On fast NVMe storage you may benefit from more threads (e.g., 8). On HDD or network storage, the default of 4 avoids saturating I/O bandwidth.
|
||||
|
||||
### Online Serving
|
||||
|
||||
```bash
|
||||
# Default (multi-thread enabled, 4 threads)
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8091
|
||||
|
||||
# Custom thread count
|
||||
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --num-weight-load-threads 8
|
||||
|
||||
# Disable multi-thread loading
|
||||
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
|
||||
```
|
||||
|
||||
### Offline Inference
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
# Default (multi-thread enabled, 4 threads)
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
|
||||
# Custom thread count
|
||||
omni = Omni(
|
||||
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
|
||||
num_weight_load_threads=8,
|
||||
)
|
||||
```
|
||||
|
||||
### Benchmarks
|
||||
|
||||
Measured on NVIDIA H800:
|
||||
|
||||
| Model | Before | After | Speedup |
|
||||
|-------|--------|-------|---------|
|
||||
| **Qwen/Qwen-Image** (53.7 GiB) | 168s | 27s | **6.2x** |
|
||||
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283s | 56s | **5.1x** |
|
||||
The loading guide now lives at [Diffusion Startup and
|
||||
Loading](diffusion/startup_and_loading.md). This heading remains so existing
|
||||
links to this section continue to work.
|
||||
|
||||
## Learn More
|
||||
|
||||
**Cache Acceleration:**
|
||||
The Diffusion Acceleration navigation groups the remaining guides as follows:
|
||||
|
||||
- **[TeaCache Configuration Guide](diffusion/cache_acceleration/teacache.md)** - Parameter tuning, performance tips, troubleshooting
|
||||
- **[Cache-DiT Advanced Guide](diffusion/cache_acceleration/cache_dit.md)** - DBCache, TaylorSeer, SCM techniques and optimization
|
||||
| Area | Guide |
|
||||
| --- | --- |
|
||||
| Compatibility | [Feature Compatibility](feature_compatibility.md) |
|
||||
| CPU offloading | [CPU Offloading](diffusion/cpu_offload.md) |
|
||||
| Cache acceleration | [TeaCache](diffusion/cache_acceleration/teacache.md), [Cache-DiT](diffusion/cache_acceleration/cache_dit.md) |
|
||||
| Parallelism | [Parallelism Overview](diffusion/parallelism/overview.md) |
|
||||
| Attention | [Attention Backends](diffusion/attention_backends.md) |
|
||||
| Compilation | [Regional Compilation](diffusion/regional_compilation.md) |
|
||||
| Video extension | [Frame Interpolation](diffusion/frame_interpolation.md) |
|
||||
| Startup | [Startup and Loading](diffusion/startup_and_loading.md) |
|
||||
| Adapters | [LoRA](diffusion/lora.md) |
|
||||
|
||||
**Parallelism Methods:**
|
||||
Related cross-model and runtime features are documented separately:
|
||||
|
||||
- **[Parallelism Overview](diffusion/parallelism/overview.md)** - Tensor Parallelism, Sequence Parallelism, CFG Parallelism, Pipeline Parallelism, HSDP, and Expert Parallelism
|
||||
|
||||
**Memory Optimization:**
|
||||
|
||||
- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
|
||||
- **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
|
||||
- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
|
||||
|
||||
**Extensions:**
|
||||
|
||||
- **[LoRA Inference Guide](diffusion/lora.md)** - Low-Rank Adaptation for style customization and fine-tuning
|
||||
- **[Frame Interpolation Guide](diffusion/frame_interpolation.md)** - Worker-side post-generation video frame interpolation for smoother motion
|
||||
|
||||
**Execution Modes:**
|
||||
|
||||
- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output
|
||||
|
||||
**Startup Optimization:**
|
||||
|
||||
- **[Multi-Thread Weight Loading](#multi-thread-weight-loading)** - Speed up model startup by loading safetensors shards in parallel
|
||||
|
||||
**Advanced Topics:**
|
||||
|
||||
- **[Feature Compatibility](feature_compatibility.md)** - How to combine multiple features for maximum performance
|
||||
- [Quantization](quantization/overview.md) covers diffusion-only models,
|
||||
multi-stage omni/TTS models, and multi-stage diffusion models.
|
||||
- [Execution Modes and Streaming](diffusion/execution_modes.md) covers the
|
||||
diffusion runtime, including batching, step execution, and streaming output.
|
||||
@@ -88,8 +88,9 @@ python text_to_image.py \
|
||||
| `--vae-use-tiling` | flag | off | Enable VAE tiling for memory optimization |
|
||||
| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
|
||||
| `--enable-cpu-offload` | flag | off | Enable CPU offloading for diffusion models |
|
||||
| `--lora-path` | str | — | Path to PEFT LoRA adapter folder |
|
||||
| `--lora-path` | str | — | Path to PEFT LoRA adapter folder or checkpoint file |
|
||||
| `--lora-scale` | float | `1.0` | Scale factor for LoRA weights |
|
||||
| `--lora-backend` | str |`"peft"`| LoRA backend for loading LoRA adapters. Default: peft. Available options: peft, distill |
|
||||
|
||||
**NextStep-1.1 specific arguments:**
|
||||
|
||||
|
||||
@@ -125,6 +125,9 @@ python text_to_video.py \
|
||||
- `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
|
||||
- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF).
|
||||
- `--flow-shift`: scheduler flow_shift parameter.
|
||||
- `--lora-path`: path to PEFT LoRA adapter folder or checkpoint file.
|
||||
- `--lora-scale`: scale factor for LoRA weights.
|
||||
- `--lora-backend`: backend for loading LoRA adapters. Default: peft. Available options: peft, distill.
|
||||
|
||||
### Wan2.2-specific
|
||||
|
||||
|
||||
@@ -12,20 +12,20 @@ For the internal architecture and backend extension points, see the
|
||||
|
||||
| Mode | Guide | Description | Methods |
|
||||
|------|-------|-------------|---------|
|
||||
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, MXFP8 W8A8, MXFP4 W4A4 |
|
||||
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, BitsAndBytes W4, MXFP8 W8A8, MXFP4 W4A4 |
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
|
||||
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, GGUF, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
|
||||
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | FP8 W8A8 | Int8 W8A8 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | GGUF | AutoRound | msModelSlim |
|
||||
|--------|----------|-----------|----------|------------|------------|------|-----------|-------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
|
||||
| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ | ✅ | ⭕ | ⭕ | ❌ |
|
||||
| Intel XPU | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| Ascend NPU | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ |
|
||||
| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim |
|
||||
|--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ |
|
||||
| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
guide. FP8 on Ampere may use a weight-only path where available.
|
||||
@@ -43,10 +43,10 @@ otherwise.
|
||||
|--------|-------|------|----------------|--------|
|
||||
| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
|
||||
| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
|
||||
| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo |
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Pre-quantized FP8 checkpoints | Qwen-Image, Z-Image, FLUX.2, HunyuanImage-3.0 | Validated for ModelOpt FP8 diffusion checkpoints |
|
||||
| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Online W8A8 or offline pre-quantized | Wan2.2-T2V-A14B, I2V-A14B, TI2V-5B | Ascend NPU only; validated for Wan2.2 |
|
||||
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy |
|
||||
| GGUF | [GGUF](gguf.md) | Pre-quantized transformer weights | Qwen-Image | Validated where a model-specific GGUF adapter exists |
|
||||
| AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path |
|
||||
|
||||
@@ -61,9 +61,9 @@ in BF16 unless the model guide explicitly adds support.
|
||||
|--------|-------|-------|----------------|--------|
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path |
|
||||
| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| GGUF | [GGUF](gguf.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
|
||||
@@ -76,10 +76,10 @@ attached to the intended stage rather than applied globally.
|
||||
|--------|-------|-------|----------------|--------|
|
||||
| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated |
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Checkpoint-defined diffusion stage | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
|
||||
| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
|
||||
| GGUF | [GGUF](gguf.md) | Stage-specific transformer weights | BAGEL, GLM-Image | No validated adapter listed |
|
||||
| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation |
|
||||
|
||||
@@ -106,7 +106,7 @@ config = build_quant_config({
|
||||
|
||||
| Component | Default quantized? | Notes |
|
||||
|-----------|--------------------|-------|
|
||||
| Diffusion transformer | Yes | Primary target for FP8, Int8, ModelOpt, MXFP8, MXFP4, GGUF, AutoRound, and msModelSlim |
|
||||
| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
|
||||
| Text encoder | No | Keep BF16 unless a method-specific guide documents support |
|
||||
| VAE | No | Keep BF16; storage-only paths are method-specific |
|
||||
| Scheduler/tokenizer | No | Loaded from the base model repository |
|
||||
@@ -139,8 +139,8 @@ from vllm_omni.quantization import build_quant_config
|
||||
|
||||
build_quant_config("fp8")
|
||||
build_quant_config({"method": "fp8", "activation_scheme": "static"})
|
||||
build_quant_config("bitsandbytes")
|
||||
build_quant_config("auto-round", bits=4, group_size=128)
|
||||
build_quant_config({"method": "gguf", "gguf_model": "/path/to/model.gguf"})
|
||||
build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
|
||||
build_quant_config(None)
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user