anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: clarify quantization support matrix

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-13 15:50:56 +08:00
1 parent c98517ce7f
commit 00dab2ac95
12 files changed
+358 -160
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/quantization/bitsandbytes.md
+14 -6
View File
@@ -9,6 +9,10 @@ stay in BF16/FP16. No pre-quantized checkpoint is required.
This is an online (dynamic) path: load the normal HuggingFace checkpoint and
quantize during model loading via the `bitsandbytes` CUDA kernels.
The validation terms on this page follow the
[support levels](overview.md#support-levels). BitsAndBytes kernel support does
not imply that every diffusion transformer has been quality-validated.
## Hardware Support
| Device | Support |
@@ -21,7 +25,7 @@ quantize during model loading via the `bitsandbytes` CUDA kernels.
| Intel XPU | ❌ |
| Ascend NPU | ❌ |
Legend: `✅` supported, `❌` unsupported. Non-CUDA platforms raise in
Legend: `✅` backend available, `❌` unsupported. Non-CUDA platforms raise in
`get_quant_method()`; CUDA requires compute capability 7.5+ (`SM 75+`).
Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
@@ -30,11 +34,11 @@ Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
### Diffusion Model (Qwen-Image, Wan2.2)
| Model | HF models | CUDA | Mode | Recommendation |
|-------|-----------|:----:|------|----------------|
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Online W4 weight-only | All heavy linear layers; sensitive embedders stay BF16 |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Not validated | Online W4 weight-only | Compare vs BF16 before enabling |
| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Online W4 weight-only | Validate before enabling in docs |
| Model | HF models | Scope | Level | Recommendation |
|-------|-----------|-------|-------|----------------|
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Diffusion transformer | Validated | All heavy linear layers; sensitive embedders stay BF16 |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Diffusion transformer | Not validated | Compare with BF16 before enabling |
| Wan2.2 | Wan2.2 diffusion pipelines | One or both diffusion transformers | Not validated | Validate quality and cascade behavior before enabling |
Other diffusion models may work if their transformer uses supported linear
layers, but they are not validated in this guide.
@@ -104,6 +108,10 @@ Multi-GPU tensor parallelism (`tensor_parallel_size` > 1) is not validated for
BitsAndBytes in diffusion models; each rank quantizes its own weight shard
independently.
Z-Image-Turbo is validated outside scheduled full-model BitsAndBytes CI. The
real CUDA kernel smoke tests cover quantization and matrix multiplication, but
do not establish model-level quality for unlisted architectures.
If quality regresses, use `ignored_layers` to keep sensitive projections in BF16
(for example `to_out` or `w2`).