anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Model] FP8 Support For Boogu-Image (#6925)

Browse Source 
Signed-off-by: Dmaner <2663515256@qq.com>
This commit is contained in:
DMAN authored and GitHub committed 2026-09-06 04:48:38 +00:00
1 parent 1cd92553cf
commit 06541d1f52
15 files changed
+809 -78
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/quantization/online.md
+8 -8
View File
@@ -7,16 +7,16 @@ loading the model. Use it when you want memory savings without preparing a
separate quantized checkpoint.
This mode is different from pre-quantized checkpoint formats such as GGUF,
AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are
prepared before serving and are documented in their method-specific guides.
For MXFP8 and MXFP4, use this page for load-time quantization from BF16
checkpoints, and use the method-specific pages for offline checkpoints produced
by msModelSlim and the merge tools.
AutoRound, msModelSlim, serialized TorchAO checkpoints, or serialized Int8
checkpoints. Those formats are prepared before serving and are documented in
their method-specific guides. For MXFP8 and MXFP4, use this page for load-time
quantization from BF16 checkpoints, and use the method-specific pages for
offline checkpoints produced by msModelSlim and the merge tools.
## Hardware Support
| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 |
|--------|----------|-----------|------------|------------|
| -------- | ---------- | ----------- | ------------ | ------------ |
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
@@ -33,7 +33,7 @@ MXFP4 are documented for the Ascend NPU path.
### Diffusion Model (Qwen-Image, Wan2.2)
| Method | Guide | Example models | Status |
|--------|-------|----------------|--------|
| -------- | ------- | ---------------- | -------- |
| FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
| Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU |
@@ -88,7 +88,7 @@ config = build_quant_config({
## Parameters
| Parameter | Methods | Description |
|-----------|---------|-------------|
| ----------- | --------- | ------------- |
| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
| `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling |
docs/user_guide/quantization/overview.md
+22 -19
View File
@@ -11,21 +11,21 @@ For the internal architecture and backend extension points, see the
## Quantization Modes
| Mode | Guide | Description | Methods |
|------|-------|-------------|---------|
| ------ | ------- | ------------- | --------- |
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, BitsAndBytes W4, MXFP8 W8A8, MXFP4 W4A4 |
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, TorchAO, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
## Hardware Support
| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim |
|--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------|
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
| AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ |
| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim | TorchAO FP8 weight-only |
| -------- | ---------- | ----------- | ------------------ | ---------- | ------------ | ------------ | ----------- | ------------- | --------- |
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ⭕ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ✅ |
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ | ⭕ |
| AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ | ⭕ |
| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ | ⭕ |
| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ⭕ |
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
guide. FP8 on Ampere may use a weight-only path where available.
@@ -40,7 +40,7 @@ encoder, and VAE stay on the base checkpoint unless a method guide says
otherwise.
| Method | Guide | Mode | Example models | Status |
|--------|-------|------|----------------|--------|
| -------- | ------- | ------ | ---------------- | -------- |
| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo |
@@ -49,6 +49,7 @@ otherwise.
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy |
| AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven |
| msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path |
| TorchAO | [TorchAO](torchao.md) | Pre-quantized FP8 weight-only (W8A16) | Boogu-Image Base/Edit `-fp8` | Validated for the Boogu-Image diffusion transformer |
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -58,13 +59,14 @@ checkpoint contains a supported `quantization_config`; the non-AR stages stay
in BF16 unless the model guide explicitly adds support.
| Method | Guide | Scope | Example models | Status |
|--------|-------|-------|----------------|--------|
| -------- | ------- | ------- | ---------------- | -------- |
| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path |
| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints |
| TorchAO | [TorchAO](torchao.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
| msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
@@ -73,7 +75,7 @@ These models split generation across multiple stages. Quantization must be
attached to the intended stage rather than applied globally.
| Method | Guide | Scope | Example models | Status |
|--------|-------|-------|----------------|--------|
| -------- | ------- | ------- | ---------------- | -------- |
| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated |
@@ -81,6 +83,7 @@ attached to the intended stage rather than applied globally.
| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed |
| TorchAO | [TorchAO](torchao.md) | Stage-specific serialized checkpoint | BAGEL, GLM-Image | Not validated |
| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation |
!!! note
@@ -105,8 +108,8 @@ config = build_quant_config({
```
| Component | Default quantized? | Notes |
|-----------|--------------------|-------|
| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
| ----------- | -------------------- | ------- |
| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, TorchAO, and msModelSlim |
| Text encoder | No | Keep BF16 unless a method-specific guide documents support |
| VAE | No | Keep BF16; storage-only paths are method-specific |
| Scheduler/tokenizer | No | Loaded from the base model repository |
@@ -114,7 +117,7 @@ config = build_quant_config({
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
| Component | Default quantized? | Notes |
|-----------|--------------------|-------|
| ----------- | -------------------- | ------- |
| Thinker or AR language model | Yes, when checkpoint config is supported | ModelOpt FP8/NVFP4 or AutoRound checkpoint config |
| Audio encoder | No | BF16 |
| Vision encoder | No | BF16 |
@@ -124,7 +127,7 @@ config = build_quant_config({
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
| Component | Default quantized? | Notes |
|-----------|--------------------|-------|
| ----------- | -------------------- | ------- |
| Selected diffusion or transformer stage | Method-specific | Must be routed to the intended stage |
| Other generation stages | No | Keep BF16 unless separately validated |
| VAE, tokenizer, scheduler | No | Loaded from the base checkpoint |
@@ -204,7 +207,7 @@ The output JSON includes `output_metrics`, `reference_generation`, and
`candidate_generation`.
| Metric | Direction | Meaning |
|--------|-----------|---------|
| -------- | ----------- | --------- |
| `cosine_similarity` | Higher is better | Vector direction similarity between output pixels or frames. Useful as a broad sanity check. |
| `mae` | Lower is better | Mean absolute pixel or frame error. For decoded outputs, values are in uint8 pixel units. |
| `mse` / `rmse` | Lower is better | Squared error and its square root. These penalize localized large differences more than `mae`. |
@@ -217,7 +220,7 @@ The output JSON includes `output_metrics`, `reference_generation`, and
Recommended starting thresholds for same-seed diffusion comparisons:
| Metric | Smoke threshold | Stricter target | Notes |
|--------|-----------------|-----------------|-------|
| -------- | ----------------- | ----------------- | ------- |
| `psnr_db` | `>= 20.0` | `>= 25.0` | Good for quick image or frame regression checks. |
| `mae` | `<= 12.0` | `<= 6.0` | Interpreted in decoded uint8 pixel units. |
| `cosine_similarity` | `>= 0.98` | `>= 0.995` | Less sensitive to global scale than L2-style metrics. |
docs/user_guide/quantization/torchao.md
+94
View File
@@ -0,0 +1,94 @@
# TorchAO Quantization
## Overview
[TorchAO](https://github.com/pytorch/ao) provides quantization tools for
PyTorch and supports both pre-quantized checkpoint loading and runtime
quantization.
## Hardware Support
| Device | Support |
| -------- | --------- |
| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
| NVIDIA Ada GPU (SM 89) | ✅ |
| NVIDIA Hopper GPU (SM 90) | ⭕ |
| NVIDIA Ampere GPU (SM 80+) | ⭕ |
| AMD ROCm | ⭕ |
| Intel XPU | ⭕ |
| Ascend NPU | ⭕ |
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
guide.
## Model Type Support
### Diffusion Model (Boogu-Image)
| Model | Checkpoint | Scope | Scheme |
| --------------------- | ------------------------------------------------------------------ | -------------------------- | ----------------------- |
| Boogu-Image Base/Edit | `Boogu/Boogu-Image-0.1-Base-fp8`, `Boogu/Boogu-Image-0.1-Edit-fp8` | Diffusion transformer only | FP8 weight-only (W8A16) |
## Configuration
Using Boogu-Image as an example:
```bash
vllm serve Boogu/Boogu-Image-0.1-Base-fp8 \
--omni \
--port 8091 \
--diffusion-quantization-config \
'{"transformer":{"method":"torchao_float8_weight_only"}}'
```
The equivalent command using the complete serialized TorchAO configuration is:
```bash
vllm serve Boogu/Boogu-Image-0.1-Base-fp8 \
--omni \
--port 8091 \
--diffusion-quantization-config \
'{
"transformer": {
"method": "torchao",
"quant_type": {
"default": {
"_type": "Float8WeightOnlyConfig",
"_version": 2,
"_data": {
"weight_dtype": {
"_type": "torch.dtype",
"_data": "float8_e4m3fn"
},
"set_inductor_config": false
}
}
}
}
}'
```
## Parameters
| Parameter | Type | Default | Description |
| ------------ | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `method` | str | - | Use `torchao_float8_weight_only` for the serialized FP8 weight-only shorthand, or `torchao` with a full `quant_type` configuration. |
| `quant_type` | dict | - | Complete serialized TorchAO configuration used with `method: "torchao"`. |
See TorchAO's [vLLM integration guide](https://docs.pytorch.org/ao/0.17/eager_tutorials/torchao_vllm_integration.html#configuration-system)
for the serialized form and [workflow configs](https://docs.pytorch.org/ao/0.17/api_reference/api_ref_quantization.html#workflow-configs)
for available configuration classes and parameters.
## Validation and Notes
At load time, vLLM-Omni builds a `TorchAOConfig` from
`--diffusion-quantization-config`. For the Boogu-Image checkpoints documented
here, indexed PyTorch `.bin` shards are loaded through `pt_weights_iterator`.
When a component map is used, only the components included in the
configuration use TorchAO. Other components keep their own checkpoint and
runtime settings.
Use either the `torchao_float8_weight_only` shorthand or the equivalent full
`quant_type` configuration shown above. The checkpoint must already contain
weights quantized with TorchAO.