anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

（Phase 2）Add ModelOpt mixed FP8/NVFP4 support for image generation (#3570)

Browse Source 
Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
Signed-off-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com>
This commit is contained in:
baonudesifeizhai authored and GitHub committed 2026-05-22 10:39:10 +08:00
1 parent 7d8816b46b
commit 076c34abfb
16 files changed
+768 -68
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/quantization/modelopt.md
+80 -27
View File
@@ -6,34 +6,46 @@ ModelOpt quantization loads checkpoints produced by NVIDIA ModelOpt. The
quantized weights and scale tensors are generated before serving, so inference
does not run online calibration or convert a BF16 checkpoint at startup.
vLLM-Omni currently validates the ModelOpt FP8 checkpoint path for diffusion
transformers. The loader auto-detects supported ModelOpt FP8 checkpoint configs
and keeps non-transformer components, such as the tokenizer, scheduler, text
vLLM-Omni validates ModelOpt FP8, ModelOpt NVFP4, and ModelOpt mixed
FP8/NVFP4 checkpoint loading for diffusion transformer stages. The loader
auto-detects supported ModelOpt checkpoint configs and keeps non-transformer
components, such as the tokenizer, scheduler, text encoder, vision/audio
encoder, and VAE, on the base checkpoint unless a model-specific guide says
otherwise.
!!! note
`--force-cutlass-fp8` is an explicit runtime override for diffusion
checkpoints that already carry a supported ModelOpt FP8 config. It does not
quantize BF16 checkpoints and it does not apply to online `--quantization
fp8`. The flag only takes effect for ModelOpt FP8 diffusion stages on CUDA
SM89+ devices; other platforms and non-ModelOpt FP8 paths fall back to the
normal vLLM kernel selection.
ModelOpt checkpoints are pre-quantized checkpoints. Do not pass
`--quantization fp8` for these checkpoints. The checkpoint
`quantization_config` selects the ModelOpt path.
!!! note
`--force-cutlass-fp8`, `--linear-backend cutlass`, and
`--moe-backend cutlass` are runtime backend selections for checkpoints that
already carry supported ModelOpt quantized weights and scales. They do not
quantize BF16 checkpoints at startup.
## Supported ModelOpt Checkpoint Formats
vLLM-Omni treats ModelOpt checkpoints as pre-quantized checkpoints. The
checkpoint config must identify ModelOpt as the quantization method or producer,
and the quantization algorithm must be one of the validated FP8 algorithms.
and the quantization algorithm must be one of the validated algorithms below.
| Checkpoint field | Supported value |
|------------------|-----------------|
| `method` / `quant_method` | `modelopt` |
| `method` / `quant_method` | `modelopt`, `modelopt_fp4`, `modelopt_mixed` |
| `producer.name` | `modelopt` |
| `quant_algo` | `FP8`, `FP8_PER_CHANNEL_PER_TOKEN` |
| `quant_algo` | `FP8`, `FP8_PER_CHANNEL_PER_TOKEN`, `NVFP4`, `MIXED_PRECISION` |
Other ModelOpt algorithms, such as NVFP4, are not enabled by this diffusion
FP8 path until they have separate model and quality validation.
| `quant_algo` | Runtime method | Typical use |
|--------------|----------------|-------------|
| `FP8`, `FP8_PER_CHANNEL_PER_TOKEN` | `modelopt` | FP8 diffusion transformer checkpoints |
| `NVFP4` | `modelopt_fp4` | NVFP4 diffusion transformer checkpoints |
| `MIXED_PRECISION` | `modelopt_mixed` | Mixed FP8/NVFP4 checkpoints with a ModelOpt per-layer policy |
For multi-component diffusion or omni models, only the checkpoint component
that contains ModelOpt quantized weights should use the ModelOpt quantization
method. Encoders, decoders, tokenizers, schedulers, and other BF16 components
stay unquantized unless the model-specific recipe validates otherwise.
## Hardware Support
@@ -47,7 +59,10 @@ FP8 path until they have separate model and quality validation.
| Ascend NPU | ❌ |
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
The optional CUTLASS FP8 runtime override requires CUDA SM89+.
The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt NVFP4
and mixed FP8/NVFP4 diffusion checkpoints are currently validated on Blackwell
CUDA systems in the recipes below; other CUDA generations require separate
backend and quality validation.
## Model Type Support
@@ -56,12 +71,19 @@ The optional CUTLASS FP8 runtime override requires CUDA SM89+.
| Model | HF checkpoint | Scope | Status |
|-------|---------------|-------|--------|
| Qwen-Image 2512 | `feizhai123/qwen-image-2512-modelopt-fp8-dynamic-all` | Diffusion transformer | Validated for ModelOpt FP8 checkpoints |
| Qwen-Image 2512 | `feizhai123/qwen-image-2512-modelopt-mixed-fp8-sensitive-nvfp4-heavy` | Diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints |
| Z-Image | `feizhai123/z-image-modelopt-fp8-conservative` | Diffusion transformer | Validated for ModelOpt FP8 checkpoints |
| FLUX.2-dev | `feizhai123/flux2-dev-modelopt-fp8` | Diffusion transformer | Validated for ModelOpt FP8 checkpoints |
| FLUX.2-klein 4B | `feizhai123/flux2-klein-4b-modelopt-fp8` | Diffusion transformer | Validated for ModelOpt FP8 checkpoints |
| HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-fp8` | MoE diffusion transformer | Validated for ModelOpt FP8 checkpoints |
| HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8` | MoE diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints |
| Wan2.2 | Not available | Diffusion transformer | Not validated |
For full serving commands and benchmark context, see
[`recipes/Qwen/Qwen-Image.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image.md)
and
[`recipes/Tencent/HunyuanImage-3.0-Instruct.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md).
### Multi-Stage Omni/TTS Model
| Model | Scope | Status |
@@ -80,15 +102,28 @@ ModelOpt targets yet.
## Configuration
For pre-quantized ModelOpt FP8 checkpoints, no `--quantization fp8` flag is
needed. The checkpoint config selects the ModelOpt path.
For pre-quantized ModelOpt checkpoints, no `--quantization fp8` flag is needed.
The checkpoint config selects the ModelOpt path.
Online serving:
```bash
vllm serve <modelopt-fp8-checkpoint> \
vllm serve <modelopt-checkpoint> \
--omni \
--tensor-parallel-size <N> \
--linear-backend cutlass \
--force-cutlass-fp8
```
For mixed FP8/NVFP4 MoE checkpoints, also select the validated MoE backend:
```bash
vllm serve <modelopt-mixed-moe-checkpoint> \
--omni \
--tensor-parallel-size <N> \
--enable-expert-parallel \
--linear-backend cutlass \
--moe-backend cutlass \
--force-cutlass-fp8
```
@@ -96,14 +131,14 @@ Offline inference:
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model <modelopt-fp8-checkpoint> \
--model <modelopt-checkpoint> \
--tensor-parallel-size <N> \
--prompt "a red ceramic teapot on a wooden table" \
--height 1024 \
--width 1024 \
--num-inference-steps 20 \
--seed 42 \
--output outputs/modelopt_fp8.png
--output outputs/modelopt.png
```
Python API:
@@ -112,7 +147,7 @@ Python API:
from vllm_omni import Omni
omni = Omni(
model="<modelopt-fp8-checkpoint>",
model="<modelopt-checkpoint>",
tensor_parallel_size=2,
force_cutlass_fp8=True,
)
@@ -123,16 +158,34 @@ omni = Omni(
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `force_cutlass_fp8` / `--force-cutlass-fp8` | bool | `False` | Force CUTLASS FP8 linear kernels for supported ModelOpt FP8 diffusion stages on CUDA SM89+ |
| `--linear-backend cutlass` | str | auto | Select the validated CUTLASS linear backend for supported ModelOpt NVFP4 or mixed FP8/NVFP4 diffusion stages |
| `--moe-backend cutlass` | str | auto | Select the validated CUTLASS MoE backend for supported ModelOpt mixed MoE checkpoints |
## Validation and Notes
1. Compare the ModelOpt FP8 checkpoint against the BF16 baseline with the same
1. Compare the ModelOpt checkpoint against the BF16 baseline with the same
prompt, resolution, seed, and inference steps.
2. Use `tests/diffusion/quantization/test_quantization_quality.py` with
`VLLM_OMNI_QUALITY_CONFIGS` to validate local baseline and quantized model
paths.
3. Report LPIPS, PSNR, MAE, throughput, latency, and peak memory when adding a
new validated ModelOpt diffusion checkpoint.
4. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this
ModelOpt path only when the checkpoint already contains ModelOpt FP8 weights
and scales.
3. For HunyuanImage-3.0 quantized DiT checkpoints, the opt-in accuracy check is:
```bash
CUDA_VISIBLE_DEVICES=2,3 \
HUNYUAN_IMAGE3_RUN_QUANT_ACCURACY=1 \
HUNYUAN_IMAGE3_QUANT_DEVICES=0,1 \
HUNYUAN_IMAGE3_QUANT_TP=2 \
HUNYUAN_IMAGE3_BF16_MODEL=/path/to/hunyuan-image3-bf16 \
HUNYUAN_IMAGE3_FP8_MODEL=/path/to/hunyuan-image3-modelopt-fp8 \
HUNYUAN_IMAGE3_NVFP4_MODEL=/path/to/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8 \
PYTHONPATH=/path/to/vllm-omni:${PYTHONPATH:-} \
python -m pytest -s -v \
tests/e2e/accuracy/test_hunyuan_image3.py \
-k quantized_dit_matches_bf16_accuracy
```
4. Report CLIP score deltas, SSIM, PSNR, throughput, latency, and peak memory
when adding a new validated ModelOpt diffusion checkpoint.
5. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this
ModelOpt path only when the checkpoint already contains ModelOpt quantized
weights and scales.