vllm-omni/docs/user_guide/quantization/modelopt.md

ModelOpt Quantization

Overview

ModelOpt quantization loads checkpoints produced by NVIDIA ModelOpt. The quantized weights and scale tensors are generated before serving, so inference does not run online calibration or convert a BF16 checkpoint at startup.

vLLM-Omni currently validates the ModelOpt FP8 checkpoint path for diffusion transformers. The loader auto-detects supported ModelOpt FP8 checkpoint configs and keeps non-transformer components, such as the tokenizer, scheduler, text encoder, and VAE, on the base checkpoint unless a model-specific guide says otherwise.

!!! note --force-cutlass-fp8 is an explicit runtime override for diffusion checkpoints that already carry a supported ModelOpt FP8 config. It does not quantize BF16 checkpoints and it does not apply to online --quantization fp8. The flag only takes effect for ModelOpt FP8 diffusion stages on CUDA SM89+ devices; other platforms and non-ModelOpt FP8 paths fall back to the normal vLLM kernel selection.

Supported ModelOpt Checkpoint Formats

vLLM-Omni treats ModelOpt checkpoints as pre-quantized checkpoints. The checkpoint config must identify ModelOpt as the quantization method or producer, and the quantization algorithm must be one of the validated FP8 algorithms.

Checkpoint field	Supported value
method / quant_method	modelopt
producer.name	modelopt
quant_algo	FP8, FP8_PER_CHANNEL_PER_TOKEN

Other ModelOpt algorithms, such as NVFP4, are not enabled by this diffusion FP8 path until they have separate model and quality validation.

Hardware Support

Device	Support
NVIDIA Blackwell GPU (SM 100+)	✅
NVIDIA Ada/Hopper GPU (SM 89+)	✅
NVIDIA Ampere GPU (SM 80+)	⭕
AMD ROCm	⭕
Intel XPU	⭕
Ascend NPU	❌

Legend: ✅ supported, ❌ unsupported, ⭕ not verified in this guide. The optional CUTLASS FP8 runtime override requires CUDA SM89+.

Model Type Support

Diffusion Model

Model	HF checkpoint	Scope	Status
Qwen-Image 2512	feizhai123/qwen-image-2512-modelopt-fp8-dynamic-all	Diffusion transformer	Validated for ModelOpt FP8 checkpoints
Z-Image	feizhai123/z-image-modelopt-fp8-conservative	Diffusion transformer	Validated for ModelOpt FP8 checkpoints
FLUX.2-dev	feizhai123/flux2-dev-modelopt-fp8	Diffusion transformer	Validated for ModelOpt FP8 checkpoints
FLUX.2-klein 4B	feizhai123/flux2-klein-4b-modelopt-fp8	Diffusion transformer	Validated for ModelOpt FP8 checkpoints
HunyuanImage-3.0	feizhai123/hunyuan-image3-modelopt-fp8	MoE diffusion transformer	Validated for ModelOpt FP8 checkpoints
Wan2.2	Not available	Diffusion transformer	Not validated

Multi-Stage Omni/TTS Model

Model	Scope	Status
Qwen3-Omni	Thinker language-model stage	ModelOpt FP8 checkpoint path
Qwen3-TTS	TTS language-model stage	Not validated

Audio encoder, vision encoder, talker, and code2wav stages stay in BF16 unless a model-specific guide documents otherwise.

Multi-Stage Diffusion Model

ModelOpt checkpoints must be routed to the stage whose checkpoint contains the ModelOpt quantization_config. BAGEL and GLM-Image are not listed as validated ModelOpt targets yet.

Configuration

For pre-quantized ModelOpt FP8 checkpoints, no --quantization fp8 flag is needed. The checkpoint config selects the ModelOpt path.

Online serving:

vllm serve <modelopt-fp8-checkpoint> \
  --omni \
  --tensor-parallel-size <N> \
  --force-cutlass-fp8

Offline inference:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model <modelopt-fp8-checkpoint> \
  --tensor-parallel-size <N> \
  --prompt "a red ceramic teapot on a wooden table" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 20 \
  --seed 42 \
  --output outputs/modelopt_fp8.png

Python API:

from vllm_omni import Omni

omni = Omni(
    model="<modelopt-fp8-checkpoint>",
    tensor_parallel_size=2,
    force_cutlass_fp8=True,
)

Parameters

Parameter	Type	Default	Description
force_cutlass_fp8 / --force-cutlass-fp8	bool	False	Force CUTLASS FP8 linear kernels for supported ModelOpt FP8 diffusion stages on CUDA SM89+

Validation and Notes

1. Compare the ModelOpt FP8 checkpoint against the BF16 baseline with the same prompt, resolution, seed, and inference steps.
2. Use tests/diffusion/quantization/test_quantization_quality.py with VLLM_OMNI_QUALITY_CONFIGS to validate local baseline and quantized model paths.
3. Report LPIPS, PSNR, MAE, throughput, latency, and peak memory when adding a new validated ModelOpt diffusion checkpoint.
4. Keep --quantization fp8 for online FP8 from BF16 checkpoints; use this ModelOpt path only when the checkpoint already contains ModelOpt FP8 weights and scales.