a1a53e5f94
Signed-off-by: david6666666 <530634352@qq.com> Co-authored-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com> Co-authored-by: lishunyang12 <125541396+lishunyang12@users.noreply.github.com>
123 lines
4.7 KiB
Markdown
123 lines
4.7 KiB
Markdown
# ModelOpt FP8 Conversion and Loading
|
|
|
|
Use this when exporting or loading ModelOpt FP8 checkpoints for diffusion
|
|
transformers. This reference captures the practical conversion lessons from the
|
|
2026-04-21 ModelOpt FP8 paste and aligns them with the current
|
|
`vllm-omni` ModelOpt adapter.
|
|
|
|
## Core Rule
|
|
|
|
Treat ModelOpt FP8 as selective checkpoint quantization, not one-click
|
|
whole-model FP8. Quantize the main transformer or DiT backbone, keep sensitive
|
|
or peripheral modules BF16, serialize a complete ModelOpt checkpoint, and let
|
|
vLLM-Omni auto-detect it from config.
|
|
|
|
Do not pass `--quantization fp8` to force a ModelOpt checkpoint path. The
|
|
checkpoint's `quantization_config` should select `modelopt`.
|
|
|
|
## Conversion Strategy
|
|
|
|
Prefer:
|
|
|
|
- backbone attention Q/K/V and FFN expansion projections
|
|
- conservative per-model ignore lists
|
|
- MHA quantizers disabled until basic linear quantization is stable
|
|
- full module-name prefixes in ignore/routing logic
|
|
- BF16 fallback for output-side and routing-sensitive layers
|
|
|
|
Avoid quantizing by default:
|
|
|
|
- VAE
|
|
- text, vision, or audio encoders
|
|
- embedders and modulation layers
|
|
- final heads and output projections
|
|
- refiner modules
|
|
- MoE routers and layernorm-sensitive blocks
|
|
|
|
## Model-Specific Starting Points
|
|
|
|
| Model Family | Quantize First | Keep BF16 First |
|
|
|--------------|----------------|-----------------|
|
|
| Z-Image | attention q/k/v, FFN w1/w3 | `to_out`, `w2`, `noise_refiner`, `context_refiner`, embedders, modulation, final/output |
|
|
| Qwen-Image | transformer backbone linear layers | `proj_out`, `img_in`, `txt_in`, norms, modulation, position/time/embed layers |
|
|
| FLUX2-dev / FLUX2-klein | backbone linear layers | `proj_out`, context/x embedders, `norm_out`, stream modulation, time embeddings |
|
|
| HunyuanImage-3.0 | `model.model` backbone | VAE, vision tower, token/time/image embedders, final image head, LM head, routers, layernorm-sensitive blocks |
|
|
|
|
These are starting points, not support claims. Validate with BF16 outputs.
|
|
|
|
## Required Checkpoint Metadata
|
|
|
|
`transformer/config.json` must include a complete `quantization_config`.
|
|
At minimum it should identify ModelOpt and the algorithm:
|
|
|
|
```json
|
|
{
|
|
"quantization_config": {
|
|
"producer": {"name": "modelopt"},
|
|
"quant_method": "modelopt",
|
|
"quant_algo": "FP8"
|
|
}
|
|
}
|
|
```
|
|
|
|
Current vLLM-Omni auto-detects:
|
|
|
|
- `FP8` and `FP8_PER_CHANNEL_PER_TOKEN` -> `modelopt`
|
|
- `NVFP4` -> `modelopt_fp4`
|
|
- `MIXED_PRECISION` -> `modelopt_mixed`
|
|
|
|
Packed modules and scales must map correctly:
|
|
|
|
- `to_qkv` from `to_q`, `to_k`, `to_v`
|
|
- `add_kv_proj` from `add_q_proj`, `add_k_proj`, `add_v_proj`
|
|
- `w13` from `w1`, `w3`
|
|
- `.input_scale`, `.weight_scale`, `.weight_scale_2`, `.weight_scale_inv`
|
|
|
|
For selective FP8, target layers that remain BF16 must be allowed to
|
|
dequantize from FP8 weights using the matching `weight_scale`, or the adapter
|
|
must skip the quantized scale tensors for that BF16 target.
|
|
|
|
## Runtime Signals
|
|
|
|
Healthy loading should show ModelOpt auto-detection and a ModelOpt linear
|
|
kernel path. Useful signals include:
|
|
|
|
- auto-detected quantization from config
|
|
- selected ModelOpt FP8 linear method
|
|
- CUTLASS or FlashInfer FP8 scaled-mm backend when enabled and supported
|
|
- non-eager path compiling the transformer when expected
|
|
|
|
If online serving fails while offline inference works, inspect warmup and
|
|
serving initialization before changing checkpoint conversion.
|
|
|
|
## Validation Order
|
|
|
|
1. Check config completeness.
|
|
2. Run an export checker such as `examples/quantization/check_modelopt_fp8_export.py`.
|
|
3. Confirm actual FP8 tensors and expected disk-size reduction.
|
|
4. Run one offline generation with fixed prompt, seed, size, and steps.
|
|
5. Run serving with `vllm serve --omni` and verify `/v1/images/generations`.
|
|
6. Compare against BF16 using visual review plus metrics such as LPIPS, PSNR,
|
|
MAE, cosine similarity, and SSIM.
|
|
7. Compare performance only under matched modes:
|
|
- BF16 eager vs FP8 eager
|
|
- BF16 non-eager vs FP8 non-eager
|
|
|
|
FP8 is not automatically faster. Check whether execution reaches FP8 kernels,
|
|
whether compile mode is comparable, and whether the workload shape fits the
|
|
kernel path.
|
|
|
|
## Triage
|
|
|
|
| Symptom | First Check |
|
|
|---------|-------------|
|
|
| Startup failure | `config.json`, `model_index.json`, `quantization_config`, missing scales |
|
|
| Loads but output is corrupted | Over-quantized sensitive layers or wrong full-name prefixes |
|
|
| Offline works but online fails | Dummy warmup, serving init, compile path |
|
|
| FP8 is not faster | Kernel selection, dtype, compile mode, workload shape |
|
|
| BF16 fallback fails | Missing `weight_scale` for dequantization or wrong packed-module mapping |
|
|
|
|
The success criterion is stable, correctly scoped checkpoint loading with
|
|
quality preserved enough for the target use case, not maximizing the number of
|
|
FP8 layers.
|