a1a53e5f94
Signed-off-by: david6666666 <530634352@qq.com> Co-authored-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com> Co-authored-by: lishunyang12 <125541396+lishunyang12@users.noreply.github.com>
48 lines
2.2 KiB
Markdown
48 lines
2.2 KiB
Markdown
# Quantization Compatibility by Modality
|
|
|
|
Use this as a routing guide. The source of truth is the local method docs under
|
|
`docs/user_guide/quantization/` and the current model implementation.
|
|
|
|
## Summary
|
|
|
|
- AR and omni language-model quantization mostly follows upstream `vllm`, with
|
|
vLLM-Omni scoping for multi-stage models.
|
|
- Diffusion quantization is method-specific and model-specific. It usually
|
|
targets the diffusion transformer only.
|
|
- Pre-quantized checkpoints should quantize only the component that has
|
|
quantized weights and scale tensors.
|
|
|
|
## Model Type Matrix
|
|
|
|
| Model Type | Typical Quantization Scope | Notes |
|
|
|------------|----------------------------|-------|
|
|
| AR text models | Language model | Upstream `vllm` methods usually apply |
|
|
| Omni/TTS models | Thinker or AR language-model stage | Audio, vision, talker, and codec stages usually stay BF16 |
|
|
| DiT image models | Diffusion transformer | Check FP8, Int8, GGUF, ModelOpt, AutoRound docs |
|
|
| DiT video models | Diffusion transformer | HunyuanVideo FP8 and Wan2.2 MXFP paths are documented; validate other video paths |
|
|
| Multi-stage diffusion | Selected stage only | Use per-component routing and per-stage validation |
|
|
| Audio diffusion | Usually unsupported | Prefer dtype, offload, cache, or parallelism unless docs say otherwise |
|
|
|
|
## Method Hints
|
|
|
|
| Goal | Candidate |
|
|
|------|-----------|
|
|
| AR memory reduction | AWQ, GPTQ, AutoRound, ModelOpt where checkpoint supports it |
|
|
| CUDA diffusion W8A8 | FP8 or Int8 |
|
|
| Pre-quantized CUDA diffusion checkpoint | ModelOpt or GGUF |
|
|
| Ascend NPU diffusion | Int8, MXFP8, MXFP4, MXFP4 DualScale, msModelSlim |
|
|
| Intel XPU checkpoint | AutoRound/INC paths |
|
|
| Conservative image quality | selective FP8/Int8 with `ignored_layers` |
|
|
| Unsupported diffusion model | cache acceleration, CPU/layerwise offload, dtype, TP/SP/CFG parallelism |
|
|
|
|
## Validation Rule
|
|
|
|
Compatibility means more than successful loading. A method is compatible only
|
|
after it has:
|
|
|
|
- correct component scope
|
|
- complete checkpoint/config metadata if offline
|
|
- fixed-seed quality comparison against BF16
|
|
- acceptable latency and memory behavior
|
|
- docs or examples that state model, method, hardware, and caveats
|