anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

（Phase 1）Add ModelOpt FP8 auto-detect support for diffusion checkpoints #2709 (#2913)

Browse Source 
Signed-off-by: roG0d <rodgarcas98@gmail.com>
Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
Signed-off-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com>
Co-authored-by: roG0d <rodgarcas98@gmail.com>
This commit is contained in:
baonudesifeizhaiandroG0d authored and GitHub committed 2026-05-09 16:55:05 +08:00
1 parent 40a07e0d80
commit c4a0990044
26 files changed
+1214 -176
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/quantization/fp8.md
+6 -6
View File
@@ -2,11 +2,11 @@
## Overview
FP8 quantization converts BF16/FP16 weights to FP8 at model load time, or loads
a checkpoint whose target stage already declares an FP8 quantization config.
Online activation scaling is the default and does not require calibration.
Static activation scaling is supported when calibrated scale information is
available.
FP8 quantization converts BF16/FP16 weights to FP8 at model load time. Online
activation scaling is the default and does not require calibration. Static
activation scaling is supported when calibrated scale information is available.
For ModelOpt-produced pre-quantized checkpoints, see
[ModelOpt Quantization](modelopt.md).
Some architectures can quantize all linear layers. Others have
quality-sensitive layers that should stay in BF16 through `ignored_layers`.
@@ -46,7 +46,7 @@ guide. FP8 on Ampere may use a weight-only path where available.
| Model | Scope | Format | Status |
|-------|-------|--------|--------|
| Qwen3-Omni | Thinker language-model stage | ModelOpt `quant_algo=FP8` | Tested for thinker memory reduction |
| Qwen3-Omni | Thinker language-model stage | [ModelOpt](modelopt.md) `quant_algo=FP8` | Tested for thinker memory reduction |
| Qwen3-TTS | TTS language-model stage | Checkpoint config | Not validated |
Audio encoder, vision encoder, talker, and code2wav stay in BF16 unless a