a1a53e5f94
Signed-off-by: david6666666 <530634352@qq.com> Co-authored-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com> Co-authored-by: lishunyang12 <125541396+lishunyang12@users.noreply.github.com>
71 lines
2.8 KiB
Markdown
71 lines
2.8 KiB
Markdown
# Adding Quantization Support to a Model
|
|
|
|
Use this checklist when extending quantization to a new architecture or stage.
|
|
|
|
## Classify Ownership
|
|
|
|
| Case | Owner |
|
|
|------|-------|
|
|
| New generic AR method, kernel, or loader semantics | upstream `vllm` first |
|
|
| Existing upstream method needs diffusion wiring | `vllm-omni` model and config integration |
|
|
| GGUF or ModelOpt tensor mapping for a diffusion architecture | `vllm-omni` adapter |
|
|
| NPU MXFP or msModelSlim diffusion path | `vllm-omni` platform/method wrapper and docs |
|
|
| Multi-stage omni checkpoint quantizes only the thinker LM | `vllm-omni` component scoping |
|
|
|
|
Avoid building a private quantization stack in model code.
|
|
|
|
## Model Wiring Checklist
|
|
|
|
1. Find the closest supported model implementation.
|
|
2. Confirm the model uses vLLM linear layers that accept `quant_config`.
|
|
3. Add `quant_config` to the transformer constructor.
|
|
4. Pass stable `prefix` values through every linear layer.
|
|
5. Handle fused modules and packed mappings explicitly.
|
|
6. Ensure `load_weights()` and any `WeightsMapper` remap `ignored_layers` or
|
|
`modules_to_not_convert`.
|
|
7. Keep tokenizer, scheduler, VAE, text encoder, audio/vision encoders, and
|
|
talker BF16 unless method-specific docs validate them.
|
|
8. Add or update a method doc under `docs/user_guide/quantization/`.
|
|
|
|
## Checkpoint Adapter Checklist
|
|
|
|
For pre-quantized checkpoints:
|
|
|
|
- Inspect `transformer/config.json` first.
|
|
- Check `quant_method`, `method`, `producer.name`, `quant_algo`, serialized
|
|
flags, `ignored_layers`, and any method-specific fields.
|
|
- Verify tensor name mapping before generation.
|
|
- Verify packed modules such as `to_qkv`, `add_kv_proj`, and `w13`.
|
|
- Confirm scale tensor names and shapes.
|
|
- Allow dequantization back to BF16 for intentionally skipped target layers.
|
|
|
|
## Sensitive Layer Discovery
|
|
|
|
Start conservative. Quantize the backbone first and keep output-side or routing
|
|
layers BF16 until quality is proven.
|
|
|
|
Frequently sensitive areas:
|
|
|
|
- attention output projections such as `to_out`
|
|
- FFN down projections such as `w2`
|
|
- final projections such as `proj_out`, `final_layer`, and output heads
|
|
- modulation, timestep, position, image, or text embedders
|
|
- refiner blocks
|
|
- MoE routers, experts selected by routing, and layernorm-adjacent modules
|
|
- VAE, text/vision towers, audio encoders, and codec/talker stages
|
|
|
|
Do not reuse one model family's ignore list blindly.
|
|
|
|
## Tests and Evidence
|
|
|
|
Minimum evidence for a PR:
|
|
|
|
- loader/config test or smoke command
|
|
- fixed-seed BF16 vs quantized output comparison
|
|
- quality metrics and saved outputs for precision-changing work
|
|
- memory and latency comparison under identical execution mode
|
|
- failure behavior for unsupported models or missing checkpoint fields
|
|
|
|
When the PR adds a method to a support table, include the exact model,
|
|
hardware, command, quality threshold, and artifact path used for validation.
|