diff --git a/.claude/skills/quantization/SKILL.md b/.claude/skills/quantization/SKILL.md new file mode 100644 index 000000000..b58daaa28 --- /dev/null +++ b/.claude/skills/quantization/SKILL.md @@ -0,0 +1,103 @@ +--- +name: quantization +description: Work on vLLM-Omni quantization for diffusion, autoregressive, omni, or multi-stage models. Use when choosing or adding methods such as fp8, int8, gguf, mxfp8, mxfp4, mxfp4_dualscale, ModelOpt, AutoRound, INC, msModelSlim, awq, or gptq; debugging quantized loading; or validating memory, speed, and output quality. +--- + +# vLLM-Omni Quantization + +Use this skill for quantization work in `vllm-omni`. Start from the local +code and docs, then pick the smallest validated path for the model, method, +hardware, and quality target. + +## First Checks + +Before changing code or recommending a command, identify: + +- Model and task: AR text, omni/TTS, text-to-image, text-to-video, or multi-stage diffusion. +- Hardware backend: CUDA generation, Ascend NPU, Intel XPU, or another platform. +- Quantization mode: online from BF16/FP16 weights, pre-quantized checkpoint, runtime KV/FA quantization, or per-component routing. +- Scope: transformer/DiT, thinker language model, text encoder, VAE, audio/vision encoder, talker, or a specific stage. +- Validation target: memory reduction, latency/throughput, output quality, or all three. + +Do not assume a method is supported just because the CLI accepts the string. +Check `docs/user_guide/quantization/` and the closest implementation first. + +## Quick Decision + +| Task | Start With | +|------|------------| +| Choose a method or command | `references/methods.md` and `references/modality-compat.md` | +| Use `build_quant_config()` or per-component routing | `references/methods.md` | +| Work on diffusion quantization | `references/diffusion.md` | +| Add quantization to a new model | `references/adding-models.md` | +| Convert or load ModelOpt FP8 checkpoints | `references/modelopt-fp8.md` | +| Debug ModelOpt, GGUF, AutoRound, MXFP, or serialized Int8 loading | `references/diffusion.md` and method docs under `docs/user_guide/quantization/` | + +## Current Runtime Shape + +The unified entrypoint is: + +```python +from vllm_omni.quantization import build_quant_config +``` + +It supports method strings, flat method dictionaries, per-component +dictionaries, existing `QuantizationConfig` objects, and `None`. The factory +delegates generic methods to upstream `vllm` and keeps vLLM-Omni overrides for +diffusion or omni-specific routing: + +- `gguf` +- `int8` +- `mxfp8` +- `mxfp4` +- `mxfp4_dualscale` +- `inc`, `auto-round`, `auto_round` +- ModelOpt auto-detected configs: `modelopt`, `modelopt_fp4`, `modelopt_mixed` + +Use `ComponentQuantizationConfig` when only one stage or component should be +quantized. Pre-quantized ModelOpt-style checkpoints should not spill into +vision/audio encoders that have no corresponding scale tensors. + +## Ownership Boundary + +- Upstream `vllm` owns generic quantization configs, kernels, loader semantics, + hardware capability rules, and generic AR quantization methods. +- `vllm-omni` owns unified config routing, diffusion-specific wrappers, + component scoping, GGUF or ModelOpt checkpoint adapters, model-specific + prefix mapping, docs, examples, and validation. + +If a new method needs missing generic kernels or loader behavior, fix upstream +`vllm` first. In `vllm-omni`, add thin integration and model wiring. + +## Working Loop + +1. Read the local method doc under `docs/user_guide/quantization/`. +2. Inspect the active factory and config class in `vllm_omni/quantization/`. +3. Inspect the target model pipeline and transformer for stable prefixes and + whether every relevant vLLM linear layer receives `quant_config`. +4. For pre-quantized checkpoints, inspect `transformer/config.json` and the + checkpoint tensor names before running full generation. +5. Validate with the same prompt, seed, size, scheduler, steps, dtype, eager or + non-eager mode, and parallelism as the BF16 baseline. +6. Report quality, latency, and memory separately. Do not call a method + supported until quality is checked. + +## Common Mistakes + +| Symptom | Likely Cause | Fix | +|---------|--------------|-----| +| `--quantization` has no visible effect | Wrong scope or unsupported model path | Check component routing and method docs | +| Some layers stay BF16 unexpectedly | `quant_config` was not threaded into all vLLM linear layers | Audit transformer constructors and prefixes | +| Quality collapses but loading succeeds | Too many sensitive layers were quantized | Add model-specific `ignored_layers` and compare to BF16 | +| ModelOpt checkpoint loads but output is corrupted | Prefixes, packed-module mapping, scale routing, or BF16 fallback is wrong | Use `references/modelopt-fp8.md` | +| GGUF shape or tensor mismatch | Missing architecture-specific adapter | Add explicit adapter mapping; avoid generic fallback | +| Online and offline MXFP4 disagree | Wrong `mxfp4` vs `mxfp4_dualscale` mode | Check `docs/user_guide/quantization/mxfp4.md` | +| Quantized path is slower than BF16 | Kernel path, compile mode, dtype, or shape mismatch | Compare eager-to-eager and non-eager-to-non-eager | + +## References + +- General methods and config routing: [references/methods.md](references/methods.md) +- Modality and model scope: [references/modality-compat.md](references/modality-compat.md) +- Diffusion implementation patterns: [references/diffusion.md](references/diffusion.md) +- Adding model support: [references/adding-models.md](references/adding-models.md) +- ModelOpt FP8 conversion and loading: [references/modelopt-fp8.md](references/modelopt-fp8.md) diff --git a/.claude/skills/quantization/references/adding-models.md b/.claude/skills/quantization/references/adding-models.md new file mode 100644 index 000000000..ac9554de2 --- /dev/null +++ b/.claude/skills/quantization/references/adding-models.md @@ -0,0 +1,70 @@ +# Adding Quantization Support to a Model + +Use this checklist when extending quantization to a new architecture or stage. + +## Classify Ownership + +| Case | Owner | +|------|-------| +| New generic AR method, kernel, or loader semantics | upstream `vllm` first | +| Existing upstream method needs diffusion wiring | `vllm-omni` model and config integration | +| GGUF or ModelOpt tensor mapping for a diffusion architecture | `vllm-omni` adapter | +| NPU MXFP or msModelSlim diffusion path | `vllm-omni` platform/method wrapper and docs | +| Multi-stage omni checkpoint quantizes only the thinker LM | `vllm-omni` component scoping | + +Avoid building a private quantization stack in model code. + +## Model Wiring Checklist + +1. Find the closest supported model implementation. +2. Confirm the model uses vLLM linear layers that accept `quant_config`. +3. Add `quant_config` to the transformer constructor. +4. Pass stable `prefix` values through every linear layer. +5. Handle fused modules and packed mappings explicitly. +6. Ensure `load_weights()` and any `WeightsMapper` remap `ignored_layers` or + `modules_to_not_convert`. +7. Keep tokenizer, scheduler, VAE, text encoder, audio/vision encoders, and + talker BF16 unless method-specific docs validate them. +8. Add or update a method doc under `docs/user_guide/quantization/`. + +## Checkpoint Adapter Checklist + +For pre-quantized checkpoints: + +- Inspect `transformer/config.json` first. +- Check `quant_method`, `method`, `producer.name`, `quant_algo`, serialized + flags, `ignored_layers`, and any method-specific fields. +- Verify tensor name mapping before generation. +- Verify packed modules such as `to_qkv`, `add_kv_proj`, and `w13`. +- Confirm scale tensor names and shapes. +- Allow dequantization back to BF16 for intentionally skipped target layers. + +## Sensitive Layer Discovery + +Start conservative. Quantize the backbone first and keep output-side or routing +layers BF16 until quality is proven. + +Frequently sensitive areas: + +- attention output projections such as `to_out` +- FFN down projections such as `w2` +- final projections such as `proj_out`, `final_layer`, and output heads +- modulation, timestep, position, image, or text embedders +- refiner blocks +- MoE routers, experts selected by routing, and layernorm-adjacent modules +- VAE, text/vision towers, audio encoders, and codec/talker stages + +Do not reuse one model family's ignore list blindly. + +## Tests and Evidence + +Minimum evidence for a PR: + +- loader/config test or smoke command +- fixed-seed BF16 vs quantized output comparison +- quality metrics and saved outputs for precision-changing work +- memory and latency comparison under identical execution mode +- failure behavior for unsupported models or missing checkpoint fields + +When the PR adds a method to a support table, include the exact model, +hardware, command, quality threshold, and artifact path used for validation. diff --git a/.claude/skills/quantization/references/diffusion.md b/.claude/skills/quantization/references/diffusion.md new file mode 100644 index 000000000..1caeef0cf --- /dev/null +++ b/.claude/skills/quantization/references/diffusion.md @@ -0,0 +1,113 @@ +# Diffusion Quantization Reference + +Use this file for DiT quantization, config routing, checkpoint adapters, and +quality or loader debugging. + +## Supported Method Families + +The current local diffusion quantization families are: + +- CUDA/GPU online or checkpoint: `fp8`, `int8`, `gguf`, ModelOpt +- Ascend NPU: `int8`, `mxfp8`, `mxfp4`, `mxfp4_dualscale`, msModelSlim paths +- Intel XPU: AutoRound/INC and online MXFP8 paths where documented + +Always cross-check the method doc before claiming model support. The CLI +accepting a method string is not enough. + +## Implementation Starting Points + +| Area | Files | +|------|-------| +| Unified factory | `vllm_omni/quantization/factory.py` | +| Per-component routing | `vllm_omni/quantization/component_config.py` | +| GGUF diffusion method | `vllm_omni/quantization/gguf_config.py` | +| Int8 diffusion method | `vllm_omni/quantization/int8_config.py` | +| MXFP8 diffusion method | `vllm_omni/quantization/mxfp8_config.py` | +| MXFP4 and dual-scale method | `vllm_omni/quantization/mxfp4_config.py` | +| AutoRound/INC omni extension | `vllm_omni/quantization/inc_config.py` | +| ModelOpt checkpoint adapters | `vllm_omni/diffusion/model_loader/checkpoint_adapters/modelopt.py` | +| FP8 scaled-mm runtime patches | `vllm_omni/patch.py` | +| Similarity tool | `vllm_omni/quantization/tools/compare_diffusion_trajectory_similarity.py` | + +## Threading `quant_config` + +For a diffusion transformer: + +1. Accept `quant_config: QuantizationConfig | None` in the transformer. +2. Pass `quant_config` and stable `prefix` values to every vLLM linear layer. +3. Preserve packed-module mappings for fused QKV, FFN, or model-specific fused + projections. +4. Call `apply_vllm_mapper()` on configs that need HF-to-vLLM name remapping. +5. For component configs, keep prefixes specific enough to avoid quantizing VAE + or encoders by accident. + +Common skipped areas are `proj_out`, output projections, modulation layers, +embedders, refiner blocks, MoE routers, and layernorm-adjacent paths. + +## Online vs Offline + +Online quantization: + +- starts from BF16/FP16 checkpoint weights +- computes or stores quantized weights at load time +- typically uses `--quantization ` or `quantization=""` + +Offline or pre-quantized checkpoints: + +- include quantized weights and scales on disk +- usually carry `quantization_config` in `transformer/config.json` +- should auto-detect when the config is complete +- may require model-specific adapter logic for tensor names and packed modules + +Do not mix online flags with incompatible offline checkpoint configs. If the +active config and disk config differ, `resolve_quant_config_from_disk()` should +raise or rebuild rather than silently loading corrupted weights. + +## GGUF Rules + +- Require `quantization_config.gguf_model`. +- Use the normal base model for tokenizer, scheduler, text encoder, and VAE. +- Add explicit architecture adapters for tensor-name mapping. +- Use `named_parameters()` and `named_buffers()`, not `state_dict()`, to + discover loadable names. +- Guard fused QKV/KV rewrites so tensors are not rewritten twice. +- Flatten N-D activations around the GGUF matmul and restore shape. + +Unsupported architectures should fail clearly instead of falling back to a fake +generic mapper. + +## MXFP Rules + +- `mxfp8` is W8A8 microscaling, currently centered on NPU and documented XPU + behavior. +- `mxfp4` is online single-scale W4A4 on NPU. +- `mxfp4_dualscale` is online/offline dual-scale W4A4 with BF16 fallback; the + offline path is the production path when calibrated `mul_scale` exists. +- Offline dual-scale checkpoints must inject per-transformer `ignored_layers` + into config because cascade transformers can have different BF16 fallback + sets. + +Do not load an offline dual-scale checkpoint with the online single-scale path. + +## Common Failures + +| Symptom | Likely Cause | Fix | +|---------|--------------|-----| +| `Unknown quantization method` | Factory override or upstream registration is missing | Add the right override or upstream method | +| Missing scale tensors | Wrong scope, wrong adapter, or BF16 layer not allowed to dequantize | Check config scope and adapter scale routing | +| Output rank collapses from 3-D to 2-D | FP8 scaled-mm kernel ignored output shape | Check the patch in `vllm_omni/patch.py` | +| Quality degrades silently | Sensitive layers quantized | Add `ignored_layers`; compare to BF16 | +| Only one cascade transformer works | Disk config differs per transformer but active config is reused | Rebuild from each transformer's config | +| Quantized path slower than BF16 | Kernel, dtype, compile, or benchmark mode mismatch | Compare eager-to-eager and non-eager-to-non-eager | + +## Verification + +For a new or changed diffusion quantization path, run: + +1. Config and loader smoke test. +2. One fixed-seed generation against BF16. +3. Numeric comparison with the similarity tool or equivalent metrics. +4. Serving test if the path is user-facing online. +5. Memory and latency check under the same execution mode as the baseline. + +Keep output artifacts when precision changes are part of the PR. diff --git a/.claude/skills/quantization/references/methods.md b/.claude/skills/quantization/references/methods.md new file mode 100644 index 000000000..4a1e0a9b8 --- /dev/null +++ b/.claude/skills/quantization/references/methods.md @@ -0,0 +1,82 @@ +# General Quantization Methods + +Use this file for method selection and unified `vllm-omni` config syntax. For +diffusion-specific wiring, see `diffusion.md`. For ModelOpt checkpoint export +and loading details, see `modelopt-fp8.md`. + +## Unified Entry Point + +`vllm_omni.quantization.build_quant_config()` accepts: + +- `None` or `"none"` to disable quantization +- a method string such as `"fp8"`, `"int8"`, `"mxfp8"`, `"awq"`, or `"auto-round"` +- a flat dict with `"method"` or `"quant_method"` +- a per-component dict such as `{"transformer": {"method": "fp8"}, "vae": None}` +- an already-built `QuantizationConfig` + +Per-component dicts are routed by `ComponentQuantizationConfig` using +longest-prefix matching. Prefix stability matters: if a model remaps HF names +to vLLM names, apply the mapper to `ignored_layers` or component prefixes. + +## Local Overrides + +The local factory delegates most methods to upstream `vllm`, but has vLLM-Omni +overrides for methods that need diffusion, NPU, XPU, or component behavior: + +| Method | Local Config | Main Use | +|--------|--------------|----------| +| `gguf` | `DiffusionGGUFConfig` | Pre-quantized diffusion transformer weights | +| `int8` | `DiffusionInt8Config` | Online or serialized W8A8 diffusion transformers | +| `mxfp8` | `DiffusionMXFP8Config` | W8A8 MXFP8 diffusion on NPU and online XPU path | +| `mxfp4` | `DiffusionMXFP4Config` | Online W4A4 MXFP4 diffusion on NPU | +| `mxfp4_dualscale` | `DiffusionMXFP4DualScaleMixedConfig` | Online/offline W4A4 dual-scale plus BF16 fallback on NPU | +| `inc`, `auto-round`, `auto_round` | `OmniINCConfig` | AutoRound/INC checkpoint loading, including MXFP8 checkpoint metadata | +| ModelOpt config dicts | upstream ModelOpt configs through auto-detect | FP8, NVFP4, or mixed ModelOpt checkpoints | + +Check `SUPPORTED_QUANTIZATION_METHODS` in `vllm_omni/quantization/factory.py` +before adding a new method name. + +## AR and Generic Quantization + +For AR-backed models, upstream `vllm` methods usually apply: + +- `awq` and `gptq`: 4-bit or 8-bit weight quantization with calibration, + commonly used for text and AR stages. +- `fp8`: FP8 weight path or KV-cache FP8 depending on flags and hardware. +- `inc` or `auto-round`: pre-quantized checkpoints with Intel Neural + Compressor / AutoRound metadata. + +Use these for the AR language model stage only when the checkpoint and model +guide support it. Omni and TTS models often keep audio, vision, talker, and +codec stages in BF16. + +## Diffusion and Offline Methods + +For diffusion transformers, prefer the method-specific docs: + +- `docs/user_guide/quantization/fp8.md` +- `docs/user_guide/quantization/int8.md` +- `docs/user_guide/quantization/gguf.md` +- `docs/user_guide/quantization/modelopt.md` +- `docs/user_guide/quantization/mxfp8.md` +- `docs/user_guide/quantization/mxfp4.md` +- `docs/user_guide/quantization/autoround.md` +- `docs/user_guide/quantization/msmodelslim.md` + +Pre-quantized checkpoints should normally auto-detect from config. Runtime +flags such as `--force-cutlass-fp8`, `--linear-backend cutlass`, and +`--moe-backend cutlass` select kernels; they do not quantize a BF16 checkpoint. + +## Validation + +Use fixed-seed A/B comparisons and record: + +- model path and quantized checkpoint path +- method config and `ignored_layers` +- prompt, size, frame count, steps, scheduler, dtype, and seed +- eager/non-eager mode, compile flags, and parallelism +- output metrics such as PSNR, MAE, cosine similarity, SSIM, LPIPS, plus visual artifacts +- latency and peak memory, reported separately from quality + +The helper `vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity` +is the default lightweight image/video comparison tool for diffusion paths. diff --git a/.claude/skills/quantization/references/modality-compat.md b/.claude/skills/quantization/references/modality-compat.md new file mode 100644 index 000000000..188850e6b --- /dev/null +++ b/.claude/skills/quantization/references/modality-compat.md @@ -0,0 +1,47 @@ +# Quantization Compatibility by Modality + +Use this as a routing guide. The source of truth is the local method docs under +`docs/user_guide/quantization/` and the current model implementation. + +## Summary + +- AR and omni language-model quantization mostly follows upstream `vllm`, with + vLLM-Omni scoping for multi-stage models. +- Diffusion quantization is method-specific and model-specific. It usually + targets the diffusion transformer only. +- Pre-quantized checkpoints should quantize only the component that has + quantized weights and scale tensors. + +## Model Type Matrix + +| Model Type | Typical Quantization Scope | Notes | +|------------|----------------------------|-------| +| AR text models | Language model | Upstream `vllm` methods usually apply | +| Omni/TTS models | Thinker or AR language-model stage | Audio, vision, talker, and codec stages usually stay BF16 | +| DiT image models | Diffusion transformer | Check FP8, Int8, GGUF, ModelOpt, AutoRound docs | +| DiT video models | Diffusion transformer | HunyuanVideo FP8 and Wan2.2 MXFP paths are documented; validate other video paths | +| Multi-stage diffusion | Selected stage only | Use per-component routing and per-stage validation | +| Audio diffusion | Usually unsupported | Prefer dtype, offload, cache, or parallelism unless docs say otherwise | + +## Method Hints + +| Goal | Candidate | +|------|-----------| +| AR memory reduction | AWQ, GPTQ, AutoRound, ModelOpt where checkpoint supports it | +| CUDA diffusion W8A8 | FP8 or Int8 | +| Pre-quantized CUDA diffusion checkpoint | ModelOpt or GGUF | +| Ascend NPU diffusion | Int8, MXFP8, MXFP4, MXFP4 DualScale, msModelSlim | +| Intel XPU checkpoint | AutoRound/INC paths | +| Conservative image quality | selective FP8/Int8 with `ignored_layers` | +| Unsupported diffusion model | cache acceleration, CPU/layerwise offload, dtype, TP/SP/CFG parallelism | + +## Validation Rule + +Compatibility means more than successful loading. A method is compatible only +after it has: + +- correct component scope +- complete checkpoint/config metadata if offline +- fixed-seed quality comparison against BF16 +- acceptable latency and memory behavior +- docs or examples that state model, method, hardware, and caveats diff --git a/.claude/skills/quantization/references/modelopt-fp8.md b/.claude/skills/quantization/references/modelopt-fp8.md new file mode 100644 index 000000000..3c500e393 --- /dev/null +++ b/.claude/skills/quantization/references/modelopt-fp8.md @@ -0,0 +1,122 @@ +# ModelOpt FP8 Conversion and Loading + +Use this when exporting or loading ModelOpt FP8 checkpoints for diffusion +transformers. This reference captures the practical conversion lessons from the +2026-04-21 ModelOpt FP8 paste and aligns them with the current +`vllm-omni` ModelOpt adapter. + +## Core Rule + +Treat ModelOpt FP8 as selective checkpoint quantization, not one-click +whole-model FP8. Quantize the main transformer or DiT backbone, keep sensitive +or peripheral modules BF16, serialize a complete ModelOpt checkpoint, and let +vLLM-Omni auto-detect it from config. + +Do not pass `--quantization fp8` to force a ModelOpt checkpoint path. The +checkpoint's `quantization_config` should select `modelopt`. + +## Conversion Strategy + +Prefer: + +- backbone attention Q/K/V and FFN expansion projections +- conservative per-model ignore lists +- MHA quantizers disabled until basic linear quantization is stable +- full module-name prefixes in ignore/routing logic +- BF16 fallback for output-side and routing-sensitive layers + +Avoid quantizing by default: + +- VAE +- text, vision, or audio encoders +- embedders and modulation layers +- final heads and output projections +- refiner modules +- MoE routers and layernorm-sensitive blocks + +## Model-Specific Starting Points + +| Model Family | Quantize First | Keep BF16 First | +|--------------|----------------|-----------------| +| Z-Image | attention q/k/v, FFN w1/w3 | `to_out`, `w2`, `noise_refiner`, `context_refiner`, embedders, modulation, final/output | +| Qwen-Image | transformer backbone linear layers | `proj_out`, `img_in`, `txt_in`, norms, modulation, position/time/embed layers | +| FLUX2-dev / FLUX2-klein | backbone linear layers | `proj_out`, context/x embedders, `norm_out`, stream modulation, time embeddings | +| HunyuanImage-3.0 | `model.model` backbone | VAE, vision tower, token/time/image embedders, final image head, LM head, routers, layernorm-sensitive blocks | + +These are starting points, not support claims. Validate with BF16 outputs. + +## Required Checkpoint Metadata + +`transformer/config.json` must include a complete `quantization_config`. +At minimum it should identify ModelOpt and the algorithm: + +```json +{ + "quantization_config": { + "producer": {"name": "modelopt"}, + "quant_method": "modelopt", + "quant_algo": "FP8" + } +} +``` + +Current vLLM-Omni auto-detects: + +- `FP8` and `FP8_PER_CHANNEL_PER_TOKEN` -> `modelopt` +- `NVFP4` -> `modelopt_fp4` +- `MIXED_PRECISION` -> `modelopt_mixed` + +Packed modules and scales must map correctly: + +- `to_qkv` from `to_q`, `to_k`, `to_v` +- `add_kv_proj` from `add_q_proj`, `add_k_proj`, `add_v_proj` +- `w13` from `w1`, `w3` +- `.input_scale`, `.weight_scale`, `.weight_scale_2`, `.weight_scale_inv` + +For selective FP8, target layers that remain BF16 must be allowed to +dequantize from FP8 weights using the matching `weight_scale`, or the adapter +must skip the quantized scale tensors for that BF16 target. + +## Runtime Signals + +Healthy loading should show ModelOpt auto-detection and a ModelOpt linear +kernel path. Useful signals include: + +- auto-detected quantization from config +- selected ModelOpt FP8 linear method +- CUTLASS or FlashInfer FP8 scaled-mm backend when enabled and supported +- non-eager path compiling the transformer when expected + +If online serving fails while offline inference works, inspect warmup and +serving initialization before changing checkpoint conversion. + +## Validation Order + +1. Check config completeness. +2. Run an export checker such as `examples/quantization/check_modelopt_fp8_export.py`. +3. Confirm actual FP8 tensors and expected disk-size reduction. +4. Run one offline generation with fixed prompt, seed, size, and steps. +5. Run serving with `vllm serve --omni` and verify `/v1/images/generations`. +6. Compare against BF16 using visual review plus metrics such as LPIPS, PSNR, + MAE, cosine similarity, and SSIM. +7. Compare performance only under matched modes: + - BF16 eager vs FP8 eager + - BF16 non-eager vs FP8 non-eager + +FP8 is not automatically faster. Check whether execution reaches FP8 kernels, +whether compile mode is comparable, and whether the workload shape fits the +kernel path. + +## Triage + +| Symptom | First Check | +|---------|-------------| +| Startup failure | `config.json`, `model_index.json`, `quantization_config`, missing scales | +| Loads but output is corrupted | Over-quantized sensitive layers or wrong full-name prefixes | +| Offline works but online fails | Dummy warmup, serving init, compile path | +| FP8 is not faster | Kernel selection, dtype, compile mode, workload shape | +| BF16 fallback fails | Missing `weight_scale` for dequantization or wrong packed-module mapping | + +The success criterion is stable, correctly scoped checkpoint loading with +quality preserved enough for the target use case, not maximizing the number of +FP8 layers. diff --git a/.claude/skills/readme.md b/.claude/skills/readme.md index 36b0b8ed3..f42fcb6ab 100644 --- a/.claude/skills/readme.md +++ b/.claude/skills/readme.md @@ -21,6 +21,8 @@ include: - `diffusion-perf-opt`: guides diffusion model performance optimization, including profiling traces, parallel strategies, stage timing analysis, and benchmark-driven tuning +- `quantization`: guides quantization method selection, model integration, + checkpoint loading, and quality/performance validation for vLLM-Omni - `add-omni-model`: covers addition of new omni-modality model support - `add-tts-model`: covers integration of new TTS models and related serving workflows