[Core] Unified quantization framework (#1764)

This commit is contained in:
SYLAR authored and GitHub committed 2026-03-24 18:56:55 +08:00
1 parent 2a630313ce
commit 6b93459826
40 files changed
+3306 -605

No files matched your search

+1
View File
@@ -79,6 +79,7 @@ nav:
- contributing/model/adding_omni_model.md
- contributing/model/adding_tts_model.md
- contributing/model/adding_diffusion_model.md
- contributing/model/adding_quantization_model.md
- CI: contributing/ci
- Design Documents:
- design/index.md
@@ -0,0 +1,478 @@
# Support FP8 Quantization
This section describes how to add FP8 quantization support to a diffusion model. We use the Qwen-Image pipeline as the reference implementation.
---
## Table of Contents
- [Overview](#overview)
- [Step-by-Step Implementation](#step-by-step-implementation)
- [Testing](#testing)
- [Troubleshooting](#troubleshooting)
- [Reference Implementations](#reference-implementations)
- [Summary](#summary)
---
## Overview
### What is FP8 Quantization?
FP8 quantization reduces model memory footprint by storing weights in 8-bit floating point (`float8_e4m3fn`) instead of 16-bit (`bfloat16`/`float16`). This saves ~50% of weight memory.
A diffusion pipeline typically has three quantizable components:
| Component | Layer Types | Quantization Approach |
|-----------|-----------|----------------------|
| **DiT (transformer)** | `nn.Linear` | Replace with vLLM quantized linear layers |
| **Text encoder** | `nn.Linear` | FP8 weight storage via hooks |
| **VAE** | `nn.Conv2d`, `nn.Conv3d` | FP8 weight storage via hooks |
### Architecture
The quantization system has two mechanisms:
**1. vLLM Quantized Linear Layers (for DiT)**
vLLM provides drop-in replacements for `nn.Linear` that perform FP8 W8A8 computation on supported hardware. These are configured via `quant_config` passed to layer constructors.
Key classes:
| Class | Location | Purpose |
|-------|----------|---------|
| `build_quant_config` | `vllm_omni/quantization/__init__.py` | Build quantization config from string/dict/per-component spec |
| `ComponentQuantizationConfig` | `vllm_omni/quantization/component_config.py` | Per-component routing by layer prefix |
**2. FP8 Weight Storage via Hooks (for text encoder and VAE)**
For models loaded via `from_pretrained()` (outside vLLM's weight pipeline), we use forward hooks to store weights in FP8 and dequantize on-the-fly:
```
At rest: weight.data = fp8_tensor (half memory)
Pre-hook: weight.data = fp8_weight.to(bf16) * scale
Forward: Normal computation in BF16
Post-hook: weight.data = fp8_weight (back to FP8)
```
Key function:
| Function | Location | Purpose |
|----------|----------|---------|
| `apply_fp8_weight_storage` | `vllm_omni/diffusion/models/utils.py` | Apply FP8 hooks to Linear/Conv layers (introduced in [#1412](https://github.com/vllm-project/vllm-omni/pull/1412)) |
---
## Step-by-Step Implementation
### Step 1: Add `quant_config` to the Transformer
Pass `quant_config` to vLLM parallel linear layers in the transformer. This enables FP8 W8A8 for DiT linear layers.
#### 1.1: Accept `quant_config` in Constructor
```python
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
class YourTransformer2DModel(nn.Module):
def __init__(
self,
*,
od_config: OmniDiffusionConfig,
quant_config: QuantizationConfig | None = None,
# ... other params
):
self.quant_config = quant_config
# Pass to sub-modules
self.blocks = nn.ModuleList([
YourTransformerBlock(quant_config=quant_config, ...)
for _ in range(num_layers)
])
```
#### 1.2: Pass `quant_config` to vLLM Linear Layers
Replace `nn.Linear` with vLLM parallel linear layers that accept `quant_config`:
```python
from vllm.model_executor.layers.linear import (
ColumnParallelLinear,
QKVParallelLinear,
RowParallelLinear,
)
class YourAttentionBlock(nn.Module):
def __init__(self, hidden_size, num_heads, quant_config=None, prefix=""):
# QKV projection
self.to_qkv = QKVParallelLinear(
hidden_size=hidden_size,
head_size=hidden_size // num_heads,
total_num_heads=num_heads,
quant_config=quant_config,
prefix=f"{prefix}.to_qkv",
)
# Output projection
self.to_out = RowParallelLinear(
input_size=hidden_size,
output_size=hidden_size,
quant_config=quant_config,
prefix=f"{prefix}.to_out",
)
class YourFeedForward(nn.Module):
def __init__(self, hidden_size, intermediate_size, quant_config=None, prefix=""):
self.gate_up_proj = ColumnParallelLinear(
input_size=hidden_size,
output_size=intermediate_size * 2,
quant_config=quant_config,
prefix=f"{prefix}.gate_up_proj",
)
self.down_proj = RowParallelLinear(
input_size=intermediate_size,
output_size=hidden_size,
quant_config=quant_config,
prefix=f"{prefix}.down_proj",
)
```
!!! important
Always pass the `prefix` parameter. It is used by `ignored_layers` matching to skip sensitive layers.
#### 1.3: Identify Sensitive Layers
Some layers degrade output quality when quantized. Common sensitive layers:
| Layer | Why Sensitive | Typical Models | Quant Methods Affected |
|-------|-------------|---------------|----------------------|
| `img_mlp` | Processes denoising latents with shifting dynamic range | Qwen-Image | FP8, Int8 |
| `feed_forward` | FFN layers in DiT blocks, large dynamic range | Z-Image | Int8 |
| `proj_out` | Final output projection, small quantization errors amplified | Various | FP8, Int8 |
| `lm_head` | Output vocabulary projection, precision-critical | Omni (thinker) | NVFP4 |
| `mlp.gate` | MoE router gates, routing decisions are precision-critical | MoE models | NVFP4 |
**Quantization quality benchmarks** (LPIPS: lower = better, <0.01 imperceptible, >0.1 noticeable):
| Model | Method | All Layers | With `ignored_layers` | Recommendation |
|-------|--------|-----------|----------------------|----------------|
| Qwen-Image-2512 | Int8 | LPIPS 0.0197 | 0.0027 (skip `img_mlp`) | Skip `img_mlp` |
| Z-Image-Turbo | Int8 | LPIPS 0.1597 | 0.0290 (skip `feed_forward`) | Skip `feed_forward` |
| Z-Image-Turbo | FP8 | LPIPS ~0.005 | — | All layers OK |
To identify sensitive layers for a new model:
1. Run inference with all layers quantized
2. Compare output quality with BF16 baseline using LPIPS (see `benchmarks/diffusion/quantization_quality.py`)
3. Selectively disable quantization per layer (via `ignored_layers`) until quality matches
4. Document the recommended `ignored_layers` in the model's supported models table
### Step 2: Wire Up Quantization in the Pipeline
#### 2.1: Create `quant_config` and Pass to Transformer
In the pipeline's `__init__`, extract the vLLM quant config and pass it to the transformer:
```python
from vllm_omni.quantization import build_quant_config
class YourPipeline(nn.Module):
def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = ""):
super().__init__()
# ... load scheduler, text_encoder, vae ...
# Build quant config from OmniDiffusionConfig
quant_config = build_quant_config(od_config.quantization_config)
# Pass to transformer
self.transformer = YourTransformer2DModel(
od_config=od_config,
quant_config=quant_config,
# ... other kwargs
)
```
#### 2.2: Apply FP8 Weight Storage to Text Encoder and VAE
For models loaded via `from_pretrained()`, apply hook-based FP8 weight storage.
!!! note
The `apply_fp8_weight_storage` function is provided by `vllm_omni/diffusion/models/utils.py`, introduced in [PR #1412](https://github.com/vllm-project/vllm-omni/pull/1412).
```python
from vllm_omni.diffusion.models.utils import apply_fp8_weight_storage
class YourPipeline(nn.Module):
def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = ""):
super().__init__()
self.text_encoder = SomeTextEncoder.from_pretrained(...)
self.vae = SomeVAE.from_pretrained(...).to(self.device)
# Apply FP8 weight storage to text encoder and VAE
if (
od_config.quantization_config is not None
and getattr(od_config.quantization_config, "quant_method", None) == "fp8"
):
apply_fp8_weight_storage(self.vae)
apply_fp8_weight_storage(self.text_encoder)
# ... rest of init
```
`apply_fp8_weight_storage` walks all `nn.Linear`, `nn.Conv2d`, and `nn.Conv3d` modules in the model and applies FP8 hooks to each.
#### 2.3: Fix `load_weights` to Include Pre-loaded Components
Since text encoders and VAEs are loaded via `from_pretrained()` (not through vLLM's weight pipeline), their parameters won't appear in the `loaded_weights` set. Mark them explicitly:
```python
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
loader = AutoWeightsLoader(self)
loaded_weights = loader.load_weights(weights)
# Mark pre-loaded component weights as loaded
loaded_weights |= {f"vae.{name}" for name, _ in self.vae.named_parameters()}
loaded_weights |= {f"text_encoder.{name}" for name, _ in self.text_encoder.named_parameters()}
return loaded_weights
```
### Step 3: Add to Supported Models Documentation
Update the FP8 supported models table in `docs/user_guide/diffusion/quantization/fp8.md`:
```markdown
| Your-Model | `org/Your-Model` | ✅ | ✅ | ✅ | `sensitive_layer` |
```
---
## Testing
### Quick Verification
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# Test with FP8 enabled
omni = Omni(model="your-model", quantization="fp8")
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
### Quality Gate Test (LPIPS)
We provide a pytest-based quality gate at `tests/diffusion/quantization/test_quantization_quality.py`.
It generates outputs with both BF16 and your quantized method using the same seed, computes
[LPIPS](https://github.com/richzhang/PerceptualSimilarity) perceptual distance, and **fails if it
exceeds a threshold**. This is the recommended way to validate that a quantization method does not
unacceptably degrade output quality.
**Step 1: Add your test config** — edit `QUALITY_CONFIGS` in the test file:
```python
QualityTestConfig(
id="int8_z_image", # pytest ID
model="Tongyi-MAI/Z-Image-Turbo",
quantization="int8", # your quantization method
task="t2i", # "t2i" or "t2v"
prompt="a cup of coffee on a wooden table, morning light",
max_lpips=0.15, # fail threshold (see guide below)
num_inference_steps=20,
),
```
**Step 2: Run the test:**
```bash
# Install dependency
pip install lpips
# Run all quality tests
pytest tests/diffusion/quantization/test_quantization_quality.py -v -m ""
# Run only your method
pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "int8"
# Run a specific model
pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "z_image"
```
**Step 3: Read the output:**
```
Quantization Quality: int8_z_image
Model: Tongyi-MAI/Z-Image-Turbo
Method: int8
LPIPS: 0.0290 (threshold: 0.15)
BF16 memory: 19.15 GiB
Quant memory: 14.80 GiB (23% reduction)
Result: PASS
```
**LPIPS threshold guide:**
| LPIPS Range | Meaning | Recommendation |
|-------------|---------|----------------|
| < 0.01 | Imperceptible | Ideal — no visible difference |
| 0.01 – 0.05 | Minor differences | Good — acceptable for most use cases |
| 0.05 – 0.15 | Noticeable on close inspection | Acceptable with `ignored_layers` tuning |
| 0.15 – 0.30 | Clearly visible | Investigate — likely needs `ignored_layers` |
| > 0.30 | Significant degradation | Quantization method may not suit this model |
!!! tip
If your test fails, don't raise the threshold — instead, identify sensitive layers and add
them to `ignored_layers` (see next section). Then re-run the test with the lower LPIPS.
### Quality Benchmark (Detailed)
For a full multi-prompt benchmark with Markdown output (suitable for PR descriptions), use the
standalone benchmark script:
```bash
python benchmarks/diffusion/quantization_quality.py \
--model Tongyi-MAI/Z-Image-Turbo \
--quantization fp8 \
--prompts \
"a cup of coffee on the table" \
"an aerial view of a coral reef" \
--height 1024 --width 1024 \
--num-inference-steps 50 --seed 42
```
This generates BF16 and quantized outputs, computes per-prompt LPIPS, and prints a Markdown
summary table ready to paste into your PR.
### Identifying Sensitive Layers with `ignored_layers`
Not all layers tolerate quantization equally. **`ignored_layers`** lets you keep specific layers
in BF16 while quantizing everything else. This is critical for achieving good quality with
aggressive quantization methods (Int8, NVFP4).
**Why some layers are sensitive:**
| Layer Type | Why Sensitive | Typical Impact |
|------------|--------------|----------------|
| `img_mlp` | Processes denoising latents with shifting dynamic range across timesteps. No built-in normalization to absorb quantization error. | Color shifts, blurriness |
| `feed_forward` | FFN layers with large dynamic range in DiT blocks. | Artifacts, loss of detail |
| `proj_out` | Final output projection — small errors get amplified since there are no subsequent layers to correct them. | Overall quality loss |
| `lm_head` | Vocabulary projection in LLMs — precision-critical for token selection. | Garbage text output |
| `mlp.gate` | MoE router gates — routing decisions are binary and precision-critical. | Wrong expert selection, quality collapse |
**How to find sensitive layers for your model:**
1. Run the quality gate test with all layers quantized:
```bash
pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "your_model"
```
2. If LPIPS exceeds your threshold, try skipping common sensitive layers one at a time:
```python
QualityTestConfig(
id="fp8_your_model_skip_mlp",
model="your-org/Your-Model",
quantization="fp8",
task="t2i",
prompt="a cup of coffee on a wooden table",
max_lpips=0.05,
),
```
And in the test, modify the Omni call:
```python
omni_qt = Omni(
model=config.model,
quantization_config={
"method": config.quantization,
"ignored_layers": ["img_mlp"],
},
)
```
3. Compare LPIPS with and without `ignored_layers` to isolate the problematic layers.
4. Document the recommended `ignored_layers` in the supported models table.
**Real-world examples:**
| Model | Method | All Layers (LPIPS) | With `ignored_layers` | Which Layers to Skip |
|-------|--------|-------------------|----------------------|---------------------|
| Qwen-Image-2512 | Int8 | 0.0197 | 0.0027 | `img_mlp` |
| Z-Image-Turbo | Int8 | 0.1597 | 0.0290 | `feed_forward` |
| Z-Image-Turbo | FP8 | ~0.005 | — | None needed |
### What to Verify (Checklist)
1. **Model loads without errors** — Check logs for quantization initialization messages
2. **Quality gate passes** — LPIPS within threshold vs BF16 baseline
3. **Memory reduction** — Expect ~30-50% savings on quantized components
4. **Sensitive layers identified** — Document which layers (if any) need `ignored_layers`
5. **`ignored_layers` pattern works** — Verify that the layer name prefix matches your transformer's module names
---
## Troubleshooting
### Issue: Output Quality Degradation
**Symptoms:** Generated images have visible artifacts, color shifts, or blurriness compared to BF16 baseline.
**Causes & Solutions:**
- **Sensitive layers being quantized:**
**Solution:** Identify and skip sensitive layers:
```python
omni = Omni(
model="your-model",
quantization_config={
"method": "fp8",
"ignored_layers": ["img_mlp"],
},
)
```
- **All layers sensitive (rare):**
**Solution:** The model may not be suitable for FP8 quantization. Consider weight-only quantization or other methods.
### Issue: Missing Weights Warning
**Symptoms:** Warnings about unloaded weights for `vae.*` or `text_encoder.*` parameters.
**Solution:** Ensure `load_weights` marks pre-loaded component weights. See [Step 2.3](#23-fix-load_weights-to-include-pre-loaded-components).
### Issue: CUDA Out of Memory
**Symptoms:** OOM errors during model loading despite FP8 being enabled.
**Causes & Solutions:**
- **Text encoder/VAE loaded in BF16 before FP8 conversion:**
FP8 weight storage is applied after `from_pretrained()`, so peak load memory is still BF16. This is a one-time cost during initialization.
---
## Reference Implementations
| Model | Pipeline File | Transformer File | Notes |
|-------|-------------|-----------------|-------|
| **Z-Image** | `models/z_image/pipeline_z_image.py` | `models/z_image/z_image_transformer.py` | DiT + text encoder FP8 |
| **Qwen-Image** | `models/qwen_image/pipeline_qwen_image.py` | `models/qwen_image/qwen_image_transformer.py` | DiT + text encoder + VAE FP8 |
All files are under `vllm_omni/diffusion/`.
---
## Summary
Adding FP8 quantization support to a new model:
1. ✅ **Add `quant_config` to transformer** — Accept and pass to vLLM parallel linear layers with proper `prefix`
2. ✅ **Identify sensitive layers** — Profile which layers need `ignored_layers` for quality
3. ✅ **Apply FP8 weight storage** — Call `apply_fp8_weight_storage()` on text encoder and VAE in pipeline `__init__`
4. ✅ **Fix `load_weights`** — Mark pre-loaded component weights as loaded
5. ✅ **Update docs** — Add model to the supported models table in `docs/user_guide/diffusion/quantization/fp8.md`
6. ✅ **Test** — Verify quality, memory savings, and sensitive layer behavior
+2
View File
@@ -139,6 +139,8 @@ def scan_package(package_name: str = "vllm_omni") -> dict[str, list[str]]:
# Skip excluded modules (avoid importing vllm during docs build)
excluded_prefixes = [
"vllm_omni.diffusion.models.qwen_image",
"vllm_omni.diffusion.quantization",
"vllm_omni.quantization",
"vllm_omni.entrypoints.async_diffusion",
"vllm_omni.entrypoints.openai",
"vllm_omni.model_executor.models.voxtral_tts.configuration_voxtral_tts",
@@ -63,6 +63,7 @@ The available `ignored_layers` names depend on the model architecture (e.g., `to
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Skip sensitive layers | `img_mlp` |
| Flux | `black-forest-labs/FLUX.1-dev` | All layers | None |
| HunyuanImage-3 | `tencent/HunyuanImage3` | All layers | None |
| HunyuanVideo-1.5 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `720p_t2v`, `480p_i2v` | All layers | None |
## Combining with Other Features
@@ -12,7 +12,7 @@
## Architecture Overview
1. `OmniDiffusionConfig` accepts `quantization` or `quantization_config`.
2. Diffusion quantization wrapper (`DiffusionGgufConfig`) produces vLLM `QuantizationConfig` objects for linear layers.
2. Diffusion quantization wrapper (`DiffusionGGUFConfig`) produces vLLM `QuantizationConfig` objects for linear layers.
3. `DiffusersPipelineLoader` branches on quantization method and loads either HF weights or GGUF weights for the transformer.
4. GGUF transformer loading is routed through model-specific adapters (e.g., Flux2Klein).
5. vLLM GGUF path uses `GGUFConfig` and `GGUFLinearMethod` for matmul.
@@ -1,22 +1,86 @@
# Quantization for Diffusion Transformers
# Quantization
vLLM-Omni supports quantization of DiT linear layers to reduce memory usage and accelerate inference.
vLLM-Omni provides a unified quantization framework that supports both diffusion models (DiT, text encoders, VAEs) and multi-stage omni models (e.g., Qwen3-Omni thinker). Quantization reduces memory usage and accelerates inference.
## Supported Methods
| Method | Guide |
|--------|-------|
| FP8 | [FP8](fp8.md) |
| Int8 | [Int8](int8.md) |
| GGUF | [GGUF](gguf.md) |
### Diffusion Models
| Method | Guide | Description | Tested Models | Min GPU |
|--------|-------|-------------|---------------|---------|
| FP8 | [FP8](fp8.md) | FP8 W8A8, dynamic or static | Z-Image, Qwen-Image, Flux, Bagel | SM 89 (Ada) |
| Int8 | [Int8](int8.md) | Int8 W8A8 | Z-Image, Qwen-Image | SM 89 (Ada) / Ascend NPU |
| GGUF | [GGUF](gguf.md) | GGUF format, dequant+GEMM for N-D tensors | Z-Image, Flux | SM 60 |
### Multi-stage Omni Models (Pre-quantized Checkpoints)
| Method | Description | Tested Models | Min GPU |
|--------|-------------|---------------|---------|
| ModelOpt FP8 | Pre-quantized FP8 via NVIDIA ModelOpt | Qwen3-Omni (thinker) | SM 89 (Ada/Hopper) |
| ModelOpt NVFP4 | Pre-quantized NVFP4 via NVIDIA ModelOpt | Qwen3-Omni (experimental, quality issues) | SM 100 (Blackwell) |
!!! note
AWQ, GPTQ, and BitsAndBytes are supported by vLLM's upstream quantization registry but have **not been tested** with vLLM-Omni pipelines. They may work via `build_quant_config()` but are not validated.
### Pre-quantized LLM Checkpoints (Multi-stage Models)
For multi-stage models like Qwen3-Omni, the unified quantization framework auto-detects
pre-quantized checkpoints via `quantization_config` in the HF config. Quantization is
automatically scoped to the thinker's `language_model` — audio encoder, vision encoder,
talker, and code2wav remain in BF16.
| Format | `quant_algo` | Hardware | Example | Status |
|--------|-------------|----------|---------|--------|
| ModelOpt FP8 | `FP8` | Ada/Hopper (SM 89+) | `asdazd/Qwen3-Omni-30B-A3B-Instruct_modelopt_FP8` | Tested — 47% memory reduction, comparable throughput |
| ModelOpt NVFP4 | `NVFP4` | Blackwell (SM 100+) | `shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4` | Experimental — loads and runs but output quality is unacceptable |
**Tested FP8 results (Qwen3-Omni, 1×H200):**
| Config | Model Memory (GiB) | Mem Reduction | Decode (tok/s) |
|--------|-------------------|---------------|----------------|
| BF16 baseline | 59.26 | — | 41.6 |
| FP8 (ModelOpt) | 31.41 | 47% | 39.9 |
FP8 enables the full 3-stage pipeline (thinker + talker + code2wav) on a single 64GB GPU,
which is impossible with BF16 (thinker alone requires 59.26 GiB).
## Quantization Scope
### Diffusion Models
When `--quantization fp8` is enabled for diffusion models:
| Component | What Gets Quantized | Mechanism |
|-----------|-------------------|-----------|
| **DiT (transformer)** | `nn.Linear` layers | vLLM W8A8 FP8 compute (Ada/Hopper) or weight-only (older GPUs) |
| **Text encoder** | `nn.Linear` layers | FP8 weight storage, BF16 compute |
| **VAE** | `nn.Conv2d`, `nn.Conv3d` layers | FP8 weight storage, BF16 compute |
### Multi-stage Omni Models
For pre-quantized omni model checkpoints, quantization is scoped per component:
| Component | Quantized? | Notes |
|-----------|-----------|-------|
| **Thinker (language_model)** | Yes | FP8 or NVFP4 via ModelOpt |
| **Audio encoder** | No | Stays BF16 |
| **Vision encoder** | No | Stays BF16 |
| **Talker** | No | Stays BF16 |
| **Code2Wav** | No | Stays BF16 |
!!! note
Not all models support all components. See the [FP8 supported models table](fp8.md#supported-models) for per-model details.
## Device Compatibility for FP8
| GPU Generation | Example GPUs | FP8 Mode |
|---------------|-------------------|----------|
| Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
| Device | Example Hardware | FP8 | NVFP4 |
|--------|-----------------|-----|-------|
| Blackwell GPU (SM 100+) | RTX 5090, B100, B200 | Yes | Yes (native FP4 HW) |
| Ada/Hopper GPU (SM 89+) | RTX 4090, H100, H200 | Yes (W8A8 native) | No |
| Turing/Ampere GPU (SM 75-86) | RTX 3090, A100 | Yes (weight-only Marlin) | No |
| Ascend NPU | Atlas 800T A2 (910B) | Not yet supported | No |
Kernel selection is automatic.
Kernel selection is automatic on CUDA GPUs.
## Device Compatibility for Int8
@@ -24,3 +88,62 @@ Kernel selection is automatic.
|-------------|---------------|-------------------|----------|
| NVIDIA GPU | Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
| Ascend NPU | Atlas A2/Atlas A3 | Atlas 800T A2/Atlas 900 A3 | Full W8A8 with native hardware |
## Python API
The `build_quant_config()` factory accepts multiple input formats:
```python
from vllm_omni.quantization import build_quant_config
# String
config = build_quant_config("fp8")
# Dict with parameters
config = build_quant_config({"method": "fp8", "activation_scheme": "static"})
# Per-component dict
config = build_quant_config({
"transformer": {"method": "fp8"},
"vae": None,
})
# None / "none"
config = build_quant_config(None) # returns None
```
## Architecture
```
build_quant_config(spec)
├── str → _build_single(method) → vLLM registry / _OVERRIDES
├── dict with "method" → _build_single(method, **kwargs)
├── per-component dict → ComponentQuantizationConfig
│ routes get_quant_method() by longest-prefix match
├── QuantizationConfig → passthrough
└── None / "none" → None
```
## Migration Guide
### Before (v0.14.x)
```python
# Old diffusion-specific API (removed)
from vllm_omni.diffusion.quantization import get_diffusion_quant_config
config = get_diffusion_quant_config("fp8", activation_scheme="static")
```
### After (v0.16.0+)
```python
# Unified API — delegates to vLLM's registry
from vllm_omni.quantization import build_quant_config
config = build_quant_config({"method": "fp8", "activation_scheme": "static"})
# Per-component (new)
config = build_quant_config({
"transformer": {"method": "fp8"},
"vae": None,
})
```