anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Update diffusion quantization guides (#3200)

Browse Source 
Signed-off-by: David Chen <530634352@qq.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
This commit is contained in:
WeiQing ChenandSYLAR authored and GitHub committed 2026-04-28 04:33:24 -07:00
1 parent c6c7fd9b6d
commit 954d0642a4
17 files changed
+799 -1177
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/diffusion/quantization/autoround.md
-93
View File
@@ -1,93 +0,0 @@
# AutoRound Quantization
## Overview
[AutoRound](https://github.com/intel/auto-round) is an advanced quantization toolkit designed for Large Language Models (LLMs), Vision-Language Models (VLMs), and diffusion models. It achieves high accuracy at ultra-low bit widths (2–4 bits) with minimal tuning by leveraging sign-gradient descent, while providing broad hardware compatibility with multi-datatype support.
The quantization config is auto-detected from the checkpoint's `config.json` (`quantization_config.quant_method = "auto-round"`). No extra CLI flags are needed.
### Supported Schemes
| Scheme | Bits | Status |
|--------|------|--------|
| W4A16 | 4 | ✅ Supported |
| W8A16 | 8 | Planned |
W4A16 is the first supported scheme. Additional schemes will be added in future releases.
## Configuration
1. **Python API**: point `model` at a pre-quantized checkpoint. The quantization is detected automatically.
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16")
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=28),
)
outputs[0].save_images("output.png")
```
2. **CLI**: pass the quantized model path directly.
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \
--prompt "A cat sitting on a windowsill" \
--num-inference-steps 28 \
--output outputs/flux_w4a16.png
```
No `--quantization` flag is needed — the quantization method is read from the checkpoint.
## How It Works
The checkpoint's `config.json` contains:
```json
{
"quantization_config": {
"quant_method": "auto-round",
"bits": 4,
"group_size": 128,
"sym": true,
"packing_format": "auto_round:auto_gptq",
"block_name_to_quantize": "transformer_blocks,single_transformer_blocks"
}
}
```
At load time:
1. `TransformerConfig.from_dict()` parses the `quantization_config` section and builds a vLLM `INCConfig` via `build_quant_config("auto-round", ...)`.
2. `OmniDiffusionConfig.set_tf_model_config()` propagates the detected config to the engine.
3. The appropriate compute kernel (e.g. GPTQ-Marlin for W4A16) is selected automatically based on the checkpoint's bit-width and packing format.
## Supported Models
| Model | HF Checkpoint | Scheme | Group Size | Backend |
|-------|--------------|--------|------------|---------|
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | W4A16 | 128 | GPTQ-Marlin |
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | W4A16 | 128 | GPTQ-Marlin |
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | W4A16 | 128 | GPTQ-Marlin |
## Creating a Quantized Checkpoint
Use [AutoRound](https://github.com/intel/auto-round) to quantize a BF16 model. The `--scheme` flag selects the quantization scheme:
```bash
# W4A16 (4-bit weight, 16-bit activation)
auto-round \
--model black-forest-labs/FLUX.1-dev \
--scheme W4A16 \
--batch_size 1 \
--disable_opt_rtn \
--dataset coco2014 \
--iters 0
```
The output directory can be used directly as the `model` argument. See the [AutoRound documentation](https://github.com/intel/auto-round) for all available schemes and options.
docs/user_guide/diffusion/quantization/fp8.md
-82
View File
@@ -1,82 +0,0 @@
# FP8 Quantization
## Overview
FP8 quantization converts BF16/FP16 weights to FP8 at model load time. No calibration or pre-quantized checkpoint needed.
Depending on the model, either all layers can be quantized, or some sensitive layers should stay in BF16. See the [per-model table](#supported-models) for which case applies.
Common sensitive layers in DiT-based diffusion models include **image-stream MLPs** (`img_mlp`). These are particularly vulnerable to FP8 precision loss because they process denoising latents whose dynamic range shifts significantly across timesteps, and unlike attention projections (which benefit from QK-Norm stabilization), MLPs have no built-in normalization to absorb quantization error. In deep architectures (e.g., 60+ residual blocks), small per-layer errors compound and degrade output quality. Other layers such as **attention projections** (`to_qkv`, `to_out`) and **text-stream MLPs** (`txt_mlp`) are generally more robust due to normalization or more stable input statistics.
## Configuration
1. **Python API**: set `quantization="fp8"`. To skip sensitive layers, use `quantization_config` with `ignored_layers`.
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# All layers quantized
omni = Omni(model="<your-model>", quantization="fp8")
# Skip sensitive layers
omni = Omni(
model="<your-model>",
quantization_config={
"method": "fp8",
"ignored_layers": ["<layer-name>"],
},
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
2. **CLI**: pass `--quantization fp8` and optionally `--ignored-layers`.
```bash
# All layers
python text_to_image.py --model <your-model> --quantization fp8
# Skip sensitive layers
python text_to_image.py --model <your-model> --quantization fp8 --ignored-layers "img_mlp"
# Online serving
vllm serve <your-model> --omni --quantization fp8
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `method` | str | — | Quantization method (`"fp8"`) |
| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16 |
| `activation_scheme` | str | `"dynamic"` | `"dynamic"` (no calibration) or `"static"` |
| `weight_block_size` | list[int] \| None | `None` | Block size for block-wise weight quantization |
The available `ignored_layers` names depend on the model architecture (e.g., `to_qkv`, `to_out`, `img_mlp`, `txt_mlp`). Consult the transformer source for your target model.
## Supported Models
| Model | HF Models | Recommendation | `ignored_layers` |
|-------|-----------|---------------|------------------|
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Skip sensitive layers | `img_mlp` |
| Flux | `black-forest-labs/FLUX.1-dev` | All layers | None |
| HunyuanImage-3 | `tencent/HunyuanImage3` | All layers | None |
| HunyuanVideo-1.5 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `720p_t2v`, `480p_i2v` | All layers | None |
| GLM-Image | `zai-org/GLM-Image` | All layers | None |
| Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | All layers | None |
## Combining with Other Features
FP8 quantization can be combined with cache acceleration:
```python
omni = Omni(
model="<your-model>",
quantization="fp8",
cache_backend="tea_cache",
cache_config={"rel_l1_thresh": 0.2},
)
```
docs/user_guide/diffusion/quantization/gguf.md
-227
View File
@@ -1,227 +0,0 @@
# GGUF Quantization
## Goals
1. Reuse vLLM quantization configs and weight loaders as much as possible.
2. Add native GGUF support to diffusion transformers without changing model definitions.
3. Keep user-facing knobs minimal and consistent across offline and online flows.
## Scope
1. Models: Z-Image, Flux2-klein, and Qwen-Image family.
2. Components: diffusion transformer weights, loader paths, and quantization configs.
3. Modes: native GGUF (transformer-only weights).
## Architecture Overview
1. `OmniDiffusionConfig` accepts `quantization` or `quantization_config`.
2. Diffusion quantization wrapper (`DiffusionGGUFConfig`) produces vLLM `QuantizationConfig` objects for linear layers.
3. `DiffusersPipelineLoader` branches on quantization method and loads either HF weights or GGUF weights for the transformer.
4. GGUF transformer loading is routed through model-specific adapters (e.g., Flux2Klein).
5. vLLM GGUF path uses `GGUFConfig` and `GGUFLinearMethod` for matmul.
## Call Chain (Offline)
```
CLI (examples/offline_inference/text_to_image/text_to_image.py)
|
v
Omni (vllm_omni/entrypoints/__init__.py)
|
v
AsyncOmniEngine
|
v
Orchestrator
|
v
StageDiffusionClient
|
v
DiffusionWorker
|
v
DiffusionModelRunner
|
v
DiffusersPipelineLoader
|
v
Pipeline.forward (Flux2/Qwen/Z-Image)
|
v
DiffusionEngine
|
v
OmniRequestOutput
|
v
Client (saved PNG)
```
## Call Chain (Online)
```
Client
|
| POST /v1/images/generations
v
APIServer (vllm_omni/entrypoints/openai/api_server.py)
|
v
_generate_with_async_omni
|
v
AsyncOmni
|
v
AsyncOmniEngine
|
v
Orchestrator
|
v
StageDiffusionClient
|
v
OmniRequestOutput
|
v
encode_image_base64
|
v
ImageGenerationResponse
|
v
Client
```
## Call Chain (GGUF Operator Path)
```
Pipeline.forward (Flux2/Qwen/Z-Image)
|
v
Transformer blocks
|
v
QKVParallelLinear / ColumnParallelLinear / RowParallelLinear
|
v
LinearBase.forward
|
v
QuantMethod.apply (GGUFLinearMethod.apply)
|
v
fused_mul_mat_gguf
|
v
_fused_mul_mat_gguf (custom op)
|
v
ops.ggml_dequantize
|
v
x @ weight.T
```
## GGUF Weight Loading Path (Transformer-Only)
1. `DiffusersPipelineLoader.load_model` detects `quantization_config.method == "gguf"`.
2. `gguf_model` is resolved as one of: local file, `repo/file.gguf`, or `repo:quant_type`.
3. GGUF weights are routed through adapters in `vllm_omni/diffusion/model_loader/gguf_adapters/`.
4. Name mapping is applied per-architecture (Z-Image, Flux2Klein, Qwen-Image).
5. GGUF weights are loaded into transformer modules, remaining non-transformer weights come from the HF checkpoint.
## GGUF Adapter Design
1. `GGUFAdapter` is an abstract base class for model-specific adapters.
2. `Flux2KleinGGUFAdapter` implements Flux2-Klein remapping + qkv split + adaLN swap.
3. `ZImageGGUFAdapter` implements Z-Image qkv + ffn shard handling and linear qweight routing.
4. `QwenImageGGUFAdapter` implements Qwen-Image fused attention projection remapping.
5. `get_gguf_adapter(...)` strictly selects by model class/config; unsupported models raise an error (no fallback adapter).
Adapter paths:
- Base: `vllm_omni/diffusion/model_loader/gguf_adapters/base.py`
- Z-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/z_image.py`
- Flux2-Klein: `vllm_omni/diffusion/model_loader/gguf_adapters/flux2_klein.py`
- Qwen-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/qwen_image.py`
## User Usage (Offline)
### Baseline BF16
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model /workspace/models/black-forest-labs/FLUX.2-klein-4B \
--prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \
--height 768 \
--width 1360 \
--seed 42 \
--cfg-scale 4.0 \
--num-images-per-prompt 1 \
--num-inference-steps 4 \
--output outputs/flux2_klein_4b.png
```
### Native GGUF (Transformer Only)
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model /workspace/models/black-forest-labs/FLUX.2-klein-4B \
--gguf-model "/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf" \
--quantization gguf \
--prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \
--height 768 \
--width 1360 \
--seed 42 \
--cfg-scale 4.0 \
--num-images-per-prompt 1 \
--num-inference-steps 4 \
--output outputs/flux2_klein_4b_gguf.png
```
Notes for GGUF:
1. Many GGUF repos do not ship `model_index.json` and configs. Use the base repo for `--model` and only pass the GGUF file via `--gguf-model`.
2. `gguf_model` supports local path, `repo/file.gguf`, or `repo:quant_type`.
## User Usage (Online)
### Start Server (Native GGUF via CLI)
```bash
vllm serve /workspace/models/black-forest-labs/FLUX.2-klein-4B \
--omni \
--port 8000 \
--quantization-config '{"method":"gguf","gguf_model":"/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf"}'
```
### Online Request (Images API)
```bash
curl -X POST http://localhost:8000/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "a dragon laying over the spine of the Green Mountains of Vermont",
"size": "1024x1024",
"seed": 42,
"num_inference_steps": 4
}'
```
## Qwen-Image Example
### Offline
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model Qwen/Qwen-Image \
--gguf-model QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf \
--quantization gguf \
--prompt "a red paper kite hanging from a pine tree in a winter courtyard" \
--height 1024 \
--width 1024 \
--seed 42 \
--num_inference_steps 20 \
--output outputs/qwen_image_q4km.png
```
### Online
```bash
vllm serve Qwen/Qwen-Image \
--omni \
--port 8000 \
--quantization-config '{"method":"gguf","gguf_model":"QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf"}'
```
Notes:
1. The GGUF repo provides transformer weights only; the base model repo still provides tokenizer, text encoder, scheduler, and VAE.
2. The same adapter path covers `QwenImagePipeline`, `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, and `QwenImageLayeredPipeline`.
docs/user_guide/diffusion/quantization/int8.md
-75
View File
@@ -1,75 +0,0 @@
# Int8 Quantization
## Overview
Int8 quantization converts BF16/FP16 weights to Int8 at model load time. No calibration or pre-quantized checkpoint needed.
Depending on the model, either all layers can be quantized, or some sensitive layers should stay in BF16/FP16. See the [per-model table](#supported-models) for which case applies.
## Configuration
1. **Python API**: set `quantization="int8"`. To skip sensitive layers, use `quantization_config` with `ignored_layers`.
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# All layers quantized
omni = Omni(model="<your-model>", quantization="int8")
# Skip sensitive layers
omni = Omni(
model="<your-model>",
quantization_config={
"method": "int8",
"ignored_layers": ["<layer-name>"],
},
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
2. **CLI**: pass `--quantization int8` and optionally `--ignored-layers`.
```bash
# All layers
python text_to_image.py --model <your-model> --quantization int8
# Skip sensitive layers
python text_to_image.py --model <your-model> --quantization int8 --ignored-layers "img_mlp"
# Online serving
vllm serve <your-model> --omni --quantization int8
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `method` | str | — | Quantization method (`"int8"`) |
| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16/FP16 |
| `activation_scheme` | str | `"dynamic"` | `"dynamic"` (no calibration) |
The available `ignored_layers` names depend on the model architecture (e.g., `to_qkv`, `to_out`, `img_mlp`, `txt_mlp`). Consult the transformer source for your target model.
## Supported Models
| Model | HF Models | Recommendation | `ignored_layers` |
|-------|-----------|---------------|------------------|
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | All layers | None |
## Combining with Other Features
Int8 quantization can be combined with cache acceleration:
```python
omni = Omni(
model="<your-model>",
quantization="int8",
cache_backend="tea_cache",
cache_config={"rel_l1_thresh": 0.2},
)
```
docs/user_guide/diffusion/quantization/msmodelslim.md
-56
View File
@@ -1,56 +0,0 @@
# msModelSlim Quantization
## Overview
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.
Once you have a quantized model which is generated by **msModelSlim**, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features.
### Supported Schemes
| Scheme | Bits | Status |
|--------|------|--------|
| W8A8 | 8 | ✅ Supported |
| W4A4 | 4 | Planned |
W8A8 is the first supported scheme. Additional schemes will be added in future releases.
## Model Quantization
The following example shows how to generate W8A8 quantized weights for the [Wan2_2 model](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/Wan2_2/README.md).
**Quantization Script:**
```bash
msmodelslim quant \
--model_path /path/to/wan2_2_t2v_float_weights \
--save_path /path/to/wan2_2_t2v_quantized_weights \
--device npu \
--model_type Wan2_2 \
--config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \
--trust_remote_code True
```
After quantization completes, the output directory will contain the quantized model files.
For more examples, refer to the [official examples](https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example).
## Configuration
1. **CLI**: pass `--quantization ascend`.
```bash
# Offline inference
python text_to_image.py --model <your-model> --quantization ascend
# Online serving
vllm serve <your-model> --omni --quantization ascend
```
## Supported Models
| Model | HF Models | Recommendation | `ignored_layers` |
|-------|-----------|---------------|------------------|
| HunyuanImage-3.0 | - | All layers | None |
Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a [HunyuanImage-3.0-adapted msModelSlim version](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the quantized weights manually. We will upload the quantized weights as soon as possible.
docs/user_guide/diffusion/quantization/overview.md
-153
View File
@@ -1,153 +0,0 @@
# Quantization
vLLM-Omni provides a unified quantization framework that supports both diffusion models (DiT, text encoders, VAEs) and multi-stage omni models (e.g., Qwen3-Omni thinker). Quantization reduces memory usage and accelerates inference.
## Supported Methods
### Diffusion Models
| Method | Guide | Description | Tested Models | Min GPU |
|--------|-------|-------------|---------------|---------|
| FP8 | [FP8](fp8.md) | FP8 W8A8, dynamic or static | Z-Image, Qwen-Image, Flux, Bagel | SM 89 (Ada) |
| Int8 | [Int8](int8.md) | Int8 W8A8 | Z-Image, Qwen-Image | SM 89 (Ada) / Ascend NPU |
| GGUF | [GGUF](gguf.md) | GGUF format, dequant+GEMM for N-D tensors | Z-Image, Flux | SM 60 |
| AutoRound | [AutoRound](autoround.md) | W4A16 (pre-quantized) | Flux | SM 80 (Ampere) |
### Multi-stage Omni Models (Pre-quantized Checkpoints)
| Method | Description | Tested Models | Min GPU |
|--------|-------------|---------------|---------|
| ModelOpt FP8 | Pre-quantized FP8 via NVIDIA ModelOpt | Qwen3-Omni (thinker) | SM 89 (Ada/Hopper) |
| ModelOpt NVFP4 | Pre-quantized NVFP4 via NVIDIA ModelOpt | Qwen3-Omni (experimental, quality issues) | SM 100 (Blackwell) |
!!! note
AWQ, GPTQ, and BitsAndBytes are supported by vLLM's upstream quantization registry but have **not been tested** with vLLM-Omni pipelines. They may work via `build_quant_config()` but are not validated.
### Pre-quantized LLM Checkpoints (Multi-stage Models)
For multi-stage models like Qwen3-Omni, the unified quantization framework auto-detects
pre-quantized checkpoints via `quantization_config` in the HF config. Quantization is
automatically scoped to the thinker's `language_model` — audio encoder, vision encoder,
talker, and code2wav remain in BF16.
| Format | `quant_algo` | Hardware | Example | Status |
|--------|-------------|----------|---------|--------|
| ModelOpt FP8 | `FP8` | Ada/Hopper (SM 89+) | `asdazd/Qwen3-Omni-30B-A3B-Instruct_modelopt_FP8` | Tested — 47% memory reduction, comparable throughput |
| ModelOpt NVFP4 | `NVFP4` | Blackwell (SM 100+) | `shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4` | Experimental — loads and runs but output quality is unacceptable |
**Tested FP8 results (Qwen3-Omni, 1×H200):**
| Config | Model Memory (GiB) | Mem Reduction | Decode (tok/s) |
|--------|-------------------|---------------|----------------|
| BF16 baseline | 59.26 | — | 41.6 |
| FP8 (ModelOpt) | 31.41 | 47% | 39.9 |
FP8 enables the full 3-stage pipeline (thinker + talker + code2wav) on a single 64GB GPU,
which is impossible with BF16 (thinker alone requires 59.26 GiB).
## Quantization Scope
### Diffusion Models
When `--quantization fp8` is enabled for diffusion models:
| Component | What Gets Quantized | Mechanism |
|-----------|-------------------|-----------|
| **DiT (transformer)** | `nn.Linear` layers | vLLM W8A8 FP8 compute (Ada/Hopper) or weight-only (older GPUs) |
| **Text encoder** | `nn.Linear` layers | FP8 weight storage, BF16 compute |
| **VAE** | `nn.Conv2d`, `nn.Conv3d` layers | FP8 weight storage, BF16 compute |
### Multi-stage Omni Models
For pre-quantized omni model checkpoints, quantization is scoped per component:
| Component | Quantized? | Notes |
|-----------|-----------|-------|
| **Thinker (language_model)** | Yes | FP8 or NVFP4 via ModelOpt |
| **Audio encoder** | No | Stays BF16 |
| **Vision encoder** | No | Stays BF16 |
| **Talker** | No | Stays BF16 |
| **Code2Wav** | No | Stays BF16 |
!!! note
Not all models support all components. See the [FP8 supported models table](fp8.md#supported-models) for per-model details.
## Device Compatibility for FP8
| Device | Example Hardware | FP8 | NVFP4 |
|--------|-----------------|-----|-------|
| Blackwell GPU (SM 100+) | RTX 5090, B100, B200 | Yes | Yes (native FP4 HW) |
| Ada/Hopper GPU (SM 89+) | RTX 4090, H100, H200 | Yes (W8A8 native) | No |
| Turing/Ampere GPU (SM 75-86) | RTX 3090, A100 | Yes (weight-only Marlin) | No |
| Ascend NPU | Atlas 800T A2 (910B) | Not yet supported | No |
Kernel selection is automatic on CUDA GPUs.
## Device Compatibility for Int8
| Device Type | Generation | Example | Int8 Mode |
|-------------|---------------|-------------------|----------|
| NVIDIA GPU | Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
| Ascend NPU | Atlas A2/Atlas A3 | Atlas 800T A2/Atlas 900 A3 | Full W8A8 with native hardware |
## Python API
The `build_quant_config()` factory accepts multiple input formats:
```python
from vllm_omni.quantization import build_quant_config
# String
config = build_quant_config("fp8")
# Dict with parameters
config = build_quant_config({"method": "fp8", "activation_scheme": "static"})
# AutoRound / INC (auto-detected from checkpoint, or explicit)
config = build_quant_config("auto-round", bits=4, group_size=128)
# Per-component dict
config = build_quant_config({
"transformer": {"method": "fp8"},
"vae": None,
})
# None / "none"
config = build_quant_config(None) # returns None
```
## Architecture
```
build_quant_config(spec)
├── str → _build_single(method) → vLLM registry / _OVERRIDES
├── dict with "method" → _build_single(method, **kwargs)
├── per-component dict → ComponentQuantizationConfig
│ routes get_quant_method() by longest-prefix match
├── QuantizationConfig → passthrough
└── None / "none" → None
```
## Migration Guide
### Before (v0.14.x)
```python
# Old diffusion-specific API (removed)
from vllm_omni.diffusion.quantization import get_diffusion_quant_config
config = get_diffusion_quant_config("fp8", activation_scheme="static")
```
### After (v0.16.0+)
```python
# Unified API — delegates to vLLM's registry
from vllm_omni.quantization import build_quant_config
config = build_quant_config({"method": "fp8", "activation_scheme": "static"})
# Per-component (new)
config = build_quant_config({
"transformer": {"method": "fp8"},
"vae": None,
})
```