diff --git a/docs/.nav.yml b/docs/.nav.yml index ddcef5d7a..099e65d19 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -62,11 +62,6 @@ nav: - Cache Acceleration: - TeaCache: user_guide/diffusion/cache_acceleration/teacache.md - Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md - - Quantization: - - Overview: user_guide/diffusion/quantization/overview.md - - FP8: user_guide/diffusion/quantization/fp8.md - - Int8: user_guide/diffusion/quantization/int8.md - - GGUF: user_guide/diffusion/quantization/gguf.md - Attention Backends: user_guide/diffusion/attention_backends.md - Frame Interpolation: user_guide/diffusion/frame_interpolation.md - Parallelism: @@ -81,6 +76,14 @@ nav: - LoRA: user_guide/diffusion/lora.md - Custom Pipeline: features/custom_pipeline.md - Step Execution: user_guide/diffusion/step_execution.md + - Quantization: + - Overview: user_guide/quantization/overview.md + - Online Quantization: user_guide/quantization/online.md + - FP8 W8A8: user_guide/quantization/fp8.md + - Int8 W8A8: user_guide/quantization/int8.md + - GGUF: user_guide/quantization/gguf.md + - AutoRound: user_guide/quantization/autoround.md + - msModelSlim: user_guide/quantization/msmodelslim.md - ComfyUI: features/comfyui.md - Developer Guide: - General: @@ -92,7 +95,6 @@ nav: - contributing/model/adding_omni_model.md - contributing/model/adding_tts_model.md - contributing/model/adding_diffusion_model.md - - contributing/model/adding_quantization_model.md - CI: contributing/ci - Design Documents: - design/index.md diff --git a/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md b/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md index e1309b1ad..688933f13 100644 --- a/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md +++ b/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md @@ -1,6 +1,6 @@ **Scope** -For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/quantization/overview/). +For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/). **Test Case Design** diff --git a/docs/contributing/model/adding_quantization_model.md b/docs/contributing/model/adding_quantization_model.md deleted file mode 100644 index f27318888..000000000 --- a/docs/contributing/model/adding_quantization_model.md +++ /dev/null @@ -1,478 +0,0 @@ -# Support FP8 Quantization - -This section describes how to add FP8 quantization support to a diffusion model. We use the Qwen-Image pipeline as the reference implementation. - ---- - -## Table of Contents - -- [Overview](#overview) -- [Step-by-Step Implementation](#step-by-step-implementation) -- [Testing](#testing) -- [Troubleshooting](#troubleshooting) -- [Reference Implementations](#reference-implementations) -- [Summary](#summary) - ---- - -## Overview - -### What is FP8 Quantization? - -FP8 quantization reduces model memory footprint by storing weights in 8-bit floating point (`float8_e4m3fn`) instead of 16-bit (`bfloat16`/`float16`). This saves ~50% of weight memory. - -A diffusion pipeline typically has three quantizable components: - -| Component | Layer Types | Quantization Approach | -|-----------|-----------|----------------------| -| **DiT (transformer)** | `nn.Linear` | Replace with vLLM quantized linear layers | -| **Text encoder** | `nn.Linear` | FP8 weight storage via hooks | -| **VAE** | `nn.Conv2d`, `nn.Conv3d` | FP8 weight storage via hooks | - -### Architecture - -The quantization system has two mechanisms: - -**1. vLLM Quantized Linear Layers (for DiT)** - -vLLM provides drop-in replacements for `nn.Linear` that perform FP8 W8A8 computation on supported hardware. These are configured via `quant_config` passed to layer constructors. - -Key classes: - -| Class | Location | Purpose | -|-------|----------|---------| -| `build_quant_config` | `vllm_omni/quantization/__init__.py` | Build quantization config from string/dict/per-component spec | -| `ComponentQuantizationConfig` | `vllm_omni/quantization/component_config.py` | Per-component routing by layer prefix | - -**2. FP8 Weight Storage via Hooks (for text encoder and VAE)** - -For models loaded via `from_pretrained()` (outside vLLM's weight pipeline), we use forward hooks to store weights in FP8 and dequantize on-the-fly: - -``` -At rest: weight.data = fp8_tensor (half memory) -Pre-hook: weight.data = fp8_weight.to(bf16) * scale -Forward: Normal computation in BF16 -Post-hook: weight.data = fp8_weight (back to FP8) -``` - -Key function: - -| Function | Location | Purpose | -|----------|----------|---------| -| `apply_fp8_weight_storage` | `vllm_omni/diffusion/models/utils.py` | Apply FP8 hooks to Linear/Conv layers (introduced in [#1412](https://github.com/vllm-project/vllm-omni/pull/1412)) | - ---- - -## Step-by-Step Implementation - -### Step 1: Add `quant_config` to the Transformer - -Pass `quant_config` to vLLM parallel linear layers in the transformer. This enables FP8 W8A8 for DiT linear layers. - -#### 1.1: Accept `quant_config` in Constructor - -```python -from vllm.model_executor.layers.quantization.base_config import QuantizationConfig - -class YourTransformer2DModel(nn.Module): - def __init__( - self, - *, - od_config: OmniDiffusionConfig, - quant_config: QuantizationConfig | None = None, - # ... other params - ): - self.quant_config = quant_config - # Pass to sub-modules - self.blocks = nn.ModuleList([ - YourTransformerBlock(quant_config=quant_config, ...) - for _ in range(num_layers) - ]) -``` - -#### 1.2: Pass `quant_config` to vLLM Linear Layers - -Replace `nn.Linear` with vLLM parallel linear layers that accept `quant_config`: - -```python -from vllm.model_executor.layers.linear import ( - ColumnParallelLinear, - QKVParallelLinear, - RowParallelLinear, -) - -class YourAttentionBlock(nn.Module): - def __init__(self, hidden_size, num_heads, quant_config=None, prefix=""): - # QKV projection - self.to_qkv = QKVParallelLinear( - hidden_size=hidden_size, - head_size=hidden_size // num_heads, - total_num_heads=num_heads, - quant_config=quant_config, - prefix=f"{prefix}.to_qkv", - ) - # Output projection - self.to_out = RowParallelLinear( - input_size=hidden_size, - output_size=hidden_size, - quant_config=quant_config, - prefix=f"{prefix}.to_out", - ) - -class YourFeedForward(nn.Module): - def __init__(self, hidden_size, intermediate_size, quant_config=None, prefix=""): - self.gate_up_proj = ColumnParallelLinear( - input_size=hidden_size, - output_size=intermediate_size * 2, - quant_config=quant_config, - prefix=f"{prefix}.gate_up_proj", - ) - self.down_proj = RowParallelLinear( - input_size=intermediate_size, - output_size=hidden_size, - quant_config=quant_config, - prefix=f"{prefix}.down_proj", - ) -``` - -!!! important - Always pass the `prefix` parameter. It is used by `ignored_layers` matching to skip sensitive layers. - -#### 1.3: Identify Sensitive Layers - -Some layers degrade output quality when quantized. Common sensitive layers: - -| Layer | Why Sensitive | Typical Models | Quant Methods Affected | -|-------|-------------|---------------|----------------------| -| `img_mlp` | Processes denoising latents with shifting dynamic range | Qwen-Image | FP8, Int8 | -| `feed_forward` | FFN layers in DiT blocks, large dynamic range | Z-Image | Int8 | -| `proj_out` | Final output projection, small quantization errors amplified | Various | FP8, Int8 | -| `lm_head` | Output vocabulary projection, precision-critical | Omni (thinker) | NVFP4 | -| `mlp.gate` | MoE router gates, routing decisions are precision-critical | MoE models | NVFP4 | - -**Quantization quality benchmarks** (LPIPS: lower = better, <0.01 imperceptible, >0.1 noticeable): - -| Model | Method | All Layers | With `ignored_layers` | Recommendation | -|-------|--------|-----------|----------------------|----------------| -| Qwen-Image-2512 | Int8 | LPIPS 0.0197 | 0.0027 (skip `img_mlp`) | Skip `img_mlp` | -| Z-Image-Turbo | Int8 | LPIPS 0.1597 | 0.0290 (skip `feed_forward`) | Skip `feed_forward` | -| Z-Image-Turbo | FP8 | LPIPS ~0.005 | — | All layers OK | - -To identify sensitive layers for a new model: - -1. Run inference with all layers quantized -2. Compare output quality with BF16 baseline using LPIPS (see `benchmarks/diffusion/quantization_quality.py`) -3. Selectively disable quantization per layer (via `ignored_layers`) until quality matches -4. Document the recommended `ignored_layers` in the model's supported models table - -### Step 2: Wire Up Quantization in the Pipeline - -#### 2.1: Create `quant_config` and Pass to Transformer - -In the pipeline's `__init__`, extract the vLLM quant config and pass it to the transformer: - -```python -from vllm_omni.quantization import build_quant_config - -class YourPipeline(nn.Module): - def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = ""): - super().__init__() - # ... load scheduler, text_encoder, vae ... - - # Build quant config from OmniDiffusionConfig - quant_config = build_quant_config(od_config.quantization_config) - - # Pass to transformer - self.transformer = YourTransformer2DModel( - od_config=od_config, - quant_config=quant_config, - # ... other kwargs - ) -``` - -#### 2.2: Apply FP8 Weight Storage to Text Encoder and VAE - -For models loaded via `from_pretrained()`, apply hook-based FP8 weight storage. - -!!! note - The `apply_fp8_weight_storage` function is provided by `vllm_omni/diffusion/models/utils.py`, introduced in [PR #1412](https://github.com/vllm-project/vllm-omni/pull/1412). - -```python -from vllm_omni.diffusion.models.utils import apply_fp8_weight_storage - -class YourPipeline(nn.Module): - def __init__(self, *, od_config: OmniDiffusionConfig, prefix: str = ""): - super().__init__() - - self.text_encoder = SomeTextEncoder.from_pretrained(...) - self.vae = SomeVAE.from_pretrained(...).to(self.device) - - # Apply FP8 weight storage to text encoder and VAE - if ( - od_config.quantization_config is not None - and getattr(od_config.quantization_config, "quant_method", None) == "fp8" - ): - apply_fp8_weight_storage(self.vae) - apply_fp8_weight_storage(self.text_encoder) - - # ... rest of init -``` - -`apply_fp8_weight_storage` walks all `nn.Linear`, `nn.Conv2d`, and `nn.Conv3d` modules in the model and applies FP8 hooks to each. - -#### 2.3: Fix `load_weights` to Include Pre-loaded Components - -Since text encoders and VAEs are loaded via `from_pretrained()` (not through vLLM's weight pipeline), their parameters won't appear in the `loaded_weights` set. Mark them explicitly: - -```python -def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]: - loader = AutoWeightsLoader(self) - loaded_weights = loader.load_weights(weights) - # Mark pre-loaded component weights as loaded - loaded_weights |= {f"vae.{name}" for name, _ in self.vae.named_parameters()} - loaded_weights |= {f"text_encoder.{name}" for name, _ in self.text_encoder.named_parameters()} - return loaded_weights -``` - -### Step 3: Add to Supported Models Documentation - -Update the FP8 supported models table in `docs/user_guide/diffusion/quantization/fp8.md`: - -```markdown -| Your-Model | `org/Your-Model` | ✅ | ✅ | ✅ | `sensitive_layer` | -``` - ---- - -## Testing - -### Quick Verification - -```python -from vllm_omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -# Test with FP8 enabled -omni = Omni(model="your-model", quantization="fp8") - -outputs = omni.generate( - "A cat sitting on a windowsill", - OmniDiffusionSamplingParams(num_inference_steps=50), -) -``` - -### Quality Gate Test (LPIPS) - -We provide a pytest-based quality gate at `tests/diffusion/quantization/test_quantization_quality.py`. -It generates outputs with both BF16 and your quantized method using the same seed, computes -[LPIPS](https://github.com/richzhang/PerceptualSimilarity) perceptual distance, and **fails if it -exceeds a threshold**. This is the recommended way to validate that a quantization method does not -unacceptably degrade output quality. - -**Step 1: Add your test config** — edit `QUALITY_CONFIGS` in the test file: - -```python -QualityTestConfig( - id="int8_z_image", # pytest ID - model="Tongyi-MAI/Z-Image-Turbo", - quantization="int8", # your quantization method - task="t2i", # "t2i" or "t2v" - prompt="a cup of coffee on a wooden table, morning light", - max_lpips=0.15, # fail threshold (see guide below) - num_inference_steps=20, -), -``` - -**Step 2: Run the test:** - -```bash -# Install dependency -pip install lpips - -# Run all quality tests -pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" - -# Run only your method -pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "int8" - -# Run a specific model -pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "z_image" -``` - -**Step 3: Read the output:** - -``` -Quantization Quality: int8_z_image - Model: Tongyi-MAI/Z-Image-Turbo - Method: int8 - LPIPS: 0.0290 (threshold: 0.15) - BF16 memory: 19.15 GiB - Quant memory: 14.80 GiB (23% reduction) - Result: PASS -``` - -**LPIPS threshold guide:** - -| LPIPS Range | Meaning | Recommendation | -|-------------|---------|----------------| -| < 0.01 | Imperceptible | Ideal — no visible difference | -| 0.01 – 0.05 | Minor differences | Good — acceptable for most use cases | -| 0.05 – 0.15 | Noticeable on close inspection | Acceptable with `ignored_layers` tuning | -| 0.15 – 0.30 | Clearly visible | Investigate — likely needs `ignored_layers` | -| > 0.30 | Significant degradation | Quantization method may not suit this model | - -!!! tip - If your test fails, don't raise the threshold — instead, identify sensitive layers and add - them to `ignored_layers` (see next section). Then re-run the test with the lower LPIPS. - -### Quality Benchmark (Detailed) - -For a full multi-prompt benchmark with Markdown output (suitable for PR descriptions), use the -standalone benchmark script: - -```bash -python benchmarks/diffusion/quantization_quality.py \ - --model Tongyi-MAI/Z-Image-Turbo \ - --quantization fp8 \ - --prompts \ - "a cup of coffee on the table" \ - "an aerial view of a coral reef" \ - --height 1024 --width 1024 \ - --num-inference-steps 50 --seed 42 -``` - -This generates BF16 and quantized outputs, computes per-prompt LPIPS, and prints a Markdown -summary table ready to paste into your PR. - -### Identifying Sensitive Layers with `ignored_layers` - -Not all layers tolerate quantization equally. **`ignored_layers`** lets you keep specific layers -in BF16 while quantizing everything else. This is critical for achieving good quality with -aggressive quantization methods (Int8, NVFP4). - -**Why some layers are sensitive:** - -| Layer Type | Why Sensitive | Typical Impact | -|------------|--------------|----------------| -| `img_mlp` | Processes denoising latents with shifting dynamic range across timesteps. No built-in normalization to absorb quantization error. | Color shifts, blurriness | -| `feed_forward` | FFN layers with large dynamic range in DiT blocks. | Artifacts, loss of detail | -| `proj_out` | Final output projection — small errors get amplified since there are no subsequent layers to correct them. | Overall quality loss | -| `lm_head` | Vocabulary projection in LLMs — precision-critical for token selection. | Garbage text output | -| `mlp.gate` | MoE router gates — routing decisions are binary and precision-critical. | Wrong expert selection, quality collapse | - -**How to find sensitive layers for your model:** - -1. Run the quality gate test with all layers quantized: - ```bash - pytest tests/diffusion/quantization/test_quantization_quality.py -v -m "" -k "your_model" - ``` - -2. If LPIPS exceeds your threshold, try skipping common sensitive layers one at a time: - ```python - QualityTestConfig( - id="fp8_your_model_skip_mlp", - model="your-org/Your-Model", - quantization="fp8", - task="t2i", - prompt="a cup of coffee on a wooden table", - max_lpips=0.05, - ), - ``` - - And in the test, modify the Omni call: - ```python - omni_qt = Omni( - model=config.model, - quantization_config={ - "method": config.quantization, - "ignored_layers": ["img_mlp"], - }, - ) - ``` - -3. Compare LPIPS with and without `ignored_layers` to isolate the problematic layers. - -4. Document the recommended `ignored_layers` in the supported models table. - -**Real-world examples:** - -| Model | Method | All Layers (LPIPS) | With `ignored_layers` | Which Layers to Skip | -|-------|--------|-------------------|----------------------|---------------------| -| Qwen-Image-2512 | Int8 | 0.0197 | 0.0027 | `img_mlp` | -| Z-Image-Turbo | Int8 | 0.1597 | 0.0290 | `feed_forward` | -| Z-Image-Turbo | FP8 | ~0.005 | — | None needed | - -### What to Verify (Checklist) - -1. **Model loads without errors** — Check logs for quantization initialization messages -2. **Quality gate passes** — LPIPS within threshold vs BF16 baseline -3. **Memory reduction** — Expect ~30-50% savings on quantized components -4. **Sensitive layers identified** — Document which layers (if any) need `ignored_layers` -5. **`ignored_layers` pattern works** — Verify that the layer name prefix matches your transformer's module names - ---- - -## Troubleshooting - -### Issue: Output Quality Degradation - -**Symptoms:** Generated images have visible artifacts, color shifts, or blurriness compared to BF16 baseline. - -**Causes & Solutions:** - -- **Sensitive layers being quantized:** - - **Solution:** Identify and skip sensitive layers: - ```python - omni = Omni( - model="your-model", - quantization_config={ - "method": "fp8", - "ignored_layers": ["img_mlp"], - }, - ) - ``` - -- **All layers sensitive (rare):** - - **Solution:** The model may not be suitable for FP8 quantization. Consider weight-only quantization or other methods. - -### Issue: Missing Weights Warning - -**Symptoms:** Warnings about unloaded weights for `vae.*` or `text_encoder.*` parameters. - -**Solution:** Ensure `load_weights` marks pre-loaded component weights. See [Step 2.3](#23-fix-load_weights-to-include-pre-loaded-components). - -### Issue: CUDA Out of Memory - -**Symptoms:** OOM errors during model loading despite FP8 being enabled. - -**Causes & Solutions:** - -- **Text encoder/VAE loaded in BF16 before FP8 conversion:** - - FP8 weight storage is applied after `from_pretrained()`, so peak load memory is still BF16. This is a one-time cost during initialization. - ---- - -## Reference Implementations - -| Model | Pipeline File | Transformer File | Notes | -|-------|-------------|-----------------|-------| -| **Z-Image** | `models/z_image/pipeline_z_image.py` | `models/z_image/z_image_transformer.py` | DiT + text encoder FP8 | -| **Qwen-Image** | `models/qwen_image/pipeline_qwen_image.py` | `models/qwen_image/qwen_image_transformer.py` | DiT + text encoder + VAE FP8 | - -All files are under `vllm_omni/diffusion/`. - ---- - -## Summary - -Adding FP8 quantization support to a new model: - -1. ✅ **Add `quant_config` to transformer** — Accept and pass to vLLM parallel linear layers with proper `prefix` -2. ✅ **Identify sensitive layers** — Profile which layers need `ignored_layers` for quality -3. ✅ **Apply FP8 weight storage** — Call `apply_fp8_weight_storage()` on text encoder and VAE in pipeline `__init__` -4. ✅ **Fix `load_weights`** — Mark pre-loaded component weights as loaded -5. ✅ **Update docs** — Add model to the supported models table in `docs/user_guide/diffusion/quantization/fp8.md` -6. ✅ **Test** — Verify quality, memory savings, and sensitive layer behavior diff --git a/docs/user_guide/diffusion/quantization/autoround.md b/docs/user_guide/diffusion/quantization/autoround.md deleted file mode 100644 index d06627d40..000000000 --- a/docs/user_guide/diffusion/quantization/autoround.md +++ /dev/null @@ -1,93 +0,0 @@ -# AutoRound Quantization - -## Overview - -[AutoRound](https://github.com/intel/auto-round) is an advanced quantization toolkit designed for Large Language Models (LLMs), Vision-Language Models (VLMs), and diffusion models. It achieves high accuracy at ultra-low bit widths (2–4 bits) with minimal tuning by leveraging sign-gradient descent, while providing broad hardware compatibility with multi-datatype support. - -The quantization config is auto-detected from the checkpoint's `config.json` (`quantization_config.quant_method = "auto-round"`). No extra CLI flags are needed. - -### Supported Schemes - -| Scheme | Bits | Status | -|--------|------|--------| -| W4A16 | 4 | ✅ Supported | -| W8A16 | 8 | Planned | - -W4A16 is the first supported scheme. Additional schemes will be added in future releases. - -## Configuration - -1. **Python API**: point `model` at a pre-quantized checkpoint. The quantization is detected automatically. - -```python -from vllm_omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16") - -outputs = omni.generate( - "A cat sitting on a windowsill", - OmniDiffusionSamplingParams(num_inference_steps=28), -) -outputs[0].save_images("output.png") -``` - -2. **CLI**: pass the quantized model path directly. - -```bash -python examples/offline_inference/text_to_image/text_to_image.py \ - --model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \ - --prompt "A cat sitting on a windowsill" \ - --num-inference-steps 28 \ - --output outputs/flux_w4a16.png -``` - -No `--quantization` flag is needed — the quantization method is read from the checkpoint. - -## How It Works - -The checkpoint's `config.json` contains: - -```json -{ - "quantization_config": { - "quant_method": "auto-round", - "bits": 4, - "group_size": 128, - "sym": true, - "packing_format": "auto_round:auto_gptq", - "block_name_to_quantize": "transformer_blocks,single_transformer_blocks" - } -} -``` - -At load time: - -1. `TransformerConfig.from_dict()` parses the `quantization_config` section and builds a vLLM `INCConfig` via `build_quant_config("auto-round", ...)`. -2. `OmniDiffusionConfig.set_tf_model_config()` propagates the detected config to the engine. -3. The appropriate compute kernel (e.g. GPTQ-Marlin for W4A16) is selected automatically based on the checkpoint's bit-width and packing format. - -## Supported Models - -| Model | HF Checkpoint | Scheme | Group Size | Backend | -|-------|--------------|--------|------------|---------| -| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | W4A16 | 128 | GPTQ-Marlin | -| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | W4A16 | 128 | GPTQ-Marlin | -| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | W4A16 | 128 | GPTQ-Marlin | - -## Creating a Quantized Checkpoint - -Use [AutoRound](https://github.com/intel/auto-round) to quantize a BF16 model. The `--scheme` flag selects the quantization scheme: - -```bash -# W4A16 (4-bit weight, 16-bit activation) -auto-round \ - --model black-forest-labs/FLUX.1-dev \ - --scheme W4A16 \ - --batch_size 1 \ - --disable_opt_rtn \ - --dataset coco2014 \ - --iters 0 -``` - -The output directory can be used directly as the `model` argument. See the [AutoRound documentation](https://github.com/intel/auto-round) for all available schemes and options. diff --git a/docs/user_guide/diffusion/quantization/fp8.md b/docs/user_guide/diffusion/quantization/fp8.md deleted file mode 100644 index ceb3d006c..000000000 --- a/docs/user_guide/diffusion/quantization/fp8.md +++ /dev/null @@ -1,82 +0,0 @@ -# FP8 Quantization - -## Overview - -FP8 quantization converts BF16/FP16 weights to FP8 at model load time. No calibration or pre-quantized checkpoint needed. - -Depending on the model, either all layers can be quantized, or some sensitive layers should stay in BF16. See the [per-model table](#supported-models) for which case applies. - -Common sensitive layers in DiT-based diffusion models include **image-stream MLPs** (`img_mlp`). These are particularly vulnerable to FP8 precision loss because they process denoising latents whose dynamic range shifts significantly across timesteps, and unlike attention projections (which benefit from QK-Norm stabilization), MLPs have no built-in normalization to absorb quantization error. In deep architectures (e.g., 60+ residual blocks), small per-layer errors compound and degrade output quality. Other layers such as **attention projections** (`to_qkv`, `to_out`) and **text-stream MLPs** (`txt_mlp`) are generally more robust due to normalization or more stable input statistics. - -## Configuration - -1. **Python API**: set `quantization="fp8"`. To skip sensitive layers, use `quantization_config` with `ignored_layers`. - -```python -from vllm_omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -# All layers quantized -omni = Omni(model="", quantization="fp8") - -# Skip sensitive layers -omni = Omni( - model="", - quantization_config={ - "method": "fp8", - "ignored_layers": [""], - }, -) - -outputs = omni.generate( - "A cat sitting on a windowsill", - OmniDiffusionSamplingParams(num_inference_steps=50), -) -``` - -2. **CLI**: pass `--quantization fp8` and optionally `--ignored-layers`. - -```bash -# All layers -python text_to_image.py --model --quantization fp8 - -# Skip sensitive layers -python text_to_image.py --model --quantization fp8 --ignored-layers "img_mlp" - -# Online serving -vllm serve --omni --quantization fp8 -``` - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `method` | str | — | Quantization method (`"fp8"`) | -| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16 | -| `activation_scheme` | str | `"dynamic"` | `"dynamic"` (no calibration) or `"static"` | -| `weight_block_size` | list[int] \| None | `None` | Block size for block-wise weight quantization | - -The available `ignored_layers` names depend on the model architecture (e.g., `to_qkv`, `to_out`, `img_mlp`, `txt_mlp`). Consult the transformer source for your target model. - -## Supported Models - -| Model | HF Models | Recommendation | `ignored_layers` | -|-------|-----------|---------------|------------------| -| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None | -| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Skip sensitive layers | `img_mlp` | -| Flux | `black-forest-labs/FLUX.1-dev` | All layers | None | -| HunyuanImage-3 | `tencent/HunyuanImage3` | All layers | None | -| HunyuanVideo-1.5 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `720p_t2v`, `480p_i2v` | All layers | None | -| GLM-Image | `zai-org/GLM-Image` | All layers | None | -| Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | All layers | None | - -## Combining with Other Features - -FP8 quantization can be combined with cache acceleration: - -```python -omni = Omni( - model="", - quantization="fp8", - cache_backend="tea_cache", - cache_config={"rel_l1_thresh": 0.2}, -) -``` diff --git a/docs/user_guide/diffusion/quantization/gguf.md b/docs/user_guide/diffusion/quantization/gguf.md deleted file mode 100644 index f3bb17b85..000000000 --- a/docs/user_guide/diffusion/quantization/gguf.md +++ /dev/null @@ -1,227 +0,0 @@ -# GGUF Quantization - -## Goals -1. Reuse vLLM quantization configs and weight loaders as much as possible. -2. Add native GGUF support to diffusion transformers without changing model definitions. -3. Keep user-facing knobs minimal and consistent across offline and online flows. - -## Scope -1. Models: Z-Image, Flux2-klein, and Qwen-Image family. -2. Components: diffusion transformer weights, loader paths, and quantization configs. -3. Modes: native GGUF (transformer-only weights). - -## Architecture Overview -1. `OmniDiffusionConfig` accepts `quantization` or `quantization_config`. -2. Diffusion quantization wrapper (`DiffusionGGUFConfig`) produces vLLM `QuantizationConfig` objects for linear layers. -3. `DiffusersPipelineLoader` branches on quantization method and loads either HF weights or GGUF weights for the transformer. -4. GGUF transformer loading is routed through model-specific adapters (e.g., Flux2Klein). -5. vLLM GGUF path uses `GGUFConfig` and `GGUFLinearMethod` for matmul. - -## Call Chain (Offline) -``` -CLI (examples/offline_inference/text_to_image/text_to_image.py) - | - v -Omni (vllm_omni/entrypoints/__init__.py) - | - v -AsyncOmniEngine - | - v -Orchestrator - | - v -StageDiffusionClient - | - v -DiffusionWorker - | - v -DiffusionModelRunner - | - v -DiffusersPipelineLoader - | - v -Pipeline.forward (Flux2/Qwen/Z-Image) - | - v -DiffusionEngine - | - v -OmniRequestOutput - | - v -Client (saved PNG) -``` - -## Call Chain (Online) -``` -Client - | - | POST /v1/images/generations - v -APIServer (vllm_omni/entrypoints/openai/api_server.py) - | - v -_generate_with_async_omni - | - v -AsyncOmni - | - v -AsyncOmniEngine - | - v -Orchestrator - | - v -StageDiffusionClient - | - v -OmniRequestOutput - | - v -encode_image_base64 - | - v -ImageGenerationResponse - | - v -Client -``` - -## Call Chain (GGUF Operator Path) -``` -Pipeline.forward (Flux2/Qwen/Z-Image) - | - v -Transformer blocks - | - v -QKVParallelLinear / ColumnParallelLinear / RowParallelLinear - | - v -LinearBase.forward - | - v -QuantMethod.apply (GGUFLinearMethod.apply) - | - v -fused_mul_mat_gguf - | - v -_fused_mul_mat_gguf (custom op) - | - v -ops.ggml_dequantize - | - v -x @ weight.T -``` - -## GGUF Weight Loading Path (Transformer-Only) -1. `DiffusersPipelineLoader.load_model` detects `quantization_config.method == "gguf"`. -2. `gguf_model` is resolved as one of: local file, `repo/file.gguf`, or `repo:quant_type`. -3. GGUF weights are routed through adapters in `vllm_omni/diffusion/model_loader/gguf_adapters/`. -4. Name mapping is applied per-architecture (Z-Image, Flux2Klein, Qwen-Image). -5. GGUF weights are loaded into transformer modules, remaining non-transformer weights come from the HF checkpoint. - -## GGUF Adapter Design -1. `GGUFAdapter` is an abstract base class for model-specific adapters. -2. `Flux2KleinGGUFAdapter` implements Flux2-Klein remapping + qkv split + adaLN swap. -3. `ZImageGGUFAdapter` implements Z-Image qkv + ffn shard handling and linear qweight routing. -4. `QwenImageGGUFAdapter` implements Qwen-Image fused attention projection remapping. -5. `get_gguf_adapter(...)` strictly selects by model class/config; unsupported models raise an error (no fallback adapter). - -Adapter paths: -- Base: `vllm_omni/diffusion/model_loader/gguf_adapters/base.py` -- Z-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/z_image.py` -- Flux2-Klein: `vllm_omni/diffusion/model_loader/gguf_adapters/flux2_klein.py` -- Qwen-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/qwen_image.py` - -## User Usage (Offline) - -### Baseline BF16 -```bash -python examples/offline_inference/text_to_image/text_to_image.py \ - --model /workspace/models/black-forest-labs/FLUX.2-klein-4B \ - --prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \ - --height 768 \ - --width 1360 \ - --seed 42 \ - --cfg-scale 4.0 \ - --num-images-per-prompt 1 \ - --num-inference-steps 4 \ - --output outputs/flux2_klein_4b.png -``` - -### Native GGUF (Transformer Only) -```bash -python examples/offline_inference/text_to_image/text_to_image.py \ - --model /workspace/models/black-forest-labs/FLUX.2-klein-4B \ - --gguf-model "/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf" \ - --quantization gguf \ - --prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \ - --height 768 \ - --width 1360 \ - --seed 42 \ - --cfg-scale 4.0 \ - --num-images-per-prompt 1 \ - --num-inference-steps 4 \ - --output outputs/flux2_klein_4b_gguf.png -``` - -Notes for GGUF: -1. Many GGUF repos do not ship `model_index.json` and configs. Use the base repo for `--model` and only pass the GGUF file via `--gguf-model`. -2. `gguf_model` supports local path, `repo/file.gguf`, or `repo:quant_type`. - -## User Usage (Online) - -### Start Server (Native GGUF via CLI) -```bash -vllm serve /workspace/models/black-forest-labs/FLUX.2-klein-4B \ - --omni \ - --port 8000 \ - --quantization-config '{"method":"gguf","gguf_model":"/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf"}' -``` - -### Online Request (Images API) -```bash -curl -X POST http://localhost:8000/v1/images/generations \ - -H "Content-Type: application/json" \ - -d '{ - "prompt": "a dragon laying over the spine of the Green Mountains of Vermont", - "size": "1024x1024", - "seed": 42, - "num_inference_steps": 4 - }' -``` - -## Qwen-Image Example - -### Offline -```bash -python examples/offline_inference/text_to_image/text_to_image.py \ - --model Qwen/Qwen-Image \ - --gguf-model QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf \ - --quantization gguf \ - --prompt "a red paper kite hanging from a pine tree in a winter courtyard" \ - --height 1024 \ - --width 1024 \ - --seed 42 \ - --num_inference_steps 20 \ - --output outputs/qwen_image_q4km.png -``` - -### Online -```bash -vllm serve Qwen/Qwen-Image \ - --omni \ - --port 8000 \ - --quantization-config '{"method":"gguf","gguf_model":"QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf"}' -``` - -Notes: -1. The GGUF repo provides transformer weights only; the base model repo still provides tokenizer, text encoder, scheduler, and VAE. -2. The same adapter path covers `QwenImagePipeline`, `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, and `QwenImageLayeredPipeline`. diff --git a/docs/user_guide/diffusion/quantization/int8.md b/docs/user_guide/diffusion/quantization/int8.md deleted file mode 100644 index 1e7853c3e..000000000 --- a/docs/user_guide/diffusion/quantization/int8.md +++ /dev/null @@ -1,75 +0,0 @@ -# Int8 Quantization - -## Overview - -Int8 quantization converts BF16/FP16 weights to Int8 at model load time. No calibration or pre-quantized checkpoint needed. - -Depending on the model, either all layers can be quantized, or some sensitive layers should stay in BF16/FP16. See the [per-model table](#supported-models) for which case applies. - -## Configuration - -1. **Python API**: set `quantization="int8"`. To skip sensitive layers, use `quantization_config` with `ignored_layers`. - -```python -from vllm_omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -# All layers quantized -omni = Omni(model="", quantization="int8") - -# Skip sensitive layers -omni = Omni( - model="", - quantization_config={ - "method": "int8", - "ignored_layers": [""], - }, -) - -outputs = omni.generate( - "A cat sitting on a windowsill", - OmniDiffusionSamplingParams(num_inference_steps=50), -) -``` - -2. **CLI**: pass `--quantization int8` and optionally `--ignored-layers`. - -```bash -# All layers -python text_to_image.py --model --quantization int8 - -# Skip sensitive layers -python text_to_image.py --model --quantization int8 --ignored-layers "img_mlp" - -# Online serving -vllm serve --omni --quantization int8 -``` - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `method` | str | — | Quantization method (`"int8"`) | -| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16/FP16 | -| `activation_scheme` | str | `"dynamic"` | `"dynamic"` (no calibration) | - - -The available `ignored_layers` names depend on the model architecture (e.g., `to_qkv`, `to_out`, `img_mlp`, `txt_mlp`). Consult the transformer source for your target model. - -## Supported Models - -| Model | HF Models | Recommendation | `ignored_layers` | -|-------|-----------|---------------|------------------| -| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None | -| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | All layers | None | - -## Combining with Other Features - -Int8 quantization can be combined with cache acceleration: - -```python -omni = Omni( - model="", - quantization="int8", - cache_backend="tea_cache", - cache_config={"rel_l1_thresh": 0.2}, -) -``` diff --git a/docs/user_guide/diffusion/quantization/msmodelslim.md b/docs/user_guide/diffusion/quantization/msmodelslim.md deleted file mode 100644 index 5492cd927..000000000 --- a/docs/user_guide/diffusion/quantization/msmodelslim.md +++ /dev/null @@ -1,56 +0,0 @@ -# msModelSlim Quantization - -## Overview - -[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc. - -Once you have a quantized model which is generated by **msModelSlim**, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features. - -### Supported Schemes - -| Scheme | Bits | Status | -|--------|------|--------| -| W8A8 | 8 | ✅ Supported | -| W4A4 | 4 | Planned | - -W8A8 is the first supported scheme. Additional schemes will be added in future releases. - -## Model Quantization - -The following example shows how to generate W8A8 quantized weights for the [Wan2_2 model](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/Wan2_2/README.md). - -**Quantization Script:** - -```bash -msmodelslim quant \ - --model_path /path/to/wan2_2_t2v_float_weights \ - --save_path /path/to/wan2_2_t2v_quantized_weights \ - --device npu \ - --model_type Wan2_2 \ - --config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \ - --trust_remote_code True -``` - -After quantization completes, the output directory will contain the quantized model files. - -For more examples, refer to the [official examples](https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example). - -## Configuration - -1. **CLI**: pass `--quantization ascend`. - -```bash -# Offline inference -python text_to_image.py --model --quantization ascend - -# Online serving -vllm serve --omni --quantization ascend -``` - -## Supported Models - -| Model | HF Models | Recommendation | `ignored_layers` | -|-------|-----------|---------------|------------------| -| HunyuanImage-3.0 | - | All layers | None | - -Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a [HunyuanImage-3.0-adapted msModelSlim version](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the quantized weights manually. We will upload the quantized weights as soon as possible. diff --git a/docs/user_guide/diffusion/quantization/overview.md b/docs/user_guide/diffusion/quantization/overview.md deleted file mode 100644 index 25d7fa5c7..000000000 --- a/docs/user_guide/diffusion/quantization/overview.md +++ /dev/null @@ -1,153 +0,0 @@ -# Quantization - -vLLM-Omni provides a unified quantization framework that supports both diffusion models (DiT, text encoders, VAEs) and multi-stage omni models (e.g., Qwen3-Omni thinker). Quantization reduces memory usage and accelerates inference. - -## Supported Methods - -### Diffusion Models - -| Method | Guide | Description | Tested Models | Min GPU | -|--------|-------|-------------|---------------|---------| -| FP8 | [FP8](fp8.md) | FP8 W8A8, dynamic or static | Z-Image, Qwen-Image, Flux, Bagel | SM 89 (Ada) | -| Int8 | [Int8](int8.md) | Int8 W8A8 | Z-Image, Qwen-Image | SM 89 (Ada) / Ascend NPU | -| GGUF | [GGUF](gguf.md) | GGUF format, dequant+GEMM for N-D tensors | Z-Image, Flux | SM 60 | -| AutoRound | [AutoRound](autoround.md) | W4A16 (pre-quantized) | Flux | SM 80 (Ampere) | - -### Multi-stage Omni Models (Pre-quantized Checkpoints) - -| Method | Description | Tested Models | Min GPU | -|--------|-------------|---------------|---------| -| ModelOpt FP8 | Pre-quantized FP8 via NVIDIA ModelOpt | Qwen3-Omni (thinker) | SM 89 (Ada/Hopper) | -| ModelOpt NVFP4 | Pre-quantized NVFP4 via NVIDIA ModelOpt | Qwen3-Omni (experimental, quality issues) | SM 100 (Blackwell) | - -!!! note - AWQ, GPTQ, and BitsAndBytes are supported by vLLM's upstream quantization registry but have **not been tested** with vLLM-Omni pipelines. They may work via `build_quant_config()` but are not validated. - -### Pre-quantized LLM Checkpoints (Multi-stage Models) - -For multi-stage models like Qwen3-Omni, the unified quantization framework auto-detects -pre-quantized checkpoints via `quantization_config` in the HF config. Quantization is -automatically scoped to the thinker's `language_model` — audio encoder, vision encoder, -talker, and code2wav remain in BF16. - -| Format | `quant_algo` | Hardware | Example | Status | -|--------|-------------|----------|---------|--------| -| ModelOpt FP8 | `FP8` | Ada/Hopper (SM 89+) | `asdazd/Qwen3-Omni-30B-A3B-Instruct_modelopt_FP8` | Tested — 47% memory reduction, comparable throughput | -| ModelOpt NVFP4 | `NVFP4` | Blackwell (SM 100+) | `shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4` | Experimental — loads and runs but output quality is unacceptable | - -**Tested FP8 results (Qwen3-Omni, 1×H200):** - -| Config | Model Memory (GiB) | Mem Reduction | Decode (tok/s) | -|--------|-------------------|---------------|----------------| -| BF16 baseline | 59.26 | — | 41.6 | -| FP8 (ModelOpt) | 31.41 | 47% | 39.9 | - -FP8 enables the full 3-stage pipeline (thinker + talker + code2wav) on a single 64GB GPU, -which is impossible with BF16 (thinker alone requires 59.26 GiB). - -## Quantization Scope - -### Diffusion Models - -When `--quantization fp8` is enabled for diffusion models: - -| Component | What Gets Quantized | Mechanism | -|-----------|-------------------|-----------| -| **DiT (transformer)** | `nn.Linear` layers | vLLM W8A8 FP8 compute (Ada/Hopper) or weight-only (older GPUs) | -| **Text encoder** | `nn.Linear` layers | FP8 weight storage, BF16 compute | -| **VAE** | `nn.Conv2d`, `nn.Conv3d` layers | FP8 weight storage, BF16 compute | - -### Multi-stage Omni Models - -For pre-quantized omni model checkpoints, quantization is scoped per component: - -| Component | Quantized? | Notes | -|-----------|-----------|-------| -| **Thinker (language_model)** | Yes | FP8 or NVFP4 via ModelOpt | -| **Audio encoder** | No | Stays BF16 | -| **Vision encoder** | No | Stays BF16 | -| **Talker** | No | Stays BF16 | -| **Code2Wav** | No | Stays BF16 | - -!!! note - Not all models support all components. See the [FP8 supported models table](fp8.md#supported-models) for per-model details. - -## Device Compatibility for FP8 - -| Device | Example Hardware | FP8 | NVFP4 | -|--------|-----------------|-----|-------| -| Blackwell GPU (SM 100+) | RTX 5090, B100, B200 | Yes | Yes (native FP4 HW) | -| Ada/Hopper GPU (SM 89+) | RTX 4090, H100, H200 | Yes (W8A8 native) | No | -| Turing/Ampere GPU (SM 75-86) | RTX 3090, A100 | Yes (weight-only Marlin) | No | -| Ascend NPU | Atlas 800T A2 (910B) | Not yet supported | No | - -Kernel selection is automatic on CUDA GPUs. - -## Device Compatibility for Int8 - -| Device Type | Generation | Example | Int8 Mode | -|-------------|---------------|-------------------|----------| -| NVIDIA GPU | Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware | -| Ascend NPU | Atlas A2/Atlas A3 | Atlas 800T A2/Atlas 900 A3 | Full W8A8 with native hardware | - -## Python API - -The `build_quant_config()` factory accepts multiple input formats: - -```python -from vllm_omni.quantization import build_quant_config - -# String -config = build_quant_config("fp8") - -# Dict with parameters -config = build_quant_config({"method": "fp8", "activation_scheme": "static"}) - -# AutoRound / INC (auto-detected from checkpoint, or explicit) -config = build_quant_config("auto-round", bits=4, group_size=128) - -# Per-component dict -config = build_quant_config({ - "transformer": {"method": "fp8"}, - "vae": None, -}) - -# None / "none" -config = build_quant_config(None) # returns None -``` - -## Architecture - -``` -build_quant_config(spec) - ├── str → _build_single(method) → vLLM registry / _OVERRIDES - ├── dict with "method" → _build_single(method, **kwargs) - ├── per-component dict → ComponentQuantizationConfig - │ routes get_quant_method() by longest-prefix match - ├── QuantizationConfig → passthrough - └── None / "none" → None -``` - -## Migration Guide - -### Before (v0.14.x) - -```python -# Old diffusion-specific API (removed) -from vllm_omni.diffusion.quantization import get_diffusion_quant_config -config = get_diffusion_quant_config("fp8", activation_scheme="static") -``` - -### After (v0.16.0+) - -```python -# Unified API — delegates to vLLM's registry -from vllm_omni.quantization import build_quant_config -config = build_quant_config({"method": "fp8", "activation_scheme": "static"}) - -# Per-component (new) -config = build_quant_config({ - "transformer": {"method": "fp8"}, - "vae": None, -}) -``` diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 573cf5ee8..3e9ddb942 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -59,7 +59,7 @@ Memory optimization methods help reduce GPU memory usage, enabling inference on | Method | Description | Best For | |--------|-------------|----------| | **[CPU Offload](diffusion/cpu_offload_diffusion.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs | -| **[Quantization](diffusion/quantization/overview.md)** | Reduces DiT layers from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss | +| **[Quantization](quantization/overview.md)** | Reduces transformer stages from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss | | **[VAE Patch Parallelism](diffusion/parallelism/vae_patch_parallel.md)** | Distributes VAE decode tiling across GPUs | High-resolution generation with reduced VAE memory peak | ### Extensions @@ -86,9 +86,9 @@ Execution modes control how the diffusion pipeline processes denoise steps. | Method | Configuration | Description | Best For | |--------|--------------|-------------|----------| -| **[FP8](diffusion/quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on Ada/Hopper, weight-only on older GPUs | Memory reduction, inference speedup | -| **[INT8](diffusion/quantization/int8.md)** | `quantization="int8"` | INT8 weight-only, no calibration or pre-quantized checkpoint needed | Memory reduction, broad GPU compatibility | -| **[GGUF](diffusion/quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs | +| **[FP8](quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on validated transformer stages | Memory reduction, inference speedup | +| **[INT8](quantization/int8.md)** | `quantization="int8"` | INT8 W8A8 on validated transformer stages | Memory reduction, broad GPU compatibility | +| **[GGUF](quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs | ## Supported Models @@ -101,7 +101,7 @@ The following tables show which models support each feature: > Notes: > 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. -> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](diffusion/quantization/overview.md) for per-method (FP8, GGUF, …) and per-model support details. +> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](quantization/overview.md) for per-method and per-model support details. ### ImageGen @@ -261,7 +261,7 @@ Measured on NVIDIA H800: - **[CPU Offload Guide](diffusion/cpu_offload_diffusion.md)** - Offload model components to CPU, reduce GPU memory usage - **[VAE Patch Parallelism Guide](diffusion/parallelism/vae_patch_parallel.md)** - Distribute VAE decode tiling across GPUs for high-resolution images -- **[Quantization Overview](diffusion/quantization/overview.md)** - Overview of quantization methods for diffusion models +- **[Quantization Overview](quantization/overview.md)** - Overview of quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models **Extensions:** diff --git a/docs/user_guide/quantization/autoround.md b/docs/user_guide/quantization/autoround.md new file mode 100644 index 000000000..2261d79a5 --- /dev/null +++ b/docs/user_guide/quantization/autoround.md @@ -0,0 +1,127 @@ +# AutoRound Quantization + +## Overview + +[AutoRound](https://github.com/intel/auto-round) produces pre-quantized +checkpoints for LLMs, VLMs, and diffusion models. vLLM-Omni reads the +checkpoint's `config.json` and auto-detects +`quantization_config.quant_method = "auto-round"`. + +AutoRound is static quantization: no `--quantization` flag is needed at +inference time when the checkpoint already contains the quantization config. + +## Hardware Support + +| Device | Support | +|--------|---------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | +| AMD ROCm | ⭕ | +| Intel XPU | ✅ | +| Ascend NPU | ❌ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. AutoRound is Intel-supported. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Model | Checkpoint | Scope | Scheme | Backend | +|-------|------------|-------|--------|---------| +| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | +| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated | +| Wan2.2 | Not listed | Diffusion transformer | W4A16 | Not validated | + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Model | Checkpoint | Scope | Scheme | Backend | +|-------|------------|-------|--------|---------| +| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound | +| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound | +| Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated | + +AutoRound support is checkpoint-driven. A model is supported when its +checkpoint uses a compatible INC/AutoRound config and the target stage maps to +vLLM-Omni's runtime module names. + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint | +| GLM-Image | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint | + +## Configuration + +Python API: + +```python +from vllm_omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams + +omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16") + +outputs = omni.generate( + "A cat sitting on a windowsill", + OmniDiffusionSamplingParams(num_inference_steps=28), +) +outputs[0].save_images("output.png") +``` + +CLI: + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \ + --prompt "A cat sitting on a windowsill" \ + --num-inference-steps 28 \ + --output outputs/flux_w4a16.png +``` + +## Parameters + +| Field | Type | Description | +|-------|------|-------------| +| `quant_method` | str | Must be `"auto-round"` | +| `bits` | int | Quantized weight bit width, usually `4` | +| `group_size` | int | Quantization group size | +| `packing_format` | str | AutoRound packing format, for example `auto_round:auto_gptq` | +| `block_name_to_quantize` | str | Checkpoint block names that should map to runtime module names | + +The checkpoint should contain a config like: + +```json +{ + "quantization_config": { + "quant_method": "auto-round", + "bits": 4, + "group_size": 128, + "sym": true, + "packing_format": "auto_round:auto_gptq", + "block_name_to_quantize": "transformer_blocks,single_transformer_blocks" + } +} +``` + +## Validation and Notes + +At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names +to runtime module names, and selects the matching vLLM compute backend. + +Example checkpoint creation: + +```bash +auto-round \ + --model black-forest-labs/FLUX.1-dev \ + --scheme W4A16 \ + --batch_size 1 \ + --disable_opt_rtn \ + --dataset coco2014 \ + --iters 0 +``` + +Use the generated output directory directly as the `model` argument. See the +[AutoRound documentation](https://github.com/intel/auto-round) for all +available schemes and options. diff --git a/docs/user_guide/quantization/fp8.md b/docs/user_guide/quantization/fp8.md new file mode 100644 index 000000000..e89bc76ca --- /dev/null +++ b/docs/user_guide/quantization/fp8.md @@ -0,0 +1,121 @@ +# FP8 Quantization + +## Overview + +FP8 quantization converts BF16/FP16 weights to FP8 at model load time, or loads +a checkpoint whose target stage already declares an FP8 quantization config. +Online activation scaling is the default and does not require calibration. +Static activation scaling is supported when calibrated scale information is +available. + +Some architectures can quantize all linear layers. Others have +quality-sensitive layers that should stay in BF16 through `ignored_layers`. +Image-stream MLPs (`img_mlp`) are a common sensitive target because denoising +latent ranges shift across timesteps and small per-layer errors can compound +in deep DiT blocks. + +## Hardware Support + +| Device | Support | +|--------|---------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | +| AMD ROCm | ⭕ | +| Intel XPU | ⭕ | +| Ascend NPU | ❌ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. FP8 on Ampere may use a weight-only path where available. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Model | HF models | Online | Pre-calibrated | Recommendation | `ignored_layers` | +|-------|-----------|:-------:|:------:|----------------|------------------| +| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes | Yes | Skip sensitive image-stream MLPs when quality regresses | `img_mlp` | +| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Not validated | Validate against BF16 before documenting as supported | TBD | +| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Yes | All layers | None | +| FLUX.1 | `black-forest-labs/FLUX.1-dev`, `black-forest-labs/FLUX.1-schnell` | Yes | Yes | All layers | None | +| FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B` | Yes | Yes | All layers | None | +| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | Yes | Yes | All layers; use the Hunyuan stage config for multi-stage runs | None | +| HunyuanVideo-1.5 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `720p_t2v`, `480p_i2v` | Yes | Yes | All layers | None | + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Model | Scope | Format | Status | +|-------|-------|--------|--------| +| Qwen3-Omni | Thinker language-model stage | ModelOpt `quant_algo=FP8` | Tested for thinker memory reduction | +| Qwen3-TTS | TTS language-model stage | Checkpoint config | Not validated | + +Audio encoder, vision encoder, talker, and code2wav stay in BF16 unless a +model-specific guide says otherwise. + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| BAGEL | Stage-specific transformer or DiT module | Not validated | Route FP8 to the intended stage before enabling | +| GLM-Image | Stage-specific transformer or DiT module | Not validated | Validate quality against BF16 baseline | + +## Configuration + +Python API: + +```python +from vllm_omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams + +omni = Omni(model="", quantization="fp8") + +omni_with_skips = Omni( + model="", + quantization_config={ + "method": "fp8", + "ignored_layers": ["img_mlp"], + }, +) + +outputs = omni.generate( + "A cat sitting on a windowsill", + OmniDiffusionSamplingParams(num_inference_steps=50), +) +``` + +CLI: + +```bash +python text_to_image.py --model --quantization fp8 +python text_to_image.py --model --quantization fp8 --ignored-layers "img_mlp" +vllm serve --omni --quantization fp8 +``` + +## Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `method` | str | - | Quantization method (`"fp8"`) | +| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16 | +| `activation_scheme` | str | `"dynamic"` | `"dynamic"` selects online activation scaling, or `"static"` when scales are available | +| `weight_block_size` | list[int] \| None | `None` | Block size for block-wise weight quantization | + +The available `ignored_layers` names depend on the model architecture, for +example `to_qkv`, `to_out`, `img_mlp`, or `txt_mlp`. + +## Validation and Notes + +FP8 quantization can be combined with cache acceleration: + +```python +omni = Omni( + model="", + quantization="fp8", + cache_backend="tea_cache", + cache_config={"rel_l1_thresh": 0.2}, +) +``` + +Compare generated outputs with a BF16 baseline before adding a new model to the +supported table. GLM-Image and Helios are not listed as FP8-supported diffusion +models until they have method-specific validation. diff --git a/docs/user_guide/quantization/gguf.md b/docs/user_guide/quantization/gguf.md new file mode 100644 index 000000000..cd2e8b875 --- /dev/null +++ b/docs/user_guide/quantization/gguf.md @@ -0,0 +1,107 @@ +# GGUF Quantization + +## Overview + +GGUF support loads pre-quantized diffusion transformer weights while keeping +the rest of the pipeline on the base Hugging Face checkpoint. Use the base +model for tokenizer, text encoder, scheduler, and VAE, then pass the GGUF file +for the transformer. + +GGUF is static quantization: the quantized weights are produced before serving. + +## Hardware Support + +| Device | Support | +|--------|---------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | +| AMD ROCm | ⭕ | +| Intel XPU | ⭕ | +| Ascend NPU | ❌ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Model | HF base model | GGUF input | Scope | Adapter | +|-------|---------------|------------|-------|---------| +| Qwen-Image family | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512`, edit and layered Qwen-Image pipelines | Local `.gguf`, `repo/file.gguf`, or `repo:quant_type` | Transformer only | `QwenImageGGUFAdapter` | +| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Transformer only | No validated adapter listed | +| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Local `.gguf`, `repo/file.gguf`, or `repo:quant_type` | Transformer only | `ZImageGGUFAdapter` | +| FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B` | Local `.gguf`, `repo/file.gguf`, or `repo:quant_type` | Transformer only | `Flux2KleinGGUFAdapter` | + +Generic FLUX.1 GGUF checkpoints are not listed here; the implemented adapter is +for the FLUX.2-klein path. + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| Qwen3-Omni | Thinker language-model stage | Not validated | GGUF is not documented for omni/TTS AR stages | +| Qwen3-TTS | TTS language-model stage | Not validated | GGUF is not documented for TTS stages | + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| BAGEL | Stage-specific transformer weights | Not validated | Requires a model-specific GGUF adapter | +| GLM-Image | Stage-specific transformer weights | Not validated | Requires a model-specific GGUF adapter | + +## Configuration + +Offline: + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model Qwen/Qwen-Image \ + --gguf-model QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf \ + --quantization gguf \ + --prompt "a red paper kite hanging from a pine tree in a winter courtyard" \ + --height 1024 \ + --width 1024 \ + --seed 42 \ + --num_inference_steps 20 \ + --output outputs/qwen_image_q4km.png +``` + +Online: + +```bash +vllm serve Qwen/Qwen-Image \ + --omni \ + --port 8000 \ + --quantization-config '{"method":"gguf","gguf_model":"QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf"}' +``` + +## Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `method` | str | - | Quantization method (`"gguf"`) | +| `gguf_model` | str | - | Local GGUF file, explicit Hugging Face file, or `repo:quant_type` selector | + +`gguf_model` accepts: + +| Form | Example | +|------|---------| +| Local file | `/models/z-image-Q4_K_M.gguf` | +| Explicit HF file | `QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf` | +| HF repo plus quant type | `owner/repo:Q4_K_M` | + +## Validation and Notes + +1. `OmniDiffusionConfig` receives `{"method": "gguf", "gguf_model": ...}`. +2. `DiffusersPipelineLoader` resolves the GGUF file. +3. A model-specific adapter remaps GGUF tensor names to vLLM-Omni transformer + names. +4. Only transformer weights are loaded from GGUF. Missing non-transformer + weights are loaded from the base model repository. +5. vLLM's GGUF linear method performs dequantization and GEMM at runtime. + +Unsupported models fail fast with a clear "No GGUF adapter matched" error +instead of falling back to a generic mapper. Many GGUF repositories do not +include `model_index.json`; always pass the normal base model through `--model`. diff --git a/docs/user_guide/quantization/int8.md b/docs/user_guide/quantization/int8.md new file mode 100644 index 000000000..85bee9a5f --- /dev/null +++ b/docs/user_guide/quantization/int8.md @@ -0,0 +1,107 @@ +# Int8 Quantization + +## Overview + +Int8 quantization supports W8A8 diffusion transformer inference on CUDA and +Ascend NPU. It can quantize BF16/FP16 weights at load time, or load serialized +Int8 checkpoints that already contain quantized weights and scales. + +Only online activation scaling is currently supported. + +## Hardware Support + +| Device | Support | +|--------|---------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | +| AMD ROCm | ⭕ | +| Intel XPU | ⭕ | +| Ascend NPU | ✅ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Model | HF models | CUDA | Ascend NPU | Mode | Recommendation | +|-------|-----------|:----:|:----------:|------|----------------| +| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes | Yes | Online W8A8 | All layers | +| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Not validated | Online W8A8 | Validate before enabling in docs | +| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Yes | Online W8A8 | All layers | + +Other diffusion models may work if their transformer uses supported linear +layers, but they are not validated in this guide. + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| Qwen3-Omni | Thinker language-model stage | Not validated | Prefer checkpoint-supported ModelOpt FP8 or AutoRound paths | +| Qwen3-TTS | TTS language-model stage | Not validated | No Int8 TTS stage support is documented | + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| BAGEL | Stage-specific transformer or DiT module | Not validated | Requires explicit stage routing | +| GLM-Image | Stage-specific transformer or DiT module | Not validated | Requires quality comparison with BF16 | + +## Configuration + +Python API: + +```python +from vllm_omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams + +omni = Omni(model="", quantization="int8") + +omni_with_skips = Omni( + model="", + quantization_config={ + "method": "int8", + "ignored_layers": [""], + }, +) + +outputs = omni.generate( + "A cat sitting on a windowsill", + OmniDiffusionSamplingParams(num_inference_steps=50), +) +``` + +CLI: + +```bash +python text_to_image.py --model --quantization int8 +python text_to_image.py --model --quantization int8 --ignored-layers "img_mlp" +vllm serve --omni --quantization int8 +``` + +## Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `method` | str | - | Quantization method (`"int8"`) | +| `activation_scheme` | str | `"dynamic"` | `"dynamic"` selects online activation scaling; static is not supported | +| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16/FP16 | +| `is_checkpoint_int8_serialized` | bool | `False` | Set by checkpoint config when loading serialized Int8 weights | + +## Validation and Notes + +Int8 quantization can be combined with cache acceleration: + +```python +omni = Omni( + model="", + quantization="int8", + cache_backend="tea_cache", + cache_config={"rel_l1_thresh": 0.2}, +) +``` + +Only add a new model to the supported table after comparing the Int8 output +against a BF16 baseline and documenting any required `ignored_layers`. diff --git a/docs/user_guide/quantization/msmodelslim.md b/docs/user_guide/quantization/msmodelslim.md new file mode 100644 index 000000000..c03628b9d --- /dev/null +++ b/docs/user_guide/quantization/msmodelslim.md @@ -0,0 +1,95 @@ +# msModelSlim Quantization + +## Overview + +[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression +toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these +checkpoints run through the Ascend/NPU path with `--quantization ascend`. + +msModelSlim is static quantization: quantized weights are generated offline +before vLLM-Omni inference starts. + +## Hardware Support + +| Device | Support | +|--------|---------| +| NVIDIA Blackwell GPU (SM 100+) | ❌ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | +| NVIDIA Ampere GPU (SM 80+) | ❌ | +| AMD ROCm | ❌ | +| Intel XPU | ❌ | +| Ascend NPU | ✅ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Model | Base model | Scope | Hardware | Notes | +|-------|------------|-------|----------|-------| +| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed | +| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide | +| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation | + +Public Hugging Face quantized weights are not available yet. Use the +[HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) +to generate the checkpoint manually. + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| Qwen3-Omni | Thinker or language-model stage | Not validated | No msModelSlim omni checkpoint path is documented | +| Qwen3-TTS | TTS language-model stage | Not validated | No msModelSlim TTS checkpoint path is documented | + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Model | Scope | Status | Notes | +|-------|-------|--------|-------| +| BAGEL | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation | +| GLM-Image | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation | + +## Configuration + +Offline inference: + +```bash +python text_to_image.py --model --quantization ascend +``` + +Online serving: + +```bash +vllm serve --omni --quantization ascend +``` + +## Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints | +| `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling | + +Example msModelSlim command for a Wan2.2 W8A8 checkpoint: + +```bash +msmodelslim quant \ + --model_path /path/to/wan2_2_t2v_float_weights \ + --save_path /path/to/wan2_2_t2v_quantized_weights \ + --device npu \ + --model_type Wan2_2 \ + --config_path /path/to/wan2_2_w8a8f8_mxfp_t2v.yaml \ + --trust_remote_code True +``` + +For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above. + +## Validation and Notes + +1. Run with the Ascend/NPU installation and environment. +2. The `ascend` quantization method expects weights produced by the Ascend + tooling; it is not a load-time CUDA quantizer. +3. Keep the quantized checkpoint aligned with the same model architecture and + stage config used for BF16 inference. diff --git a/docs/user_guide/quantization/online.md b/docs/user_guide/quantization/online.md new file mode 100644 index 000000000..3dcb8297e --- /dev/null +++ b/docs/user_guide/quantization/online.md @@ -0,0 +1,93 @@ +# Online Quantization + +## Overview + +Online quantization means vLLM-Omni computes quantized weights and scales while +loading the model. Use it when you want memory savings without preparing a +separate quantized checkpoint. + +This mode is different from pre-quantized checkpoint formats such as GGUF, +AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are +prepared before serving and are documented in their method-specific guides. + +## Hardware Support + +| Device | FP8 W8A8 | Int8 W8A8 | +|--------|----------|-----------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | +| AMD ROCm | ⭕ | ⭕ | +| Intel XPU | ⭕ | ⭕ | +| Ascend NPU | ❌ | ✅ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. FP8 on Ampere may use a weight-only path where available. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +| Method | Guide | Example models | Status | +|--------|-------|----------------|--------| +| FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models | +| Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image | + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +Online quantization is not currently validated for the omni/TTS stages. For +Qwen3-Omni and related models, prefer checkpoint-declared ModelOpt or +AutoRound paths when available. + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +Online quantization must be routed to the intended stage. BAGEL and GLM-Image +need model-specific validation before they are listed as supported targets. + +## Configuration + +Python API: + +```python +from vllm_omni import Omni + +omni_fp8 = Omni(model="", quantization="fp8") +omni_int8 = Omni(model="", quantization="int8") +``` + +CLI: + +```bash +vllm serve --omni --quantization fp8 +vllm serve --omni --quantization int8 +``` + +Per-component routing: + +```python +from vllm_omni.quantization import build_quant_config + +config = build_quant_config({ + "transformer": {"method": "fp8"}, + "vae": None, +}) +``` + +## Parameters + +| Parameter | Methods | Description | +|-----------|---------|-------------| +| `method` | FP8, Int8 | Quantization method, usually `"fp8"` or `"int8"` | +| `ignored_layers` | FP8, Int8 | Layer name patterns to keep in BF16/FP16 | +| `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling | +| `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size | + +## Validation and Notes + +1. Compare the online-quantized output against a BF16 baseline with the same + seed and generation parameters. +2. Use `ignored_layers` for quality-sensitive MLPs or output projections. +3. Document any required skipped layers in the method page before marking a new + model as supported. +4. If a model already ships quantized weights, use the matching pre-quantized + method guide instead of online quantization. diff --git a/docs/user_guide/quantization/overview.md b/docs/user_guide/quantization/overview.md new file mode 100644 index 000000000..762fb2fc6 --- /dev/null +++ b/docs/user_guide/quantization/overview.md @@ -0,0 +1,134 @@ +# Quantization + +vLLM-Omni exposes quantization through the unified `quantization_config` +path. The same configuration entrypoint is used across diffusion-only models, +multi-stage omni/TTS models, and multi-stage diffusion models, but each model +type has a different quantization scope. + +## Quantization Modes + +| Mode | Guide | Description | Methods | +|------|-------|-------------|---------| +| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8 | +| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | GGUF, AutoRound, msModelSlim, serialized Int8 | + +## Hardware Support + +| Device | FP8 W8A8 | Int8 W8A8 | GGUF | AutoRound | msModelSlim | +|--------|----------|-----------|------|-----------|-------------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ❌ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ❌ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ✅ | ❌ | +| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ | ❌ | +| Intel XPU | ⭕ | ⭕ | ⭕ | ✅ | ❌ | +| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | + +Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +guide. FP8 on Ampere may use a weight-only path where available. + +## Model Type Support + +### Diffusion Model (Qwen-Image, Wan2.2) + +These models run a diffusion transformer as the primary inference module. The +default quantization target is the transformer; tokenizer, scheduler, text +encoder, and VAE stay on the base checkpoint unless a method guide says +otherwise. + +| Method | Guide | Mode | Example models | Status | +|--------|-------|------|----------------|--------| +| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models | +| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image | +| GGUF | [GGUF](gguf.md) | Pre-quantized transformer weights | Qwen-Image | Validated where a model-specific GGUF adapter exists | +| AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven | +| msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path | + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +These models combine an AR language model with audio, vision, talker, or TTS +stages. Quantization is scoped to the AR language-model stage when the +checkpoint contains a supported `quantization_config`; the non-AR stages stay +in BF16 unless the model guide explicitly adds support. + +| Method | Guide | Scope | Example models | Status | +|--------|-------|-------|----------------|--------| +| FP8 | [FP8](fp8.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path | +| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | +| GGUF | [GGUF](gguf.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | +| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints | +| msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +These models split generation across multiple stages. Quantization must be +attached to the intended stage rather than applied globally. + +| Method | Guide | Scope | Example models | Status | +|--------|-------|-------|----------------|--------| +| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation | +| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation | +| GGUF | [GGUF](gguf.md) | Stage-specific transformer weights | BAGEL, GLM-Image | No validated adapter listed | +| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed | +| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation | + +!!! note + "Online quantization" means vLLM-Omni computes the quantization data while + loading the model. "Pre-quantized" means the checkpoint or external + quantizer provides the required quantized weights and scales. + +## Quantization Scope + +### Diffusion Model (Qwen-Image, Wan2.2) + +The default target is the diffusion transformer. Component routing is available +through `build_quant_config()`: + +```python +from vllm_omni.quantization import build_quant_config + +config = build_quant_config({ + "transformer": {"method": "fp8"}, + "vae": None, +}) +``` + +| Component | Default quantized? | Notes | +|-----------|--------------------|-------| +| Diffusion transformer | Yes | Primary target for FP8, Int8, GGUF, AutoRound, and msModelSlim | +| Text encoder | No | Keep BF16 unless a method-specific guide documents support | +| VAE | No | Keep BF16; storage-only paths are method-specific | +| Scheduler/tokenizer | No | Loaded from the base model repository | + +### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) + +| Component | Default quantized? | Notes | +|-----------|--------------------|-------| +| Thinker or AR language model | Yes, when checkpoint config is supported | ModelOpt FP8/NVFP4 or AutoRound checkpoint config | +| Audio encoder | No | BF16 | +| Vision encoder | No | BF16 | +| Talker or TTS stage | No | BF16 unless model-specific support is documented | +| Code2Wav | No | BF16 | + +### Multi-Stage Diffusion Model (BAGEL, GLM-Image) + +| Component | Default quantized? | Notes | +|-----------|--------------------|-------| +| Selected diffusion or transformer stage | Method-specific | Must be routed to the intended stage | +| Other generation stages | No | Keep BF16 unless separately validated | +| VAE, tokenizer, scheduler | No | Loaded from the base checkpoint | + +## Python API + +`build_quant_config()` accepts strings, dictionaries, per-component +dictionaries, existing `QuantizationConfig` objects, or `None`. + +```python +from vllm_omni.quantization import build_quant_config + +build_quant_config("fp8") +build_quant_config({"method": "fp8", "activation_scheme": "static"}) +build_quant_config("auto-round", bits=4, group_size=128) +build_quant_config({"method": "gguf", "gguf_model": "/path/to/model.gguf"}) +build_quant_config({"transformer": {"method": "fp8"}, "vae": None}) +build_quant_config(None) +```