[Feature]: FP8 Quantization Support for DiT (#1034)

Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
This commit is contained in:
SYLAR authored and GitHub committed 2026-02-11 21:01:35 +08:00
1 parent f0bca6f66b
commit 830d261f9d
17 files changed
+688 -13

No files matched your search

+3
View File
@@ -43,6 +43,9 @@ nav:
- Overview: user_guide/diffusion_acceleration.md - Overview: user_guide/diffusion_acceleration.md
- TeaCache: user_guide/diffusion/teacache.md - TeaCache: user_guide/diffusion/teacache.md
- Cache-DiT: user_guide/diffusion/cache_dit_acceleration.md - Cache-DiT: user_guide/diffusion/cache_dit_acceleration.md
- Quantization:
- Overview: user_guide/diffusion/quantization/overview.md
- FP8: user_guide/diffusion/quantization/fp8.md
- Parallelism Acceleration: user_guide/diffusion/parallelism_acceleration.md - Parallelism Acceleration: user_guide/diffusion/parallelism_acceleration.md
- CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md - CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
- ComfyUI: features/comfyui.md - ComfyUI: features/comfyui.md
@@ -0,0 +1,77 @@
# FP8 Quantization
## Overview
FP8 quantization converts BF16/FP16 weights to FP8 at model load time. No calibration or pre-quantized checkpoint needed.
Depending on the model, either all layers can be quantized, or some sensitive layers should stay in BF16. See the [per-model table](#supported-models) for which case applies.
Common sensitive layers in DiT-based diffusion models include **image-stream MLPs** (`img_mlp`). These are particularly vulnerable to FP8 precision loss because they process denoising latents whose dynamic range shifts significantly across timesteps, and unlike attention projections (which benefit from QK-Norm stabilization), MLPs have no built-in normalization to absorb quantization error. In deep architectures (e.g., 60+ residual blocks), small per-layer errors compound and degrade output quality. Other layers such as **attention projections** (`to_qkv`, `to_out`) and **text-stream MLPs** (`txt_mlp`) are generally more robust due to normalization or more stable input statistics.
## Configuration
1. **Python API**: set `quantization="fp8"`. To skip sensitive layers, use `quantization_config` with `ignored_layers`.
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# All layers quantized
omni = Omni(model="<your-model>", quantization="fp8")
# Skip sensitive layers
omni = Omni(
model="<your-model>",
quantization_config={
"method": "fp8",
"ignored_layers": ["<layer-name>"],
},
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
2. **CLI**: pass `--quantization fp8` and optionally `--ignored-layers`.
```bash
# All layers
python text_to_image.py --model <your-model> --quantization fp8
# Skip sensitive layers
python text_to_image.py --model <your-model> --quantization fp8 --ignored-layers "img_mlp"
# Online serving
vllm serve <your-model> --omni --quantization fp8
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `method` | str | — | Quantization method (`"fp8"`) |
| `ignored_layers` | list[str] | `[]` | Layer name patterns to keep in BF16 |
| `activation_scheme` | str | `"dynamic"` | `"dynamic"` (no calibration) or `"static"` |
| `weight_block_size` | list[int] \| None | `None` | Block size for block-wise weight quantization |
The available `ignored_layers` names depend on the model architecture (e.g., `to_qkv`, `to_out`, `img_mlp`, `txt_mlp`). Consult the transformer source for your target model.
## Supported Models
| Model | HF Models | Recommendation | `ignored_layers` |
|-------|-----------|---------------|------------------|
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | All layers | None |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Skip sensitive layers | `img_mlp` |
## Combining with Other Features
FP8 quantization can be combined with cache acceleration:
```python
omni = Omni(
model="<your-model>",
quantization="fp8",
cache_backend="tea_cache",
cache_config={"rel_l1_thresh": 0.2},
)
```
@@ -0,0 +1,17 @@
# Quantization for Diffusion Transformers
vLLM-Omni supports quantization of DiT linear layers to reduce memory usage and accelerate inference.
## Supported Methods
| Method | Guide |
|--------|-------|
| FP8 | [FP8](fp8.md) |
## Device Compatibility
| GPU Generation | Example GPUs | FP8 Mode |
|---------------|-------------------|----------|
| Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
Kernel selection is automatic.
+38 -2
View File
@@ -1,6 +1,6 @@
# Diffusion Acceleration Overview # Diffusion Acceleration Overview
vLLM-Omni supports various cache acceleration methods to speed up diffusion model inference with minimal quality degradation. These methods include **cache methods** that intelligently cache intermediate computations to avoid redundant work across diffusion timesteps, and **parallelism methods** that distribute the computation across multiple devices. vLLM-Omni supports various acceleration methods to speed up diffusion model inference with minimal quality degradation. These include **cache methods** that intelligently cache intermediate computations to avoid redundant work across diffusion timesteps, **parallelism methods** that distribute the computation across multiple devices, and **quantization methods** that reduce memory footprint while preserving accuracy.
## Supported Acceleration Methods ## Supported Acceleration Methods
@@ -14,6 +14,10 @@ vLLM-Omni currently supports two main cache acceleration backends:
Both methods can provide significant speedups (typically **1.5x-2.0x**) while maintaining high output quality. Both methods can provide significant speedups (typically **1.5x-2.0x**) while maintaining high output quality.
vLLM-Omni also supports quantization methods:
3. **[FP8 Quantization](diffusion/quantization/overview.md)** - Reduces DiT linear layers from BF16 to FP8, providing ~1.28x speedup with minimal quality loss. Supports per-layer skip for sensitive layers.
vLLM-Omni also supports parallelism methods for diffusion models, including: vLLM-Omni also supports parallelism methods for diffusion models, including:
1. [Ulysses-SP](diffusion/parallelism_acceleration.md#ulysses-sp) - splits the input along the sequence dimension and uses all-to-all communication to allow each device to compute only a subset of attention heads. 1. [Ulysses-SP](diffusion/parallelism_acceleration.md#ulysses-sp) - splits the input along the sequence dimension and uses all-to-all communication to allow each device to compute only a subset of attention heads.
@@ -35,6 +39,12 @@ vLLM-Omni also supports parallelism methods for diffusion models, including:
| **TeaCache** | `cache_backend="tea_cache"` | Simple, adaptive caching with minimal configuration | Quick setup, balanced speed/quality | | **TeaCache** | `cache_backend="tea_cache"` | Simple, adaptive caching with minimal configuration | Quick setup, balanced speed/quality |
| **Cache-DiT** | `cache_backend="cache_dit"` | Advanced caching with multiple techniques (DBCache, TaylorSeer, SCM) | Maximum acceleration, fine-grained control | | **Cache-DiT** | `cache_backend="cache_dit"` | Advanced caching with multiple techniques (DBCache, TaylorSeer, SCM) | Maximum acceleration, fine-grained control |
### Quantization Methods
| Method | Configuration | Description | Best For |
|--------|--------------|-------------|----------|
| **FP8** | `quantization="fp8"` | FP8 W8A8 on Ada/Hopper, weight-only on older GPUs | Memory reduction, inference speedup |
## Supported Models ## Supported Models
The following table shows which models are currently supported by each acceleration method: The following table shows which models are currently supported by each acceleration method:
@@ -58,10 +68,18 @@ The following table shows which models are currently supported by each accelerat
### VideoGen ### VideoGen
| Model | Model Identifier | TeaCache | Cache-DiT | Ulysses-SP | Ring-Attention |CFG-Parallel | | Model | Model Identifier | TeaCache | Cache-DiT | Ulysses-SP | Ring-Attention | CFG-Parallel |
|-------|------------------|:--------:|:---------:|:----------:|:--------------:|:----------------:| |-------|------------------|:--------:|:---------:|:----------:|:--------------:|:----------------:|
| **Wan2.2** | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | ❌ | ✅ | ✅ | ✅ | ✅ | | **Wan2.2** | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | ❌ | ✅ | ✅ | ✅ | ✅ |
### Quantization
| Model | Model Identifier | FP8 |
|-------|------------------|:---:|
| **Qwen-Image** | `Qwen/Qwen-Image` | ✅ |
| **Qwen-Image-2512** | `Qwen/Qwen-Image-2512` | ✅ |
| **Z-Image** | `Tongyi-MAI/Z-Image-Turbo` | ✅ |
## Performance Benchmarks ## Performance Benchmarks
@@ -272,12 +290,30 @@ outputs = omni.generate(
) )
``` ```
### Using FP8 Quantization
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(
model="<your-model>",
quantization="fp8",
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
## Documentation ## Documentation
For detailed information on each acceleration method: For detailed information on each acceleration method:
- **[TeaCache Guide](diffusion/teacache.md)** - Complete TeaCache documentation, configuration options, and best practices - **[TeaCache Guide](diffusion/teacache.md)** - Complete TeaCache documentation, configuration options, and best practices
- **[Cache-DiT Acceleration Guide](diffusion/cache_dit_acceleration.md)** - Comprehensive Cache-DiT guide covering DBCache, TaylorSeer, SCM, and configuration parameters - **[Cache-DiT Acceleration Guide](diffusion/cache_dit_acceleration.md)** - Comprehensive Cache-DiT guide covering DBCache, TaylorSeer, SCM, and configuration parameters
- **[FP8 Quantization Guide](diffusion/quantization/overview.md)** - FP8 quantization for DiT models with per-layer control
- **[Tensor Parallelism](diffusion/parallelism_acceleration.md#tensor-parallelism)** - Guidance on how to enable TP for diffusion models. - **[Tensor Parallelism](diffusion/parallelism_acceleration.md#tensor-parallelism)** - Guidance on how to enable TP for diffusion models.
- **[Sequence Parallelism](diffusion/parallelism_acceleration.md#sequence-parallelism)** - Guidance on how to set sequence parallelism with configuration. - **[Sequence Parallelism](diffusion/parallelism_acceleration.md#sequence-parallelism)** - Guidance on how to set sequence parallelism with configuration.
- **[CFG-Parallel](diffusion/parallelism_acceleration.md#cfg-parallel)** - Guidance on how to set CFG-Parallel to run positive/negative branches across ranks. - **[CFG-Parallel](diffusion/parallelism_acceleration.md#cfg-parallel)** - Guidance on how to set CFG-Parallel to run positive/negative branches across ranks.
@@ -5,6 +5,7 @@ import argparse
import os import os
import time import time
from pathlib import Path from pathlib import Path
from typing import Any
import torch import torch
@@ -118,6 +119,24 @@ def parse_args() -> argparse.Namespace:
default=1, default=1,
help="Number of ready layers (blocks) to keep on GPU during generation.", help="Number of ready layers (blocks) to keep on GPU during generation.",
) )
parser.add_argument(
"--quantization",
type=str,
default=None,
choices=["fp8"],
help="Quantization method for the transformer. "
"Options: 'fp8' (FP8 W8A8 on Ada/Hopper, weight-only on older GPUs). "
"Default: None (no quantization, uses BF16).",
)
parser.add_argument(
"--ignored-layers",
type=str,
default=None,
help="Comma-separated list of layer name patterns to skip quantization. "
"Only used when --quantization is set. "
"Available layers: to_qkv, to_out, add_kv_proj, to_add_out, img_mlp, txt_mlp, proj_out. "
"Example: --ignored-layers 'add_kv_proj,to_add_out'",
)
parser.add_argument( parser.add_argument(
"--vae-use-slicing", "--vae-use-slicing",
action="store_true", action="store_true",
@@ -188,6 +207,18 @@ def main():
# Check if profiling is requested via environment variable # Check if profiling is requested via environment variable
profiler_enabled = bool(os.getenv("VLLM_TORCH_PROFILER_DIR")) profiler_enabled = bool(os.getenv("VLLM_TORCH_PROFILER_DIR"))
# Build quantization kwargs: use quantization_config dict when
# ignored_layers is specified so the list flows through OmniDiffusionConfig
quant_kwargs: dict[str, Any] = {}
ignored_layers = [s.strip() for s in args.ignored_layers.split(",") if s.strip()] if args.ignored_layers else None
if args.quantization and ignored_layers:
quant_kwargs["quantization_config"] = {
"method": args.quantization,
"ignored_layers": ignored_layers,
}
elif args.quantization:
quant_kwargs["quantization"] = args.quantization
omni = Omni( omni = Omni(
model=args.model, model=args.model,
enable_layerwise_offload=args.enable_layerwise_offload, enable_layerwise_offload=args.enable_layerwise_offload,
@@ -200,6 +231,7 @@ def main():
parallel_config=parallel_config, parallel_config=parallel_config,
enforce_eager=args.enforce_eager, enforce_eager=args.enforce_eager,
enable_cpu_offload=args.enable_cpu_offload, enable_cpu_offload=args.enable_cpu_offload,
**quant_kwargs,
) )
if profiler_enabled: if profiler_enabled:
@@ -212,6 +244,9 @@ def main():
print(f" Model: {args.model}") print(f" Model: {args.model}")
print(f" Inference steps: {args.num_inference_steps}") print(f" Inference steps: {args.num_inference_steps}")
print(f" Cache backend: {args.cache_backend if args.cache_backend else 'None (no acceleration)'}") print(f" Cache backend: {args.cache_backend if args.cache_backend else 'None (no acceleration)'}")
print(f" Quantization: {args.quantization if args.quantization else 'None (BF16)'}")
if ignored_layers:
print(f" Ignored layers: {ignored_layers}")
print( print(
f" Parallel configuration: tensor_parallel_size={args.tensor_parallel_size}, " f" Parallel configuration: tensor_parallel_size={args.tensor_parallel_size}, "
f"ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, " f"ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, "
+1
View File
@@ -94,6 +94,7 @@ plugins:
exclude: exclude:
- "re:vllm_omni\\._.*" # Internal modules - "re:vllm_omni\\._.*" # Internal modules
- "vllm_omni.diffusion.models.qwen_image" # avoid importing vllm in mkdocs building - "vllm_omni.diffusion.models.qwen_image" # avoid importing vllm in mkdocs building
- "vllm_omni.diffusion.quantization" # avoid importing vllm in mkdocs building
- "vllm_omni.entrypoints.async_diffusion" # avoid importing vllm in mkdocs building - "vllm_omni.entrypoints.async_diffusion" # avoid importing vllm in mkdocs building
- "vllm_omni.entrypoints.openai" # avoid importing vllm in mkdocs building - "vllm_omni.entrypoints.openai" # avoid importing vllm in mkdocs building
- "vllm_omni.entrypoints.openai.protocol" # avoid importing vllm in mkdocs building - "vllm_omni.entrypoints.openai.protocol" # avoid importing vllm in mkdocs building
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
@@ -0,0 +1,137 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Unit tests for FP8 quantization config."""
import pytest
def test_fp8_config_creation():
"""Test that FP8 config can be created."""
from vllm_omni.diffusion.quantization import get_diffusion_quant_config
config = get_diffusion_quant_config("fp8")
assert config is not None
assert config.get_name() == "fp8"
def test_vllm_config_extraction():
"""Test that vLLM config can be extracted from diffusion config."""
from vllm_omni.diffusion.quantization import (
get_diffusion_quant_config,
get_vllm_quant_config_for_layers,
)
diff_config = get_diffusion_quant_config("fp8")
vllm_config = get_vllm_quant_config_for_layers(diff_config)
assert vllm_config is not None
assert vllm_config.activation_scheme == "dynamic"
def test_none_quantization():
"""Test that None quantization returns None config."""
from vllm_omni.diffusion.quantization import (
get_diffusion_quant_config,
get_vllm_quant_config_for_layers,
)
config = get_diffusion_quant_config(None)
assert config is None
vllm_config = get_vllm_quant_config_for_layers(config)
assert vllm_config is None
def test_invalid_quantization():
"""Test that invalid quantization method raises error."""
from vllm_omni.diffusion.quantization import get_diffusion_quant_config
with pytest.raises(ValueError, match="Unknown quantization method"):
get_diffusion_quant_config("invalid_method")
def test_fp8_config_with_custom_params():
"""Test FP8 config with custom parameters."""
from vllm_omni.diffusion.quantization import get_diffusion_quant_config
config = get_diffusion_quant_config(
"fp8",
activation_scheme="static",
ignored_layers=["proj_out"],
)
assert config is not None
assert config.activation_scheme == "static"
assert "proj_out" in config.ignored_layers
def test_supported_methods():
"""Test that supported methods list is correct."""
from vllm_omni.diffusion.quantization import SUPPORTED_QUANTIZATION_METHODS
assert "fp8" in SUPPORTED_QUANTIZATION_METHODS
def test_quantization_integration():
"""Test end-to-end quantization flow through OmniDiffusionConfig."""
from vllm_omni.diffusion.data import OmniDiffusionConfig
# Test with quantization string only
config = OmniDiffusionConfig(model="test", quantization="fp8")
assert config.quantization_config is not None
assert config.quantization_config.get_name() == "fp8"
# Test with quantization_config dict
config2 = OmniDiffusionConfig(
model="test",
quantization_config={"method": "fp8", "activation_scheme": "static"},
)
assert config2.quantization_config is not None
assert config2.quantization_config.get_name() == "fp8"
assert config2.quantization_config.activation_scheme == "static"
# Test that vLLM config can be extracted
vllm_config = config.quantization_config.get_vllm_quant_config()
assert vllm_config is not None
def test_quantization_dict_not_mutated():
"""Test that passing a dict to quantization_config doesn't mutate it."""
from vllm_omni.diffusion.data import OmniDiffusionConfig
original_dict = {"method": "fp8", "activation_scheme": "static"}
dict_copy = original_dict.copy()
OmniDiffusionConfig(model="test", quantization_config=original_dict)
# Original dict should be unchanged
assert original_dict == dict_copy
def test_quantization_conflicting_methods_warning(caplog):
"""Test warning when quantization and quantization_config['method'] conflict."""
import logging
from vllm_omni.diffusion.data import OmniDiffusionConfig
with caplog.at_level(logging.WARNING):
config = OmniDiffusionConfig(
model="test",
quantization="fp8", # This should be overridden
quantization_config={"method": "fp8", "activation_scheme": "static"},
)
# No warning when methods match
assert config.quantization_config is not None
def test_fp8_delegates_to_vllm_config():
"""Test that DiffusionFp8Config delegates to vLLM's Fp8Config."""
from vllm.model_executor.layers.quantization.fp8 import Fp8Config
from vllm_omni.diffusion.quantization import DiffusionFp8Config
# Test that quant_config_cls is set correctly
assert DiffusionFp8Config.quant_config_cls is Fp8Config
# Test that get_name() delegates to vLLM
assert DiffusionFp8Config.get_name() == Fp8Config.get_name()
# Test that get_min_capability() delegates to vLLM
assert DiffusionFp8Config.get_min_capability() == Fp8Config.get_min_capability()
+37 -1
View File
@@ -4,7 +4,7 @@
import enum import enum
import os import os
import random import random
from collections.abc import Callable from collections.abc import Callable, Mapping
from dataclasses import dataclass, field, fields from dataclasses import dataclass, field, fields
from typing import Any from typing import Any
@@ -14,6 +14,10 @@ from typing_extensions import Self
from vllm.config.utils import config from vllm.config.utils import config
from vllm.logger import init_logger from vllm.logger import init_logger
from vllm_omni.diffusion.quantization import (
DiffusionQuantizationConfig,
get_diffusion_quant_config,
)
from vllm_omni.diffusion.utils.network_utils import is_port_available from vllm_omni.diffusion.utils.network_utils import is_port_available
logger = init_logger(__name__) logger = init_logger(__name__)
@@ -369,6 +373,11 @@ class OmniDiffusionConfig:
# Omni configuration (injected from stage config) # Omni configuration (injected from stage config)
omni_kv_config: dict[str, Any] = field(default_factory=dict) omni_kv_config: dict[str, Any] = field(default_factory=dict)
# Quantization settings
# Supported methods: "fp8" (FP8 W8A8 on Ada/Hopper, weight-only on older GPUs)
quantization: str | None = None
quantization_config: "DiffusionQuantizationConfig | dict[str, Any] | None" = None
def settle_port(self, port: int, port_inc: int = 42, max_attempts: int = 100) -> int: def settle_port(self, port: int, port_inc: int = 42, max_attempts: int = 100) -> int:
""" """
Find an available port with retry logic. Find an available port with retry logic.
@@ -455,6 +464,33 @@ class OmniDiffusionConfig:
# If it's neither dict nor DiffusionCacheConfig, convert to empty config # If it's neither dict nor DiffusionCacheConfig, convert to empty config
self.cache_config = DiffusionCacheConfig() self.cache_config = DiffusionCacheConfig()
# Convert quantization config
if self.quantization is not None or self.quantization_config is not None:
# Handle dict or DictConfig (from OmegaConf) - use Mapping for broader compatibility
if isinstance(self.quantization_config, Mapping):
# Convert DictConfig to dict if needed (OmegaConf compatibility)
config_dict = dict(self.quantization_config)
# Use get() instead of pop() to avoid mutating original dict
quant_method = config_dict.get("method", self.quantization)
# Filter out "method" key for kwargs
quant_kwargs = {k: v for k, v in config_dict.items() if k != "method"}
# Validate conflicting methods
if self.quantization is not None and quant_method is not None and quant_method != self.quantization:
logger.warning(
f"Conflicting quantization methods: quantization={self.quantization!r}, "
f"quantization_config['method']={quant_method!r}. Using quantization_config['method']."
)
self.quantization_config = get_diffusion_quant_config(quant_method, **quant_kwargs)
elif self.quantization_config is None and self.quantization is not None:
self.quantization_config = get_diffusion_quant_config(self.quantization)
elif not isinstance(self.quantization_config, DiffusionQuantizationConfig):
raise TypeError(
f"quantization_config must be a DiffusionQuantizationConfig, dict, or None, "
f"got {type(self.quantization_config)!r}"
)
if self.max_cpu_loras is None: if self.max_cpu_loras is None:
self.max_cpu_loras = 1 self.max_cpu_loras = 1
elif self.max_cpu_loras < 1: elif self.max_cpu_loras < 1:
@@ -13,6 +13,7 @@ from torch import nn
from vllm.config import ModelConfig from vllm.config import ModelConfig
from vllm.config.load import LoadConfig from vllm.config.load import LoadConfig
from vllm.logger import init_logger from vllm.logger import init_logger
from vllm.model_executor.layers.quantization.base_config import QuantizeMethodBase
from vllm.model_executor.model_loader.weight_utils import ( from vllm.model_executor.model_loader.weight_utils import (
download_safetensors_index_file_from_hf, download_safetensors_index_file_from_hf,
download_weights_from_hf, download_weights_from_hf,
@@ -217,8 +218,36 @@ class DiffusersPipelineLoader:
logger.debug("Loading weights on %s ...", load_device) logger.debug("Loading weights on %s ...", load_device)
# Quantization does not happen in `load_weights` but after it # Quantization does not happen in `load_weights` but after it
self.load_weights(model) self.load_weights(model)
# Process weights after loading for quantization (e.g., FP8 online quantization)
# This is needed for vLLM's quantization methods that need to transform weights
self._process_weights_after_loading(model, target_device)
return model.eval() return model.eval()
def _process_weights_after_loading(self, model: nn.Module, target_device: torch.device) -> None:
"""Process weights after loading for quantization methods.
This handles vLLM's quantization methods that need to process weights
after loading (e.g., FP8 online quantization from BF16/FP16 weights).
"""
for _, module in model.named_modules():
quant_method = getattr(module, "quant_method", None)
if isinstance(quant_method, QuantizeMethodBase):
# Move module to target device for processing if needed
module_device = next(module.parameters(), None)
if module_device is not None:
module_device = module_device.device
needs_device_move = module_device != target_device
if needs_device_move:
module.to(target_device)
quant_method.process_weights_after_loading(module)
if needs_device_move:
module.to(module_device)
def load_weights(self, model: nn.Module) -> None: def load_weights(self, model: nn.Module) -> None:
weights_to_load = {name for name, _ in model.named_parameters()} weights_to_load = {name for name, _ in model.named_parameters()}
loaded_weights = model.load_weights(self.get_all_weights(model)) loaded_weights = model.load_weights(self.get_all_weights(model))
@@ -33,6 +33,7 @@ from vllm_omni.diffusion.models.qwen_image.cfg_parallel import (
from vllm_omni.diffusion.models.qwen_image.qwen_image_transformer import ( from vllm_omni.diffusion.models.qwen_image.qwen_image_transformer import (
QwenImageTransformer2DModel, QwenImageTransformer2DModel,
) )
from vllm_omni.diffusion.quantization import get_vllm_quant_config_for_layers
from vllm_omni.diffusion.request import OmniDiffusionRequest from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.diffusion.utils.tf_utils import get_transformer_config_kwargs from vllm_omni.diffusion.utils.tf_utils import get_transformer_config_kwargs
from vllm_omni.model_executor.model_loader.weight_utils import ( from vllm_omni.model_executor.model_loader.weight_utils import (
@@ -272,7 +273,10 @@ class QwenImagePipeline(nn.Module, QwenImageCFGParallelMixin):
self.device self.device
) )
transformer_kwargs = get_transformer_config_kwargs(od_config.tf_model_config, QwenImageTransformer2DModel) transformer_kwargs = get_transformer_config_kwargs(od_config.tf_model_config, QwenImageTransformer2DModel)
self.transformer = QwenImageTransformer2DModel(od_config=od_config, **transformer_kwargs) quant_config = get_vllm_quant_config_for_layers(od_config.quantization_config)
self.transformer = QwenImageTransformer2DModel(
od_config=od_config, quant_config=quant_config, **transformer_kwargs
)
self.tokenizer = Qwen2Tokenizer.from_pretrained(model, subfolder="tokenizer", local_files_only=local_files_only) self.tokenizer = Qwen2Tokenizer.from_pretrained(model, subfolder="tokenizer", local_files_only=local_files_only)
@@ -1,10 +1,12 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from __future__ import annotations
from collections.abc import Iterable from collections.abc import Iterable
from functools import lru_cache from functools import lru_cache
from math import prod from math import prod
from typing import Any from typing import TYPE_CHECKING, Any
import torch import torch
import torch.nn as nn import torch.nn as nn
@@ -23,6 +25,11 @@ from vllm.model_executor.layers.linear import (
) )
from vllm.model_executor.model_loader.weight_utils import default_weight_loader from vllm.model_executor.model_loader.weight_utils import default_weight_loader
if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import (
QuantizationConfig,
)
from vllm_omni.diffusion.attention.backends.abstract import ( from vllm_omni.diffusion.attention.backends.abstract import (
AttentionMetadata, AttentionMetadata,
) )
@@ -390,7 +397,16 @@ class QwenEmbedRope(nn.Module):
class ColumnParallelApproxGELU(nn.Module): class ColumnParallelApproxGELU(nn.Module):
def __init__(self, dim_in: int, dim_out: int, *, approximate: str, bias: bool = True): def __init__(
self,
dim_in: int,
dim_out: int,
*,
approximate: str,
bias: bool = True,
quant_config: QuantizationConfig | None = None,
prefix: str = "",
):
super().__init__() super().__init__()
self.proj = ColumnParallelLinear( self.proj = ColumnParallelLinear(
dim_in, dim_in,
@@ -398,6 +414,8 @@ class ColumnParallelApproxGELU(nn.Module):
bias=bias, bias=bias,
gather_output=False, gather_output=False,
return_bias=False, return_bias=False,
quant_config=quant_config,
prefix=prefix,
) )
self.approximate = approximate self.approximate = approximate
@@ -415,6 +433,8 @@ class FeedForward(nn.Module):
activation_fn: str = "gelu-approximate", activation_fn: str = "gelu-approximate",
inner_dim: int | None = None, inner_dim: int | None = None,
bias: bool = True, bias: bool = True,
quant_config: QuantizationConfig | None = None,
prefix: str = "",
) -> None: ) -> None:
super().__init__() super().__init__()
@@ -424,13 +444,17 @@ class FeedForward(nn.Module):
dim_out = dim_out or dim dim_out = dim_out or dim
layers: list[nn.Module] = [ layers: list[nn.Module] = [
ColumnParallelApproxGELU(dim, inner_dim, approximate="tanh", bias=bias), ColumnParallelApproxGELU(
dim, inner_dim, approximate="tanh", bias=bias, quant_config=quant_config, prefix=prefix
),
nn.Identity(), # placeholder for weight loading nn.Identity(), # placeholder for weight loading
RowParallelLinear( RowParallelLinear(
inner_dim, inner_dim,
dim_out, dim_out,
input_is_parallel=True, input_is_parallel=True,
return_bias=False, return_bias=False,
quant_config=quant_config,
prefix=prefix,
), ),
] ]
@@ -456,6 +480,7 @@ class QwenImageCrossAttention(nn.Module):
pre_only: bool = False, pre_only: bool = False,
context_pre_only: bool = False, context_pre_only: bool = False,
out_dim: int | None = None, out_dim: int | None = None,
quant_config: QuantizationConfig | None = None,
) -> None: ) -> None:
super().__init__() super().__init__()
assert dim % num_heads == 0 assert dim % num_heads == 0
@@ -471,6 +496,8 @@ class QwenImageCrossAttention(nn.Module):
hidden_size=dim, hidden_size=dim,
head_size=self.head_dim, head_size=self.head_dim,
total_num_heads=num_heads, total_num_heads=num_heads,
quant_config=quant_config,
prefix="to_qkv",
) )
self.query_num_heads = self.to_qkv.num_heads self.query_num_heads = self.to_qkv.num_heads
self.kv_num_heads = self.to_qkv.num_kv_heads self.kv_num_heads = self.to_qkv.num_kv_heads
@@ -485,6 +512,8 @@ class QwenImageCrossAttention(nn.Module):
hidden_size=added_kv_proj_dim, hidden_size=added_kv_proj_dim,
head_size=head_dim, head_size=head_dim,
total_num_heads=num_heads, total_num_heads=num_heads,
quant_config=quant_config,
prefix="add_kv_proj",
) )
self.add_query_num_heads = self.add_kv_proj.num_heads self.add_query_num_heads = self.add_kv_proj.num_heads
self.add_kv_num_heads = self.add_kv_proj.num_kv_heads self.add_kv_num_heads = self.add_kv_proj.num_kv_heads
@@ -496,6 +525,8 @@ class QwenImageCrossAttention(nn.Module):
bias=out_bias, bias=out_bias,
input_is_parallel=True, input_is_parallel=True,
return_bias=False, return_bias=False,
quant_config=quant_config,
prefix="to_add_out",
) )
assert not pre_only assert not pre_only
@@ -505,6 +536,8 @@ class QwenImageCrossAttention(nn.Module):
bias=out_bias, bias=out_bias,
input_is_parallel=True, input_is_parallel=True,
return_bias=False, return_bias=False,
quant_config=quant_config,
prefix="to_out",
) )
self.norm_added_q = RMSNorm(head_dim, eps=eps) self.norm_added_q = RMSNorm(head_dim, eps=eps)
@@ -637,6 +670,7 @@ class QwenImageTransformerBlock(nn.Module):
qk_norm: str = "rms_norm", qk_norm: str = "rms_norm",
eps: float = 1e-6, eps: float = 1e-6,
zero_cond_t: bool = False, zero_cond_t: bool = False,
quant_config: QuantizationConfig | None = None,
): ):
super().__init__() super().__init__()
@@ -656,9 +690,10 @@ class QwenImageTransformerBlock(nn.Module):
added_kv_proj_dim=dim, added_kv_proj_dim=dim,
context_pre_only=False, context_pre_only=False,
head_dim=attention_head_dim, head_dim=attention_head_dim,
quant_config=quant_config,
) )
self.img_norm2 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps) self.img_norm2 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps)
self.img_mlp = FeedForward(dim=dim, dim_out=dim) self.img_mlp = FeedForward(dim=dim, dim_out=dim, quant_config=quant_config, prefix="img_mlp")
# Text processing modules # Text processing modules
self.txt_mod = nn.Sequential( self.txt_mod = nn.Sequential(
@@ -668,7 +703,7 @@ class QwenImageTransformerBlock(nn.Module):
self.txt_norm1 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps) self.txt_norm1 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps)
# Text doesn't need separate attention - it's handled by img_attn joint computation # Text doesn't need separate attention - it's handled by img_attn joint computation
self.txt_norm2 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps) self.txt_norm2 = AdaLayerNorm(dim, elementwise_affine=False, eps=eps)
self.txt_mlp = FeedForward(dim=dim, dim_out=dim) self.txt_mlp = FeedForward(dim=dim, dim_out=dim, quant_config=quant_config, prefix="txt_mlp")
self.zero_cond_t = zero_cond_t self.zero_cond_t = zero_cond_t
@@ -862,6 +897,7 @@ class QwenImageTransformer2DModel(CachedTransformer):
zero_cond_t: bool = False, zero_cond_t: bool = False,
use_additional_t_cond: bool = False, use_additional_t_cond: bool = False,
use_layer3d_rope: bool = False, use_layer3d_rope: bool = False,
quant_config: QuantizationConfig | None = None,
): ):
super().__init__() super().__init__()
self.parallel_config = od_config.parallel_config self.parallel_config = od_config.parallel_config
@@ -891,6 +927,7 @@ class QwenImageTransformer2DModel(CachedTransformer):
num_attention_heads=num_attention_heads, num_attention_heads=num_attention_heads,
attention_head_dim=attention_head_dim, attention_head_dim=attention_head_dim,
zero_cond_t=zero_cond_t, zero_cond_t=zero_cond_t,
quant_config=quant_config,
) )
for _ in range(num_layers) for _ in range(num_layers)
] ]
@@ -37,6 +37,7 @@ from vllm_omni.diffusion.model_loader.diffusers_loader import DiffusersPipelineL
from vllm_omni.diffusion.models.z_image.z_image_transformer import ( from vllm_omni.diffusion.models.z_image.z_image_transformer import (
ZImageTransformer2DModel, ZImageTransformer2DModel,
) )
from vllm_omni.diffusion.quantization import get_vllm_quant_config_for_layers
from vllm_omni.diffusion.request import OmniDiffusionRequest from vllm_omni.diffusion.request import OmniDiffusionRequest
from vllm_omni.model_executor.model_loader.weight_utils import ( from vllm_omni.model_executor.model_loader.weight_utils import (
download_weights_from_hf_specific, download_weights_from_hf_specific,
@@ -173,7 +174,9 @@ class ZImagePipeline(nn.Module):
self.vae = AutoencoderKL.from_pretrained(model, subfolder="vae", local_files_only=local_files_only).to( self.vae = AutoencoderKL.from_pretrained(model, subfolder="vae", local_files_only=local_files_only).to(
self._execution_device self._execution_device
) )
self.transformer = ZImageTransformer2DModel() # Get vLLM quantization config for linear layers
quant_config = get_vllm_quant_config_for_layers(od_config.quantization_config)
self.transformer = ZImageTransformer2DModel(quant_config=quant_config)
self.tokenizer = AutoTokenizer.from_pretrained(model, subfolder="tokenizer", local_files_only=local_files_only) self.tokenizer = AutoTokenizer.from_pretrained(model, subfolder="tokenizer", local_files_only=local_files_only)
# Note: Context parallelism is applied centrally in registry.initialize_model() # Note: Context parallelism is applied centrally in registry.initialize_model()
@@ -18,6 +18,7 @@
import math import math
from collections.abc import Iterable from collections.abc import Iterable
from typing import TYPE_CHECKING
import torch import torch
import torch.nn as nn import torch.nn as nn
@@ -32,6 +33,11 @@ from vllm.model_executor.layers.linear import (
) )
from vllm.model_executor.model_loader.weight_utils import default_weight_loader from vllm.model_executor.model_loader.weight_utils import default_weight_loader
if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import (
QuantizationConfig,
)
from vllm_omni.diffusion.attention.layer import Attention from vllm_omni.diffusion.attention.layer import Attention
from vllm_omni.diffusion.cache.base import CachedTransformer from vllm_omni.diffusion.cache.base import CachedTransformer
from vllm_omni.diffusion.distributed.sp_plan import ( from vllm_omni.diffusion.distributed.sp_plan import (
@@ -250,6 +256,7 @@ class ZImageAttention(nn.Module):
num_kv_heads: int, num_kv_heads: int,
qk_norm: bool = True, qk_norm: bool = True,
eps: float = 1e-6, eps: float = 1e-6,
quant_config: "QuantizationConfig | None" = None,
) -> None: ) -> None:
super().__init__() super().__init__()
self.dim = dim self.dim = dim
@@ -264,6 +271,7 @@ class ZImageAttention(nn.Module):
total_num_heads=num_heads, total_num_heads=num_heads,
total_num_kv_heads=num_kv_heads, total_num_kv_heads=num_kv_heads,
bias=False, bias=False,
quant_config=quant_config,
) )
assert qk_norm is True assert qk_norm is True
@@ -281,6 +289,7 @@ class ZImageAttention(nn.Module):
bias=False, bias=False,
input_is_parallel=True, input_is_parallel=True,
return_bias=False, return_bias=False,
quant_config=quant_config,
) )
] ]
) )
@@ -343,13 +352,19 @@ class ZImageAttention(nn.Module):
class FeedForward(nn.Module): class FeedForward(nn.Module):
def __init__(self, dim: int, hidden_dim: int): def __init__(
self,
dim: int,
hidden_dim: int,
quant_config: "QuantizationConfig | None" = None,
):
super().__init__() super().__init__()
self.w13 = MergedColumnParallelLinear( self.w13 = MergedColumnParallelLinear(
dim, dim,
[hidden_dim] * 2, [hidden_dim] * 2,
bias=False, bias=False,
return_bias=False, return_bias=False,
quant_config=quant_config,
) )
self.act = SiluAndMul() self.act = SiluAndMul()
self.w2 = RowParallelLinear( self.w2 = RowParallelLinear(
@@ -358,6 +373,7 @@ class FeedForward(nn.Module):
bias=False, bias=False,
input_is_parallel=True, input_is_parallel=True,
return_bias=False, return_bias=False,
quant_config=quant_config,
) )
def forward(self, x): def forward(self, x):
@@ -374,6 +390,7 @@ class ZImageTransformerBlock(nn.Module):
norm_eps: float, norm_eps: float,
qk_norm: bool, qk_norm: bool,
modulation=True, modulation=True,
quant_config: "QuantizationConfig | None" = None,
): ):
super().__init__() super().__init__()
self.dim = dim self.dim = dim
@@ -384,9 +401,14 @@ class ZImageTransformerBlock(nn.Module):
num_kv_heads=n_kv_heads, num_kv_heads=n_kv_heads,
qk_norm=qk_norm, qk_norm=qk_norm,
eps=1e-5, eps=1e-5,
quant_config=quant_config,
) )
self.feed_forward = FeedForward(dim=dim, hidden_dim=int(dim / 3 * 8)) self.feed_forward = FeedForward(
dim=dim,
hidden_dim=int(dim / 3 * 8),
quant_config=quant_config,
)
self.layer_id = layer_id self.layer_id = layer_id
self.attention_norm1 = RMSNorm(dim, eps=norm_eps) self.attention_norm1 = RMSNorm(dim, eps=norm_eps)
@@ -589,6 +611,7 @@ class ZImageTransformer2DModel(CachedTransformer):
t_scale=1000.0, t_scale=1000.0,
axes_dims=[32, 48, 48], axes_dims=[32, 48, 48],
axes_lens=[1024, 512, 512], axes_lens=[1024, 512, 512],
quant_config: "QuantizationConfig | None" = None,
) -> None: ) -> None:
super().__init__() super().__init__()
self.dtype = torch.bfloat16 self.dtype = torch.bfloat16
@@ -648,6 +671,7 @@ class ZImageTransformer2DModel(CachedTransformer):
norm_eps, norm_eps,
qk_norm, qk_norm,
modulation=True, modulation=True,
quant_config=quant_config,
) )
for layer_id in range(n_refiner_layers) for layer_id in range(n_refiner_layers)
] ]
@@ -662,6 +686,7 @@ class ZImageTransformer2DModel(CachedTransformer):
norm_eps, norm_eps,
qk_norm, qk_norm,
modulation=False, modulation=False,
quant_config=quant_config,
) )
for layer_id in range(n_refiner_layers) for layer_id in range(n_refiner_layers)
] ]
@@ -677,7 +702,15 @@ class ZImageTransformer2DModel(CachedTransformer):
self.layers = nn.ModuleList( self.layers = nn.ModuleList(
[ [
ZImageTransformerBlock(layer_id, dim, n_heads, n_kv_heads, norm_eps, qk_norm) ZImageTransformerBlock(
layer_id,
dim,
n_heads,
n_kv_heads,
norm_eps,
qk_norm,
quant_config=quant_config,
)
for layer_id in range(n_layers) for layer_id in range(n_layers)
] ]
) )
@@ -0,0 +1,114 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Quantization support for diffusion models.
This module provides a unified interface for quantizing diffusion transformers
using various methods (FP8, etc.). It wraps vLLM's quantization infrastructure
while allowing diffusion-model-specific defaults and optimizations.
Example usage:
from vllm_omni.diffusion.quantization import (
get_diffusion_quant_config,
get_vllm_quant_config_for_layers,
)
# Create FP8 config for diffusion model
diff_config = get_diffusion_quant_config("fp8")
# Get vLLM config to pass to linear layers
vllm_config = get_vllm_quant_config_for_layers(diff_config)
# Use in model initialization
linear_layer = QKVParallelLinear(..., quant_config=vllm_config)
"""
from typing import TYPE_CHECKING
from vllm.logger import init_logger
from .base import DiffusionQuantizationConfig
from .fp8 import DiffusionFp8Config
if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import (
QuantizationConfig,
)
logger = init_logger(__name__)
# Registry of supported quantization methods
# To add a new method, create a new config class and register it here
_QUANT_CONFIG_REGISTRY: dict[str, type[DiffusionQuantizationConfig]] = {
"fp8": DiffusionFp8Config,
}
SUPPORTED_QUANTIZATION_METHODS = list(_QUANT_CONFIG_REGISTRY.keys())
def get_diffusion_quant_config(
quantization: str | None,
**kwargs,
) -> DiffusionQuantizationConfig | None:
"""Factory function to create quantization config for diffusion models.
Args:
quantization: Quantization method name ("fp8", etc.) or None to disable
**kwargs: Method-specific parameters passed to the config constructor
Returns:
DiffusionQuantizationConfig instance or None if quantization is disabled
Raises:
ValueError: If the quantization method is not supported
Example:
# Default FP8 with dynamic activation scaling
config = get_diffusion_quant_config("fp8")
# FP8 with custom parameters
config = get_diffusion_quant_config(
"fp8",
activation_scheme="static",
ignored_layers=["proj_out"],
)
"""
if quantization is None or quantization.lower() == "none":
return None
quantization = quantization.lower()
if quantization not in _QUANT_CONFIG_REGISTRY:
raise ValueError(
f"Unknown quantization method: {quantization!r}. Supported methods: {SUPPORTED_QUANTIZATION_METHODS}"
)
config_cls = _QUANT_CONFIG_REGISTRY[quantization]
logger.info("Creating diffusion quantization config: %s", quantization)
return config_cls(**kwargs)
def get_vllm_quant_config_for_layers(
diffusion_quant_config: DiffusionQuantizationConfig | None,
) -> "QuantizationConfig | None":
"""Get the vLLM QuantizationConfig to pass to linear layers.
This extracts the underlying vLLM config from a DiffusionQuantizationConfig,
which can then be passed to vLLM linear layers (QKVParallelLinear, etc.).
Args:
diffusion_quant_config: The diffusion quantization config, or None
Returns:
vLLM QuantizationConfig instance, or None if input is None
"""
if diffusion_quant_config is None:
return None
return diffusion_quant_config.get_vllm_quant_config()
__all__ = [
"DiffusionQuantizationConfig",
"DiffusionFp8Config",
"get_diffusion_quant_config",
"get_vllm_quant_config_for_layers",
"SUPPORTED_QUANTIZATION_METHODS",
]
+61
View File
@@ -0,0 +1,61 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Base class for diffusion model quantization configurations."""
from abc import ABC
from typing import TYPE_CHECKING, ClassVar
import torch
if TYPE_CHECKING:
from vllm.model_executor.layers.quantization.base_config import (
QuantizationConfig,
)
class DiffusionQuantizationConfig(ABC):
"""Base class for diffusion model quantization configurations.
This provides a thin wrapper over vLLM's quantization configs,
allowing diffusion-model-specific defaults and future extensibility.
Subclasses should:
- Set quant_config_cls to the vLLM QuantizationConfig class
- Call super().__init__() after creating self._vllm_config
- Optionally override get_name() and get_min_capability() if needed
"""
# Subclasses should set this to the vLLM QuantizationConfig class
quant_config_cls: ClassVar[type["QuantizationConfig"] | None] = None
# The underlying vLLM config instance
_vllm_config: "QuantizationConfig | None" = None
@classmethod
def get_name(cls) -> str:
"""Return the quantization method name (e.g., 'fp8', 'int8').
By default, delegates to the underlying vLLM config class.
"""
if cls.quant_config_cls is not None:
return cls.quant_config_cls.get_name()
raise NotImplementedError("Subclass must set quant_config_cls or override get_name()")
def get_vllm_quant_config(self) -> "QuantizationConfig | None":
"""Return the underlying vLLM QuantizationConfig for linear layers."""
return self._vllm_config
@classmethod
def get_supported_act_dtypes(cls) -> list[torch.dtype]:
"""Return supported activation dtypes."""
return [torch.bfloat16, torch.float16]
@classmethod
def get_min_capability(cls) -> int:
"""Minimum GPU compute capability required.
By default, delegates to the underlying vLLM config class.
"""
if cls.quant_config_cls is not None:
return cls.quant_config_cls.get_min_capability()
return 80 # Ampere default
+50
View File
@@ -0,0 +1,50 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""FP8 quantization config for diffusion transformers."""
from vllm.model_executor.layers.quantization.fp8 import Fp8Config
from .base import DiffusionQuantizationConfig
class DiffusionFp8Config(DiffusionQuantizationConfig):
"""FP8 quantization config optimized for diffusion transformers.
Uses dynamic activation scaling (no calibration dataset needed) and
online weight quantization from BF16/FP16 checkpoints.
Device Compatibility:
- Turing (SM 75+): Weight-only FP8 via Marlin kernel
- Ada/Hopper (SM 89+): Full W8A8 FP8 with native hardware support
The kernel selection is automatic based on GPU capability.
Args:
activation_scheme: Activation quantization scheme.
- "dynamic": Per-token dynamic scaling (default, no calibration)
- "static": Single per-tensor scale (requires calibration)
weight_block_size: Block size for block-wise weight quantization.
Format: [block_n, block_k]. If None, uses per-tensor scaling.
ignored_layers: List of layer name patterns to skip quantization.
"""
# Tight coupling with vLLM's Fp8Config - delegates get_name() and get_min_capability()
quant_config_cls = Fp8Config
def __init__(
self,
activation_scheme: str = "dynamic",
weight_block_size: list[int] | None = None,
ignored_layers: list[str] | None = None,
):
self.activation_scheme = activation_scheme
self.weight_block_size = weight_block_size
self.ignored_layers = ignored_layers or []
# Create underlying vLLM FP8 config
self._vllm_config = Fp8Config(
is_checkpoint_fp8_serialized=False, # Online quantization from BF16
activation_scheme=activation_scheme,
weight_block_size=weight_block_size,
ignored_layers=ignored_layers,
)