[Feature]: Native GGUF Quantization Support for DiT (#1285)
Signed-off-by: David Chen <530634352@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
23 files changed
+1104
-67
No files matched your search
@@ -46,6 +46,7 @@ nav:
|
||||
- Quantization:
|
||||
- Overview: user_guide/diffusion/quantization/overview.md
|
||||
- FP8: user_guide/diffusion/quantization/fp8.md
|
||||
- GGUF: user_guide/diffusion/quantization/gguf.md
|
||||
- Parallelism Acceleration: user_guide/diffusion/parallelism_acceleration.md
|
||||
- CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
|
||||
- LoRA: user_guide/diffusion/lora.md
|
||||
|
||||
@@ -124,6 +124,7 @@ def extract_omni_serve_subparser_init():
|
||||
"logger": logger,
|
||||
"DummySubparsers": DummySubparsers,
|
||||
"argparse": __import__("argparse"),
|
||||
"json": __import__("json"),
|
||||
"DESCRIPTION": DESCRIPTION,
|
||||
}
|
||||
exec(code, exec_globals, local_vars)
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
# GGUF Quantization
|
||||
|
||||
## Goals
|
||||
1. Reuse vLLM quantization configs and weight loaders as much as possible.
|
||||
2. Add native GGUF support to diffusion transformers without changing model definitions.
|
||||
3. Keep user-facing knobs minimal and consistent across offline and online flows.
|
||||
|
||||
## Scope
|
||||
1. Models: Z-Image, and Flux2-klein.
|
||||
2. Components: diffusion transformer weights, loader paths, and quantization configs.
|
||||
3. Modes: native GGUF (transformer-only weights).
|
||||
|
||||
## Architecture Overview
|
||||
1. `OmniDiffusionConfig` accepts `quantization` or `quantization_config`.
|
||||
2. Diffusion quantization wrapper (`DiffusionGgufConfig`) produces vLLM `QuantizationConfig` objects for linear layers.
|
||||
3. `DiffusersPipelineLoader` branches on quantization method and loads either HF weights or GGUF weights for the transformer.
|
||||
4. GGUF transformer loading is routed through model-specific adapters (e.g., Flux2Klein).
|
||||
5. vLLM GGUF path uses `GGUFConfig` and `GGUFLinearMethod` for matmul.
|
||||
|
||||
## Call Chain (Offline)
|
||||
```
|
||||
CLI (examples/offline_inference/text_to_image/text_to_image.py)
|
||||
|
|
||||
v
|
||||
Omni (vllm_omni/entrypoints/omni.py)
|
||||
|
|
||||
v
|
||||
OmniStage (diffusion)
|
||||
|
|
||||
v
|
||||
DiffusionWorker
|
||||
|
|
||||
v
|
||||
DiffusionModelRunner
|
||||
|
|
||||
v
|
||||
DiffusersPipelineLoader
|
||||
|
|
||||
v
|
||||
Pipeline.forward (Flux2/Qwen/Z-Image)
|
||||
|
|
||||
v
|
||||
DiffusionEngine
|
||||
|
|
||||
v
|
||||
OmniRequestOutput
|
||||
|
|
||||
v
|
||||
Client (saved PNG)
|
||||
```
|
||||
|
||||
## Call Chain (Online)
|
||||
```
|
||||
Client
|
||||
|
|
||||
| POST /v1/images/generations
|
||||
v
|
||||
APIServer (vllm_omni/entrypoints/openai/api_server.py)
|
||||
|
|
||||
v
|
||||
_generate_with_async_omni
|
||||
|
|
||||
v
|
||||
AsyncOmni
|
||||
|
|
||||
v
|
||||
DiffusionEngine
|
||||
|
|
||||
v
|
||||
OmniRequestOutput
|
||||
|
|
||||
v
|
||||
encode_image_base64
|
||||
|
|
||||
v
|
||||
ImageGenerationResponse
|
||||
|
|
||||
v
|
||||
Client
|
||||
```
|
||||
|
||||
## Call Chain (GGUF Operator Path)
|
||||
```
|
||||
Pipeline.forward (Flux2/Qwen/Z-Image)
|
||||
|
|
||||
v
|
||||
Transformer blocks
|
||||
|
|
||||
v
|
||||
QKVParallelLinear / ColumnParallelLinear / RowParallelLinear
|
||||
|
|
||||
v
|
||||
LinearBase.forward
|
||||
|
|
||||
v
|
||||
QuantMethod.apply (GGUFLinearMethod.apply)
|
||||
|
|
||||
v
|
||||
fused_mul_mat_gguf
|
||||
|
|
||||
v
|
||||
_fused_mul_mat_gguf (custom op)
|
||||
|
|
||||
v
|
||||
ops.ggml_dequantize
|
||||
|
|
||||
v
|
||||
x @ weight.T
|
||||
```
|
||||
|
||||
## GGUF Weight Loading Path (Transformer-Only)
|
||||
1. `DiffusersPipelineLoader.load_model` detects `quantization_config.method == "gguf"`.
|
||||
2. `gguf_model` is resolved as one of: local file, `repo/file.gguf`, or `repo:quant_type`.
|
||||
3. GGUF weights are routed through adapters in `vllm_omni/diffusion/model_loader/gguf_adapters/`.
|
||||
4. Name mapping is applied per-architecture (Z-Image, Flux2Klein).
|
||||
5. GGUF weights are loaded into transformer modules, remaining non-transformer weights come from the HF checkpoint.
|
||||
|
||||
## GGUF Adapter Design
|
||||
1. `GGUFAdapter` is an abstract base class for model-specific adapters.
|
||||
2. `Flux2KleinGGUFAdapter` implements Flux2-Klein remapping + qkv split + adaLN swap.
|
||||
3. `ZImageGGUFAdapter` implements Z-Image qkv + ffn shard handling and linear qweight routing.
|
||||
4. `get_gguf_adapter(...)` strictly selects by model class/config; unsupported models raise an error (no fallback adapter).
|
||||
|
||||
Adapter paths:
|
||||
- Base: `vllm_omni/diffusion/model_loader/gguf_adapters/base.py`
|
||||
- Z-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/z_image.py`
|
||||
- Flux2-Klein: `vllm_omni/diffusion/model_loader/gguf_adapters/flux2_klein.py`
|
||||
|
||||
## User Usage (Offline)
|
||||
|
||||
### Baseline BF16
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model /workspace/models/black-forest-labs/FLUX.2-klein-4B \
|
||||
--prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \
|
||||
--height 768 \
|
||||
--width 1360 \
|
||||
--seed 42 \
|
||||
--cfg_scale 4.0 \
|
||||
--num_images_per_prompt 1 \
|
||||
--num_inference_steps 4 \
|
||||
--output outputs/flux2_klein_4b.png
|
||||
```
|
||||
|
||||
### Native GGUF (Transformer Only)
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model /workspace/models/black-forest-labs/FLUX.2-klein-4B \
|
||||
--gguf-model "/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf" \
|
||||
--quantization gguf \
|
||||
--prompt "a photo of a forest with mist swirling around the tree trunks. The word 'FLUX.2' is painted over it in big, red brush strokes with visible texture" \
|
||||
--height 768 \
|
||||
--width 1360 \
|
||||
--seed 42 \
|
||||
--cfg_scale 4.0 \
|
||||
--num_images_per_prompt 1 \
|
||||
--num_inference_steps 4 \
|
||||
--output outputs/flux2_klein_4b_gguf.png
|
||||
```
|
||||
|
||||
Notes for GGUF:
|
||||
1. Many GGUF repos do not ship `model_index.json` and configs. Use the base repo for `--model` and only pass the GGUF file via `--gguf-model`.
|
||||
2. `gguf_model` supports local path, `repo/file.gguf`, or `repo:quant_type`.
|
||||
|
||||
## User Usage (Online)
|
||||
|
||||
### Start Server (Native GGUF via CLI)
|
||||
```bash
|
||||
vllm serve /workspace/models/black-forest-labs/FLUX.2-klein-4B \
|
||||
--omni \
|
||||
--port 8000 \
|
||||
--quantization-config '{"method":"gguf","gguf_model":"/workspace/models/unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q8_0.gguf"}'
|
||||
```
|
||||
|
||||
### Online Request (Images API)
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/images/generations \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": "a dragon laying over the spine of the Green Mountains of Vermont",
|
||||
"size": "1024x1024",
|
||||
"seed": 42,
|
||||
"num_inference_steps": 4
|
||||
}'
|
||||
```
|
||||
@@ -7,6 +7,7 @@ vLLM-Omni supports quantization of DiT linear layers to reduce memory usage and
|
||||
| Method | Guide |
|
||||
|--------|-------|
|
||||
| FP8 | [FP8](fp8.md) |
|
||||
| GGUF | [GGUF](gguf.md) |
|
||||
|
||||
## Device Compatibility
|
||||
|
||||
|
||||
@@ -132,10 +132,18 @@ def parse_args() -> argparse.Namespace:
|
||||
"--quantization",
|
||||
type=str,
|
||||
default=None,
|
||||
choices=["fp8"],
|
||||
help="Quantization method for the transformer. "
|
||||
"Options: 'fp8' (FP8 W8A8 on Ada/Hopper, weight-only on older GPUs). "
|
||||
"Default: None (no quantization, uses BF16).",
|
||||
choices=["fp8", "gguf"],
|
||||
help=(
|
||||
"Quantization method for the transformer. "
|
||||
"Options: 'fp8' (FP8 W8A8), 'gguf' (GGUF quantized weights). "
|
||||
"Default: None (no quantization, uses BF16)."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--gguf-model",
|
||||
type=str,
|
||||
default=None,
|
||||
help=("GGUF file path or HF reference for transformer weights. Required when --quantization gguf is set."),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ignored-layers",
|
||||
@@ -265,7 +273,14 @@ def main():
|
||||
# ignored_layers is specified so the list flows through OmniDiffusionConfig
|
||||
quant_kwargs: dict[str, Any] = {}
|
||||
ignored_layers = [s.strip() for s in args.ignored_layers.split(",") if s.strip()] if args.ignored_layers else None
|
||||
if args.quantization and ignored_layers:
|
||||
if args.quantization == "gguf":
|
||||
if not args.gguf_model:
|
||||
raise ValueError("--gguf-model is required when --quantization gguf is set.")
|
||||
quant_kwargs["quantization_config"] = {
|
||||
"method": "gguf",
|
||||
"gguf_model": args.gguf_model,
|
||||
}
|
||||
elif args.quantization and ignored_layers:
|
||||
quant_kwargs["quantization_config"] = {
|
||||
"method": args.quantization,
|
||||
"ignored_layers": ignored_layers,
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from vllm_omni.diffusion.model_loader.diffusers_loader import DiffusersPipelineLoader
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.diffusion, pytest.mark.cpu]
|
||||
|
||||
|
||||
class _DummyPipelineModel(nn.Module):
|
||||
def __init__(self, *, source_prefix: str):
|
||||
super().__init__()
|
||||
self.transformer = nn.Linear(2, 2, bias=False)
|
||||
self.vae = nn.Linear(2, 2, bias=False)
|
||||
self.weights_sources = [
|
||||
DiffusersPipelineLoader.ComponentSource(
|
||||
model_or_path="dummy",
|
||||
subfolder="transformer",
|
||||
revision=None,
|
||||
prefix=source_prefix,
|
||||
fall_back_to_pt=True,
|
||||
)
|
||||
]
|
||||
|
||||
def load_weights(self, weights):
|
||||
params = dict(self.named_parameters())
|
||||
loaded: set[str] = set()
|
||||
for name, tensor in weights:
|
||||
if name not in params:
|
||||
continue
|
||||
params[name].data.copy_(tensor.to(dtype=params[name].dtype))
|
||||
loaded.add(name)
|
||||
return loaded
|
||||
|
||||
|
||||
def _make_loader_with_weights(weight_names: list[str]) -> DiffusersPipelineLoader:
|
||||
loader = object.__new__(DiffusersPipelineLoader)
|
||||
loader.counter_before_loading_weights = 0.0
|
||||
loader.counter_after_loading_weights = 0.0
|
||||
|
||||
def _iter_weights(_model):
|
||||
for name in weight_names:
|
||||
yield name, torch.zeros((2, 2))
|
||||
|
||||
loader.get_all_weights = _iter_weights # type: ignore[assignment]
|
||||
return loader
|
||||
|
||||
|
||||
def test_strict_check_only_validates_source_prefix_parameters():
|
||||
model = _DummyPipelineModel(source_prefix="transformer.")
|
||||
loader = _make_loader_with_weights(["transformer.weight"])
|
||||
|
||||
# Should not require VAE parameters because they are outside weights_sources.
|
||||
loader.load_weights(model)
|
||||
|
||||
|
||||
def test_strict_check_raises_when_source_parameters_are_missing():
|
||||
model = _DummyPipelineModel(source_prefix="transformer.")
|
||||
loader = _make_loader_with_weights([])
|
||||
|
||||
with pytest.raises(ValueError, match="transformer.weight"):
|
||||
loader.load_weights(model)
|
||||
|
||||
|
||||
def test_empty_source_prefix_keeps_full_model_strict_check():
|
||||
model = _DummyPipelineModel(source_prefix="")
|
||||
loader = _make_loader_with_weights(["transformer.weight"])
|
||||
|
||||
with pytest.raises(ValueError, match="vae.weight"):
|
||||
loader.load_weights(model)
|
||||
@@ -6,7 +6,7 @@ import os
|
||||
import random
|
||||
from collections.abc import Callable, Mapping
|
||||
from dataclasses import dataclass, field, fields
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import torch
|
||||
from pydantic import model_validator
|
||||
@@ -20,6 +20,12 @@ from vllm_omni.diffusion.quantization import (
|
||||
)
|
||||
from vllm_omni.diffusion.utils.network_utils import is_port_available
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm_omni.diffusion.quantization import DiffusionQuantizationConfig
|
||||
|
||||
# Import after TYPE_CHECKING to avoid circular imports at runtime
|
||||
# The actual import is deferred to __post_init__ to avoid import order issues
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
|
||||
@@ -527,8 +533,12 @@ class OmniDiffusionConfig:
|
||||
# If it's neither dict nor DiffusionCacheConfig, convert to empty config
|
||||
self.cache_config = DiffusionCacheConfig()
|
||||
|
||||
# Convert quantization config
|
||||
# Convert quantization config (deferred import to avoid circular imports)
|
||||
if self.quantization is not None or self.quantization_config is not None:
|
||||
from vllm_omni.diffusion.quantization import (
|
||||
DiffusionQuantizationConfig,
|
||||
)
|
||||
|
||||
# Handle dict or DictConfig (from OmegaConf) - use Mapping for broader compatibility
|
||||
if isinstance(self.quantization_config, Mapping):
|
||||
# Convert DictConfig to dict if needed (OmegaConf compatibility)
|
||||
|
||||
@@ -9,12 +9,14 @@ from pathlib import Path
|
||||
from typing import cast
|
||||
|
||||
import torch
|
||||
from huggingface_hub import hf_hub_download
|
||||
from torch import nn
|
||||
from vllm.config import ModelConfig
|
||||
from vllm.config.load import LoadConfig
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.layers.quantization.base_config import QuantizeMethodBase
|
||||
from vllm.model_executor.model_loader.weight_utils import (
|
||||
download_gguf,
|
||||
download_safetensors_index_file_from_hf,
|
||||
download_weights_from_hf,
|
||||
filter_duplicate_safetensors_files,
|
||||
@@ -27,6 +29,7 @@ from vllm.utils.torch_utils import set_default_torch_dtype
|
||||
|
||||
from vllm_omni.diffusion.data import OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.distributed.hsdp import HSDPInferenceConfig
|
||||
from vllm_omni.diffusion.model_loader.gguf_adapters import get_gguf_adapter
|
||||
from vllm_omni.diffusion.registry import initialize_model
|
||||
|
||||
logger = init_logger(__name__)
|
||||
@@ -194,13 +197,36 @@ class DiffusersPipelineLoader:
|
||||
self,
|
||||
model: nn.Module,
|
||||
) -> Generator[tuple[str, torch.Tensor], None, None]:
|
||||
sources = cast(
|
||||
Iterable[DiffusersPipelineLoader.ComponentSource],
|
||||
getattr(model, "weights_sources", ()),
|
||||
)
|
||||
sources = self._get_weight_sources(model)
|
||||
for source in sources:
|
||||
yield from self._get_weights_iterator(source)
|
||||
|
||||
def _get_weight_sources(self, model: nn.Module) -> tuple["ComponentSource", ...]:
|
||||
return tuple(
|
||||
cast(
|
||||
Iterable[DiffusersPipelineLoader.ComponentSource],
|
||||
getattr(model, "weights_sources", ()),
|
||||
)
|
||||
)
|
||||
|
||||
def _get_expected_parameter_names(self, model: nn.Module) -> set[str]:
|
||||
"""Return parameter names that should be covered by strict load checks."""
|
||||
all_parameter_names = {name for name, _ in model.named_parameters()}
|
||||
sources = self._get_weight_sources(model)
|
||||
|
||||
# Keep strict behavior if no source metadata exists.
|
||||
if not sources:
|
||||
return all_parameter_names
|
||||
|
||||
# Empty prefix means "root" source, i.e. entire model should be covered.
|
||||
if any(source.prefix == "" for source in sources):
|
||||
return all_parameter_names
|
||||
|
||||
source_prefixes = tuple(source.prefix for source in sources if source.prefix)
|
||||
if not source_prefixes:
|
||||
return all_parameter_names
|
||||
return {name for name in all_parameter_names if name.startswith(source_prefixes)}
|
||||
|
||||
def download_model(self, model_config: ModelConfig) -> None:
|
||||
self._prepare_weights(
|
||||
model_name_or_path=model_config.model,
|
||||
@@ -232,8 +258,11 @@ class DiffusersPipelineLoader:
|
||||
model_cls = resolve_obj_by_qualname(custom_pipeline_name)
|
||||
model = model_cls(od_config=od_config)
|
||||
logger.debug("Loading weights on %s ...", load_device)
|
||||
# Quantization does not happen in `load_weights` but after it
|
||||
self.load_weights(model)
|
||||
if self._is_gguf_quantization(od_config):
|
||||
self._load_weights_with_gguf(model, od_config)
|
||||
else:
|
||||
# Quantization does not happen in `load_weights` but after it
|
||||
self.load_weights(model)
|
||||
|
||||
# Process weights after loading for quantization (e.g., FP8 online quantization)
|
||||
# This is needed for vLLM's quantization methods that need to transform weights
|
||||
@@ -265,7 +294,7 @@ class DiffusersPipelineLoader:
|
||||
module.to(module_device)
|
||||
|
||||
def load_weights(self, model: nn.Module) -> None:
|
||||
weights_to_load = {name for name, _ in model.named_parameters()}
|
||||
weights_to_load = self._get_expected_parameter_names(model)
|
||||
loaded_weights = model.load_weights(self.get_all_weights(model))
|
||||
|
||||
self.counter_after_loading_weights = time.perf_counter()
|
||||
@@ -278,12 +307,122 @@ class DiffusersPipelineLoader:
|
||||
# We only enable strict check for non-quantized models
|
||||
# that have loaded weights tracking currently.
|
||||
if loaded_weights is not None:
|
||||
_ = weights_to_load - loaded_weights
|
||||
# if weights_not_loaded:
|
||||
# raise ValueError(
|
||||
# "Following weights were not initialized from "
|
||||
# f"checkpoint: {weights_not_loaded}"
|
||||
# )
|
||||
weights_not_loaded = weights_to_load - loaded_weights
|
||||
if weights_not_loaded:
|
||||
raise ValueError(f"Following weights were not initialized from checkpoint: {weights_not_loaded}")
|
||||
|
||||
def _is_gguf_quantization(self, od_config: OmniDiffusionConfig) -> bool:
|
||||
quant_config = od_config.quantization_config
|
||||
if quant_config is None:
|
||||
return False
|
||||
# Fast path: mapping-style config (e.g., DictConfig)
|
||||
if isinstance(quant_config, dict):
|
||||
method = str(quant_config.get("method", "")).lower()
|
||||
if method != "gguf":
|
||||
return False
|
||||
gguf_model = quant_config.get("gguf_model")
|
||||
if not gguf_model:
|
||||
raise ValueError("GGUF quantization requires quantization_config.gguf_model")
|
||||
return True
|
||||
|
||||
# Normal path: DiffusionQuantizationConfig
|
||||
if not hasattr(quant_config, "get_name"):
|
||||
# Fallback: if it carries gguf_model, treat as GGUF
|
||||
gguf_model = getattr(quant_config, "gguf_model", None)
|
||||
return bool(gguf_model)
|
||||
is_gguf = quant_config.get_name() == "gguf"
|
||||
if not is_gguf:
|
||||
return False
|
||||
gguf_model = getattr(quant_config, "gguf_model", None)
|
||||
if gguf_model is None:
|
||||
raise ValueError("GGUF quantization requires quantization_config.gguf_model")
|
||||
return True
|
||||
|
||||
def _is_transformer_source(self, source: "ComponentSource") -> bool:
|
||||
if source.subfolder == "transformer":
|
||||
return True
|
||||
return source.prefix.startswith("transformer.")
|
||||
|
||||
def _get_model_loadable_names(self, model: nn.Module) -> set[str]:
|
||||
# Avoid model.state_dict() here because GGUF uses UninitializedParameter
|
||||
# which raises during detach(). Collect names directly.
|
||||
names = {name for name, _ in model.named_parameters()}
|
||||
names.update(name for name, _ in model.named_buffers())
|
||||
return names
|
||||
|
||||
def _resolve_gguf_model_path(self, gguf_model: str, revision: str | None) -> str:
|
||||
if os.path.isfile(gguf_model):
|
||||
return gguf_model
|
||||
# repo_id/filename.gguf
|
||||
if "/" in gguf_model and gguf_model.endswith(".gguf"):
|
||||
repo_id, filename = gguf_model.rsplit("/", 1)
|
||||
return hf_hub_download(
|
||||
repo_id=repo_id,
|
||||
filename=filename,
|
||||
revision=revision,
|
||||
cache_dir=self.load_config.download_dir,
|
||||
)
|
||||
# repo_id:quant_type
|
||||
if "/" in gguf_model and ":" in gguf_model:
|
||||
repo_id, quant_type = gguf_model.rsplit(":", 1)
|
||||
return download_gguf(
|
||||
repo_id,
|
||||
quant_type,
|
||||
cache_dir=self.load_config.download_dir,
|
||||
revision=revision,
|
||||
ignore_patterns=self.load_config.ignore_patterns,
|
||||
)
|
||||
raise ValueError(
|
||||
f"Unrecognized GGUF reference: {gguf_model!r} (expected local file, "
|
||||
"<repo_id>/<filename>.gguf, or <repo_id>:<quant_type>)"
|
||||
)
|
||||
|
||||
def _get_gguf_weights_iterator(
|
||||
self,
|
||||
source: "ComponentSource",
|
||||
model: nn.Module,
|
||||
od_config: OmniDiffusionConfig,
|
||||
) -> Generator[tuple[str, torch.Tensor], None, None]:
|
||||
quant_config = od_config.quantization_config
|
||||
gguf_model = getattr(quant_config, "gguf_model", None)
|
||||
if gguf_model is None:
|
||||
raise ValueError("GGUF quantization requires quantization_config.gguf_model")
|
||||
gguf_file = self._resolve_gguf_model_path(gguf_model, od_config.revision)
|
||||
adapter = get_gguf_adapter(gguf_file, model, source, od_config)
|
||||
weights_iter = adapter.weights_iterator()
|
||||
return ((source.prefix + name, tensor) for (name, tensor) in weights_iter)
|
||||
|
||||
def _load_weights_with_gguf(self, model: nn.Module, od_config: OmniDiffusionConfig) -> set[str]:
|
||||
sources = self._get_weight_sources(model)
|
||||
loaded: set[str] = set()
|
||||
loadable_names: set[str] | None = None
|
||||
|
||||
for source in sources:
|
||||
if self._is_transformer_source(source):
|
||||
loaded |= model.load_weights(self._get_gguf_weights_iterator(source, model, od_config))
|
||||
|
||||
# GGUF checkpoints can be transformer-only or partially quantized.
|
||||
# Only fall back to HF if this source still has missing loadable weights.
|
||||
loadable_names = loadable_names or self._get_model_loadable_names(model)
|
||||
has_missing_for_source = any(
|
||||
name.startswith(source.prefix) and name not in loaded for name in loadable_names
|
||||
)
|
||||
if not has_missing_for_source:
|
||||
continue
|
||||
|
||||
hf_iter = self._get_weights_iterator(source)
|
||||
hf_iter = (
|
||||
(name, tensor) for (name, tensor) in hf_iter if name in loadable_names and name not in loaded
|
||||
)
|
||||
loaded |= model.load_weights(hf_iter)
|
||||
else:
|
||||
loaded |= model.load_weights(self._get_weights_iterator(source))
|
||||
|
||||
weights_to_load = self._get_expected_parameter_names(model)
|
||||
weights_not_loaded = weights_to_load - loaded
|
||||
if weights_not_loaded:
|
||||
raise ValueError(f"Following weights were not initialized from checkpoint: {weights_not_loaded}")
|
||||
return loaded
|
||||
|
||||
def _load_model_with_hsdp(
|
||||
self,
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from .base import GGUFAdapter
|
||||
from .flux2_klein import Flux2KleinGGUFAdapter
|
||||
from .z_image import ZImageGGUFAdapter
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm_omni.diffusion.data import OmniDiffusionConfig
|
||||
from vllm_omni.diffusion.model_loader.diffusers_loader import (
|
||||
DiffusersPipelineLoader,
|
||||
)
|
||||
|
||||
|
||||
def get_gguf_adapter(
|
||||
gguf_file: str,
|
||||
model: torch.nn.Module,
|
||||
source: DiffusersPipelineLoader.ComponentSource,
|
||||
od_config: OmniDiffusionConfig,
|
||||
) -> GGUFAdapter:
|
||||
adapter_classes = (ZImageGGUFAdapter, Flux2KleinGGUFAdapter)
|
||||
for adapter_cls in adapter_classes:
|
||||
if adapter_cls.is_compatible(od_config, model, source):
|
||||
return adapter_cls(gguf_file, model, source, od_config)
|
||||
model_type = None
|
||||
if od_config.tf_model_config is not None:
|
||||
model_type = od_config.tf_model_config.get("model_type")
|
||||
supported = ", ".join(cls.__name__ for cls in adapter_classes)
|
||||
raise ValueError(
|
||||
"No GGUF adapter matched diffusion model "
|
||||
f"(model_class_name={od_config.model_class_name!r}, model_type={model_type!r}). "
|
||||
f"Supported adapters: {supported}."
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GGUFAdapter",
|
||||
"Flux2KleinGGUFAdapter",
|
||||
"ZImageGGUFAdapter",
|
||||
"get_gguf_adapter",
|
||||
]
|
||||
@@ -0,0 +1,245 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Generator
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import gguf
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
|
||||
@dataclass
|
||||
class MappedTensor:
|
||||
name: str
|
||||
tensor: Any
|
||||
tensor_type: Any
|
||||
row_slice: slice | None = None
|
||||
swap_scale_shift: bool = False
|
||||
|
||||
|
||||
class GGUFAdapter(ABC):
|
||||
"""Base class for model-specific GGUF adapters."""
|
||||
|
||||
_include_qkv_virtuals: bool = False
|
||||
_include_add_kv_proj_virtuals: bool = False
|
||||
_include_to_out_virtuals: bool = False
|
||||
_include_w13_virtuals: bool = False
|
||||
_shard_tokens: tuple[str, ...] = ()
|
||||
_prefer_exact_qweight: bool = True
|
||||
|
||||
def __init__(self, gguf_file: str, model: torch.nn.Module, source, od_config) -> None:
|
||||
self.gguf_file = gguf_file
|
||||
self.model = model
|
||||
self.source = source
|
||||
self.od_config = od_config
|
||||
|
||||
@staticmethod
|
||||
def is_compatible(od_config, model: torch.nn.Module, source) -> bool:
|
||||
return False
|
||||
|
||||
@abstractmethod
|
||||
def weights_iterator(self) -> Generator[tuple[str, torch.Tensor], None, None]:
|
||||
raise NotImplementedError
|
||||
|
||||
def _get_target_module(self) -> torch.nn.Module:
|
||||
prefix = getattr(self.source, "prefix", "")
|
||||
return self.model.get_submodule(prefix.rstrip(".")) if prefix else self.model
|
||||
|
||||
def _build_allowed_names(self) -> set[str]:
|
||||
target = self._get_target_module()
|
||||
allowed = {name for name, _ in target.named_parameters()}
|
||||
allowed.update(name for name, _ in target.named_buffers())
|
||||
for name in list(allowed):
|
||||
if name.endswith(".qweight"):
|
||||
allowed.add(name.replace(".qweight", ".weight"))
|
||||
elif name.endswith(".qweight_type"):
|
||||
allowed.add(name.replace(".qweight_type", ".weight"))
|
||||
|
||||
virtual_names = set()
|
||||
for name in allowed:
|
||||
if self._include_qkv_virtuals and ".to_qkv." in name:
|
||||
virtual_names.add(name.replace(".to_qkv.", ".to_q."))
|
||||
virtual_names.add(name.replace(".to_qkv.", ".to_k."))
|
||||
virtual_names.add(name.replace(".to_qkv.", ".to_v."))
|
||||
if self._include_add_kv_proj_virtuals and ".add_kv_proj." in name:
|
||||
virtual_names.add(name.replace(".add_kv_proj.", ".add_q_proj."))
|
||||
virtual_names.add(name.replace(".add_kv_proj.", ".add_k_proj."))
|
||||
virtual_names.add(name.replace(".add_kv_proj.", ".add_v_proj."))
|
||||
if self._include_w13_virtuals and ".w13." in name:
|
||||
virtual_names.add(name.replace(".w13.", ".w1."))
|
||||
virtual_names.add(name.replace(".w13.", ".w3."))
|
||||
if self._include_to_out_virtuals and ".to_out." in name:
|
||||
virtual_names.add(name.replace(".to_out.", ".to_out.0."))
|
||||
allowed.update(virtual_names)
|
||||
return allowed
|
||||
|
||||
def _build_param_names(self) -> set[str]:
|
||||
target = self._get_target_module()
|
||||
return {name for name, _ in target.named_parameters()}
|
||||
|
||||
def _resolve_linear_qweight(self, name: str, param_names: set[str]) -> str | None:
|
||||
if not name.endswith(".weight"):
|
||||
return None
|
||||
if self._prefer_exact_qweight:
|
||||
candidate = name.replace(".weight", ".qweight")
|
||||
if candidate in param_names:
|
||||
return candidate
|
||||
if ".to_out.0." in name:
|
||||
alt_name = name.replace(".to_out.0.", ".to_out.")
|
||||
candidate = alt_name.replace(".weight", ".qweight")
|
||||
if candidate in param_names:
|
||||
return candidate
|
||||
name = alt_name
|
||||
for shard_token in self._shard_tokens:
|
||||
if shard_token in name:
|
||||
return name.replace(".weight", ".qweight")
|
||||
candidate = name.replace(".weight", ".qweight")
|
||||
if candidate in param_names:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _build_gguf_name_map(self) -> dict[str, str]:
|
||||
def resolve_model_type() -> str:
|
||||
cfg = self.od_config.tf_model_config
|
||||
model_type = None
|
||||
if cfg is not None:
|
||||
model_type = cfg.get("model_type")
|
||||
if model_type:
|
||||
return model_type
|
||||
model_class = self.od_config.model_class_name or ""
|
||||
if model_class.startswith("QwenImage"):
|
||||
return "qwen_image"
|
||||
if model_class.startswith("Flux2"):
|
||||
return "flux"
|
||||
raise ValueError("Cannot infer gguf model_type for diffusion model.")
|
||||
|
||||
def resolve_arch(model_type: str):
|
||||
for key, value in gguf.MODEL_ARCH_NAMES.items():
|
||||
if value == model_type:
|
||||
return key
|
||||
raise RuntimeError(f"Unknown gguf model_type: {model_type}")
|
||||
|
||||
def resolve_num_layers(target_module: torch.nn.Module) -> int:
|
||||
if hasattr(target_module, "transformer_blocks"):
|
||||
return len(getattr(target_module, "transformer_blocks"))
|
||||
if hasattr(target_module, "double_blocks"):
|
||||
return len(getattr(target_module, "double_blocks"))
|
||||
cfg = self.od_config.tf_model_config
|
||||
if cfg is not None:
|
||||
for key in ("num_hidden_layers", "num_layers", "n_layers"):
|
||||
value = cfg.get(key)
|
||||
if isinstance(value, int) and value > 0:
|
||||
return value
|
||||
raise ValueError("Cannot infer gguf num_layers for diffusion model.")
|
||||
|
||||
def get_target_module(root: torch.nn.Module, prefix: str) -> torch.nn.Module:
|
||||
if not prefix:
|
||||
return root
|
||||
prefix = prefix.rstrip(".")
|
||||
if hasattr(root, "get_submodule"):
|
||||
return root.get_submodule(prefix)
|
||||
current = root
|
||||
for part in prefix.split("."):
|
||||
current = getattr(current, part)
|
||||
return current
|
||||
|
||||
def split_name(name: str) -> tuple[str, str]:
|
||||
if name.endswith("_weight"):
|
||||
return name[:-7], "weight"
|
||||
if "." in name:
|
||||
base, suffix = name.rsplit(".", 1)
|
||||
return base, suffix
|
||||
return name, ""
|
||||
|
||||
reader = gguf.GGUFReader(self.gguf_file)
|
||||
gguf_tensor_names = {tensor.name for tensor in reader.tensors}
|
||||
|
||||
model_type = resolve_model_type()
|
||||
arch = resolve_arch(model_type)
|
||||
target_module = get_target_module(self.model, self.source.prefix)
|
||||
num_layers = resolve_num_layers(target_module)
|
||||
name_map = gguf.get_tensor_name_map(arch, num_layers)
|
||||
|
||||
gguf_to_model_map: dict[str, str] = {}
|
||||
for name, _ in target_module.named_parameters():
|
||||
base_name, suffix = split_name(name)
|
||||
gguf_base = name_map.get_name(base_name)
|
||||
if gguf_base is None:
|
||||
continue
|
||||
candidates = []
|
||||
if suffix:
|
||||
candidates.append(f"{gguf_base}.{suffix}")
|
||||
if suffix == "weight":
|
||||
candidates.append(f"{gguf_base}.scale")
|
||||
else:
|
||||
candidates.append(gguf_base)
|
||||
gguf_name = next((c for c in candidates if c in gguf_tensor_names), None)
|
||||
if gguf_name is None:
|
||||
continue
|
||||
gguf_to_model_map[gguf_name] = name
|
||||
|
||||
for name, _ in target_module.named_buffers():
|
||||
base_name, suffix = split_name(name)
|
||||
gguf_base = name_map.get_name(base_name)
|
||||
if gguf_base is None:
|
||||
continue
|
||||
candidates = []
|
||||
if suffix:
|
||||
candidates.append(f"{gguf_base}.{suffix}")
|
||||
if suffix == "weight":
|
||||
candidates.append(f"{gguf_base}.scale")
|
||||
else:
|
||||
candidates.append(gguf_base)
|
||||
gguf_name = next((c for c in candidates if c in gguf_tensor_names), None)
|
||||
if gguf_name is None:
|
||||
continue
|
||||
gguf_to_model_map[gguf_name] = name
|
||||
|
||||
if not gguf_to_model_map:
|
||||
raise RuntimeError(f"No GGUF tensors were mapped for model_class_name={self.od_config.model_class_name!r}.")
|
||||
return gguf_to_model_map
|
||||
|
||||
|
||||
# FIXME(Isotr0py): Sync implemnentation with upstream vLLM?
|
||||
def gguf_quant_weights_iterator(gguf_file: str) -> Generator[tuple[str, torch.Tensor]]:
|
||||
"""
|
||||
Iterate over the quant weights in the model gguf files and convert
|
||||
them to torch tensors.
|
||||
Be careful of the order of yielding weight types and weights data,
|
||||
we have to yield all weight types first before yielding any weights.
|
||||
Otherwise it would cause issue when loading weights with for packed
|
||||
layer with different quant types.
|
||||
"""
|
||||
|
||||
reader = gguf.GGUFReader(gguf_file)
|
||||
|
||||
for tensor in reader.tensors:
|
||||
weight_type = tensor.tensor_type
|
||||
name = tensor.name
|
||||
|
||||
if weight_type.name not in ("F32", "F16"):
|
||||
weight_type_name = name.replace("weight", "qweight_type")
|
||||
weight_type = torch.tensor(weight_type)
|
||||
yield weight_type_name, weight_type
|
||||
|
||||
for tensor in reader.tensors:
|
||||
weight = tensor.data
|
||||
weight_type = tensor.tensor_type
|
||||
name = tensor.name
|
||||
if weight_type.name not in ("F32", "F16"):
|
||||
name = name.replace("weight", "qweight")
|
||||
if weight_type.name == "BF16" and tensor.data.dtype == np.uint8:
|
||||
# BF16 is currently the only "quantization" type that isn't
|
||||
# actually quantized but is read as a raw byte tensor.
|
||||
# Reinterpret as `torch.bfloat16` tensor.
|
||||
weight = weight.view(np.uint16)
|
||||
if reader.byte_order == "S":
|
||||
# GGUF endianness != system endianness
|
||||
weight = weight.byteswap()
|
||||
param = torch.tensor(weight).view(torch.bfloat16)
|
||||
else:
|
||||
param = torch.tensor(weight)
|
||||
yield name, param
|
||||
@@ -0,0 +1,98 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
import torch
|
||||
from vllm.model_executor.models.utils import WeightsMapper
|
||||
|
||||
from .base import GGUFAdapter, gguf_quant_weights_iterator
|
||||
|
||||
FLUX2_TRANSFORMER_KEYS_RENAME_DICT = {
|
||||
"single_blocks.": "single_transformer_blocks.",
|
||||
# Image and text input projections
|
||||
"img_in": "x_embedder",
|
||||
"txt_in": "context_embedder",
|
||||
# Timestep and guidance embeddings
|
||||
"time_in.in_layer": "time_guidance_embed.timestep_embedder.linear_1",
|
||||
"time_in.out_layer": "time_guidance_embed.timestep_embedder.linear_2",
|
||||
"guidance_in.in_layer": "time_guidance_embed.guidance_embedder.linear_1",
|
||||
"guidance_in.out_layer": "time_guidance_embed.guidance_embedder.linear_2",
|
||||
# Modulation parameters
|
||||
"double_stream_modulation_img.lin": "double_stream_modulation_img.linear",
|
||||
"double_stream_modulation_txt.lin": "double_stream_modulation_txt.linear",
|
||||
"single_stream_modulation.lin": "single_stream_modulation.linear",
|
||||
# Final output layer
|
||||
# "final_layer.adaLN_modulation.1": "norm_out.linear", # Handle separately since we need to swap mod params
|
||||
"final_layer.linear": "proj_out",
|
||||
}
|
||||
|
||||
FLUX2_TRANSFORMER_ADA_LAYER_NORM_KEY_MAP = {
|
||||
"final_layer.adaLN_modulation.1": "norm_out.linear",
|
||||
}
|
||||
|
||||
FLUX2_TRANSFORMER_DOUBLE_BLOCK_KEY_MAP = {
|
||||
"double_blocks.": "transformer_blocks.",
|
||||
# Handle fused QKV projections separately as we need to break into Q, K, V projections
|
||||
"img_attn.norm.query_norm": "attn.norm_q",
|
||||
"img_attn.norm.key_norm": "attn.norm_k",
|
||||
"img_attn.proj": "attn.to_out.0",
|
||||
"img_mlp.0": "ff.linear_in",
|
||||
"img_mlp.2": "ff.linear_out",
|
||||
"txt_attn.norm.query_norm": "attn.norm_added_q",
|
||||
"txt_attn.norm.key_norm": "attn.norm_added_k",
|
||||
"txt_attn.proj": "attn.to_add_out",
|
||||
"txt_mlp.0": "ff_context.linear_in",
|
||||
"txt_mlp.2": "ff_context.linear_out",
|
||||
# Additional for fuse qkv
|
||||
"img_attn.qkv": "attn.to_qkv",
|
||||
"txt_attn.qkv": "attn.add_kv_proj",
|
||||
}
|
||||
|
||||
FLUX2_TRANSFORMER_SINGLE_BLOCK_KEY_MAP = {
|
||||
"linear1": "attn.to_qkv_mlp_proj",
|
||||
"linear2": "attn.to_out",
|
||||
"norm.query_norm": "attn.norm_q",
|
||||
"norm.key_norm": "attn.norm_k",
|
||||
}
|
||||
|
||||
|
||||
class Flux2KleinGGUFAdapter(GGUFAdapter):
|
||||
"""GGUF adapter for Flux2-Klein models with qkv splitting and adaLN swap."""
|
||||
|
||||
@staticmethod
|
||||
def is_compatible(od_config, model: torch.nn.Module, source) -> bool:
|
||||
model_class = od_config.model_class_name or ""
|
||||
if model_class.startswith("Flux2"):
|
||||
return True
|
||||
cfg = od_config.tf_model_config
|
||||
if cfg is not None:
|
||||
model_type = str(cfg.get("model_type", "")).lower()
|
||||
if model_type.startswith("flux"):
|
||||
return True
|
||||
return False
|
||||
|
||||
gguf_to_hf_mapper = WeightsMapper(
|
||||
# double_stream_modulation
|
||||
orig_to_new_prefix=FLUX2_TRANSFORMER_KEYS_RENAME_DICT | FLUX2_TRANSFORMER_ADA_LAYER_NORM_KEY_MAP,
|
||||
orig_to_new_substr=FLUX2_TRANSFORMER_DOUBLE_BLOCK_KEY_MAP | FLUX2_TRANSFORMER_SINGLE_BLOCK_KEY_MAP,
|
||||
)
|
||||
|
||||
def weights_iterator(self) -> Iterable[tuple[str, torch.Tensor]]:
|
||||
def custom_weights_adapter(
|
||||
weights: Iterable[tuple[str, torch.Tensor]],
|
||||
) -> Iterable[tuple[str, torch.Tensor]]:
|
||||
for name, weight in weights:
|
||||
# Handle the special case for adaLN modulation parameters that require swapping shift and scale
|
||||
if name.endswith(".scale"):
|
||||
name = name.replace(".scale", ".weight")
|
||||
if name == "norm_out.linear.weight":
|
||||
shift, scale = weight.chunk(2, dim=0)
|
||||
weight = torch.cat([scale, shift], dim=0)
|
||||
yield name, weight
|
||||
else:
|
||||
yield name, weight
|
||||
|
||||
weights = gguf_quant_weights_iterator(self.gguf_file)
|
||||
weights = self.gguf_to_hf_mapper.apply(weights)
|
||||
yield from custom_weights_adapter(weights)
|
||||
@@ -0,0 +1,43 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable
|
||||
|
||||
import torch
|
||||
from vllm.model_executor.models.utils import WeightsMapper
|
||||
|
||||
from .base import GGUFAdapter, gguf_quant_weights_iterator
|
||||
|
||||
Z_IMAGE_KEYS_RENAME_DICT = {
|
||||
"final_layer.": "all_final_layer.2-1.",
|
||||
"x_embedder.": "all_x_embedder.2-1.",
|
||||
".attention.qkv": ".attention.to_qkv",
|
||||
".attention.k_norm": ".attention.norm_k",
|
||||
".attention.q_norm": ".attention.norm_q",
|
||||
".attention.out": ".attention.to_out.0",
|
||||
"model.diffusion_model.": "",
|
||||
}
|
||||
|
||||
|
||||
class ZImageGGUFAdapter(GGUFAdapter):
|
||||
"""GGUF adapter for Z-Image models with QKV/FFN shard support."""
|
||||
|
||||
@staticmethod
|
||||
def is_compatible(od_config, model: torch.nn.Module, source) -> bool:
|
||||
model_class = od_config.model_class_name or ""
|
||||
if model_class.startswith("ZImage"):
|
||||
return True
|
||||
cfg = od_config.tf_model_config
|
||||
if cfg is not None:
|
||||
model_type = str(cfg.get("model_type", "")).lower()
|
||||
if model_type in {"z_image", "zimage", "z-image"}:
|
||||
return True
|
||||
return False
|
||||
|
||||
gguf_to_hf_mapper = WeightsMapper(
|
||||
orig_to_new_substr=Z_IMAGE_KEYS_RENAME_DICT,
|
||||
)
|
||||
|
||||
def weights_iterator(self) -> Iterable[tuple[str, torch.Tensor]]:
|
||||
weights = gguf_quant_weights_iterator(self.gguf_file)
|
||||
yield from self.gguf_to_hf_mapper.apply(weights)
|
||||
@@ -17,7 +17,7 @@
|
||||
|
||||
from collections.abc import Iterable
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
@@ -41,6 +41,9 @@ from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
|
||||
|
||||
|
||||
class Flux2SwiGLU(nn.Module):
|
||||
"""SwiGLU activation used by Flux2."""
|
||||
@@ -62,6 +65,7 @@ class Flux2FeedForward(nn.Module):
|
||||
mult: float = 3.0,
|
||||
inner_dim: int | None = None,
|
||||
bias: bool = False,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
if inner_dim is None:
|
||||
@@ -73,6 +77,7 @@ class Flux2FeedForward(nn.Module):
|
||||
[inner_dim, inner_dim],
|
||||
bias=bias,
|
||||
return_bias=False,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
self.act_fn = Flux2SwiGLU()
|
||||
self.linear_out = RowParallelLinear(
|
||||
@@ -81,6 +86,7 @@ class Flux2FeedForward(nn.Module):
|
||||
bias=bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
@@ -103,6 +109,7 @@ class Flux2Attention(nn.Module):
|
||||
eps: float = 1e-5,
|
||||
out_dim: int = None,
|
||||
elementwise_affine: bool = True,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.head_dim = dim_head
|
||||
@@ -118,6 +125,7 @@ class Flux2Attention(nn.Module):
|
||||
head_size=self.head_dim,
|
||||
total_num_heads=self.heads,
|
||||
bias=bias,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
self.query_num_heads = self.to_qkv.num_heads
|
||||
self.kv_num_heads = self.to_qkv.num_kv_heads
|
||||
@@ -133,6 +141,7 @@ class Flux2Attention(nn.Module):
|
||||
bias=out_bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
quant_config=quant_config,
|
||||
),
|
||||
nn.Dropout(dropout),
|
||||
]
|
||||
@@ -146,6 +155,7 @@ class Flux2Attention(nn.Module):
|
||||
head_size=self.head_dim,
|
||||
total_num_heads=self.heads,
|
||||
bias=added_proj_bias,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
self.add_query_num_heads = self.add_kv_proj.num_heads
|
||||
self.add_kv_num_heads = self.add_kv_proj.num_kv_heads
|
||||
@@ -155,6 +165,7 @@ class Flux2Attention(nn.Module):
|
||||
bias=out_bias,
|
||||
input_is_parallel=True,
|
||||
return_bias=False,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
|
||||
self.rope = RotaryEmbedding(is_neox_style=False)
|
||||
@@ -251,6 +262,7 @@ class Flux2ParallelSelfAttention(nn.Module):
|
||||
elementwise_affine: bool = True,
|
||||
mlp_ratio: float = 4.0,
|
||||
mlp_mult_factor: int = 2,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.head_dim = dim_head
|
||||
@@ -269,6 +281,7 @@ class Flux2ParallelSelfAttention(nn.Module):
|
||||
self.inner_dim * 3 + self.mlp_hidden_dim * self.mlp_mult_factor,
|
||||
bias=bias,
|
||||
gather_output=True,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
self.mlp_act_fn = Flux2SwiGLU()
|
||||
|
||||
@@ -280,6 +293,7 @@ class Flux2ParallelSelfAttention(nn.Module):
|
||||
self.out_dim,
|
||||
bias=out_bias,
|
||||
gather_output=True,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
self.rope = RotaryEmbedding(is_neox_style=False)
|
||||
self.attn = Attention(
|
||||
@@ -342,6 +356,7 @@ class Flux2SingleTransformerBlock(nn.Module):
|
||||
mlp_ratio: float = 3.0,
|
||||
eps: float = 1e-6,
|
||||
bias: bool = False,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
@@ -355,6 +370,7 @@ class Flux2SingleTransformerBlock(nn.Module):
|
||||
eps=eps,
|
||||
mlp_ratio=mlp_ratio,
|
||||
mlp_mult_factor=2,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
|
||||
def forward(
|
||||
@@ -402,6 +418,7 @@ class Flux2TransformerBlock(nn.Module):
|
||||
mlp_ratio: float = 3.0,
|
||||
eps: float = 1e-6,
|
||||
bias: bool = False,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.norm1 = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
@@ -417,13 +434,20 @@ class Flux2TransformerBlock(nn.Module):
|
||||
added_proj_bias=bias,
|
||||
out_bias=bias,
|
||||
eps=eps,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
|
||||
self.norm2 = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.ff = Flux2FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio, bias=bias)
|
||||
self.ff = Flux2FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio, bias=bias, quant_config=quant_config)
|
||||
|
||||
self.norm2_context = nn.LayerNorm(dim, elementwise_affine=False, eps=eps)
|
||||
self.ff_context = Flux2FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio, bias=bias)
|
||||
self.ff_context = Flux2FeedForward(
|
||||
dim=dim,
|
||||
dim_out=dim,
|
||||
mult=mlp_ratio,
|
||||
bias=bias,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
|
||||
def forward(
|
||||
self,
|
||||
@@ -580,6 +604,7 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
rope_theta: int = 2000,
|
||||
eps: float = 1e-6,
|
||||
guidance_embeds: bool = True,
|
||||
quant_config: "QuantizationConfig | None" = None,
|
||||
):
|
||||
super().__init__()
|
||||
self.out_channels = out_channels or in_channels
|
||||
@@ -625,6 +650,7 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
mlp_ratio=mlp_ratio,
|
||||
eps=eps,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
for _ in range(num_layers)
|
||||
]
|
||||
@@ -639,6 +665,7 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
mlp_ratio=mlp_ratio,
|
||||
eps=eps,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
)
|
||||
for _ in range(num_single_layers)
|
||||
]
|
||||
@@ -728,9 +755,9 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
stacked_params_mapping = [
|
||||
(".to_qkv", ".to_q", "q"),
|
||||
(".to_qkv", ".to_k", "k"),
|
||||
(".to_qkv", ".to_v", "v"),
|
||||
(".to_qkv.", ".to_q.", "q"),
|
||||
(".to_qkv.", ".to_k.", "k"),
|
||||
(".to_qkv.", ".to_v.", "v"),
|
||||
(".add_kv_proj", ".add_q_proj", "q"),
|
||||
(".add_kv_proj", ".add_k_proj", "k"),
|
||||
(".add_kv_proj", ".add_v_proj", "v"),
|
||||
@@ -744,25 +771,32 @@ class Flux2Transformer2DModel(nn.Module):
|
||||
|
||||
loaded_params: set[str] = set()
|
||||
for name, loaded_weight in weights:
|
||||
if "to_qkvkv_mlp_proj" in name:
|
||||
name = name.replace("to_qkvkv_mlp_proj", "to_qkv_mlp_proj")
|
||||
if "to_qkv_mlp_proj" in name:
|
||||
param = params_dict[name]
|
||||
weight_loader = getattr(param, "weight_loader", default_weight_loader)
|
||||
weight_loader(param, loaded_weight)
|
||||
loaded_params.add(name)
|
||||
continue
|
||||
original_name = name
|
||||
mapped = False
|
||||
for param_name, weight_name, shard_id in stacked_params_mapping:
|
||||
if weight_name not in name:
|
||||
if weight_name not in original_name:
|
||||
continue
|
||||
name = name.replace(weight_name, param_name)
|
||||
param = params_dict[name]
|
||||
name = original_name.replace(weight_name, param_name)
|
||||
param = params_dict.get(name)
|
||||
if param is None:
|
||||
break
|
||||
weight_loader = param.weight_loader
|
||||
weight_loader(param, loaded_weight, shard_id)
|
||||
loaded_params.add(name)
|
||||
mapped = True
|
||||
break
|
||||
else:
|
||||
param = params_dict[name]
|
||||
weight_loader = getattr(param, "weight_loader", default_weight_loader)
|
||||
weight_loader(param, loaded_weight)
|
||||
if mapped:
|
||||
continue
|
||||
|
||||
name = original_name
|
||||
if name not in params_dict and ".to_out.0." in name:
|
||||
name = name.replace(".to_out.0.", ".to_out.")
|
||||
# Some GGUF checkpoints include quantized tensors for modules that
|
||||
# are intentionally left unquantized in this model.
|
||||
param = params_dict.get(name)
|
||||
if param is None:
|
||||
continue
|
||||
weight_loader = getattr(param, "weight_loader", default_weight_loader)
|
||||
weight_loader(param, loaded_weight)
|
||||
loaded_params.add(name)
|
||||
return loaded_params
|
||||
@@ -43,6 +43,7 @@ from vllm_omni.diffusion.models.flux2_klein.flux2_klein_transformer import (
|
||||
Flux2Transformer2DModel,
|
||||
)
|
||||
from vllm_omni.diffusion.models.interface import SupportImageInput
|
||||
from vllm_omni.diffusion.quantization import get_vllm_quant_config_for_layers
|
||||
from vllm_omni.diffusion.request import OmniDiffusionRequest
|
||||
from vllm_omni.diffusion.utils.tf_utils import get_transformer_config_kwargs
|
||||
from vllm_omni.model_executor.model_loader.weight_utils import download_weights_from_hf_specific
|
||||
@@ -230,7 +231,8 @@ class Flux2KleinPipeline(nn.Module, CFGParallelMixin, SupportImageInput):
|
||||
).to(self._execution_device)
|
||||
|
||||
transformer_kwargs = get_transformer_config_kwargs(od_config.tf_model_config, Flux2Transformer2DModel)
|
||||
self.transformer = Flux2Transformer2DModel(**transformer_kwargs)
|
||||
quant_config = get_vllm_quant_config_for_layers(od_config.quantization_config)
|
||||
self.transformer = Flux2Transformer2DModel(quant_config=quant_config, **transformer_kwargs)
|
||||
|
||||
self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8
|
||||
self.image_processor = Flux2ImageProcessor(vae_scale_factor=self.vae_scale_factor * 2)
|
||||
@@ -993,4 +995,8 @@ class Flux2KleinPipeline(nn.Module, CFGParallelMixin, SupportImageInput):
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(weights)
|
||||
loaded_weights = loader.load_weights(weights)
|
||||
# Record components loaded by diffusers submodules to satisfy strict checks.
|
||||
loaded_weights |= {f"vae.{name}" for name, _ in self.vae.named_parameters()}
|
||||
loaded_weights |= {f"text_encoder.{name}" for name, _ in self.text_encoder.named_parameters()}
|
||||
return loaded_weights
|
||||
@@ -30,6 +30,7 @@ from vllm_omni.diffusion.model_loader.diffusers_loader import DiffusersPipelineL
|
||||
from vllm_omni.diffusion.models.interface import SupportAudioOutput
|
||||
from vllm_omni.diffusion.models.stable_audio.stable_audio_transformer import StableAudioDiTModel
|
||||
from vllm_omni.diffusion.request import OmniDiffusionRequest
|
||||
from vllm_omni.diffusion.utils.tf_utils import get_transformer_config_kwargs
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
@@ -127,8 +128,9 @@ class StableAudioPipeline(nn.Module, SupportAudioOutput):
|
||||
local_files_only=local_files_only,
|
||||
).to(self.device)
|
||||
|
||||
# Initialize our custom transformer (weights loaded via load_weights)
|
||||
self.transformer = StableAudioDiTModel(od_config=od_config)
|
||||
# Initialize transformer from HF config to keep architecture aligned with checkpoint.
|
||||
transformer_kwargs = get_transformer_config_kwargs(od_config.tf_model_config, StableAudioDiTModel)
|
||||
self.transformer = StableAudioDiTModel(od_config=od_config, **transformer_kwargs)
|
||||
|
||||
# Load scheduler
|
||||
self.scheduler = CosineDPMSolverMultistepScheduler.from_pretrained(
|
||||
|
||||
@@ -644,4 +644,8 @@ class ZImagePipeline(nn.Module):
|
||||
|
||||
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
|
||||
loader = AutoWeightsLoader(self)
|
||||
return loader.load_weights(weights)
|
||||
loaded_weights = loader.load_weights(weights)
|
||||
# Record components loaded by diffusers submodules to satisfy strict checks.
|
||||
loaded_weights |= {f"vae.{name}" for name, _ in self.vae.named_parameters()}
|
||||
loaded_weights |= {f"text_encoder.{name}" for name, _ in self.text_encoder.named_parameters()}
|
||||
return loaded_weights
|
||||
@@ -29,6 +29,7 @@ from vllm.model_executor.layers.layernorm import RMSNorm
|
||||
from vllm.model_executor.layers.linear import (
|
||||
MergedColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
ReplicatedLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
|
||||
@@ -207,21 +208,27 @@ def validate_zimage_tp_constraints(
|
||||
|
||||
|
||||
class TimestepEmbedder(nn.Module):
|
||||
def __init__(self, out_size, mid_size=None, frequency_embedding_size=256):
|
||||
def __init__(
|
||||
self, out_size, mid_size=None, frequency_embedding_size=256, quant_config: "QuantizationConfig | None" = None
|
||||
):
|
||||
super().__init__()
|
||||
if mid_size is None:
|
||||
mid_size = out_size
|
||||
self.mlp = nn.Sequential(
|
||||
nn.Linear(
|
||||
ReplicatedLinear(
|
||||
frequency_embedding_size,
|
||||
mid_size,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
return_bias=False,
|
||||
),
|
||||
nn.SiLU(),
|
||||
nn.Linear(
|
||||
ReplicatedLinear(
|
||||
mid_size,
|
||||
out_size,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
return_bias=False,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -241,7 +248,7 @@ class TimestepEmbedder(nn.Module):
|
||||
|
||||
def forward(self, t):
|
||||
t_freq = self.timestep_embedding(t, self.frequency_embedding_size)
|
||||
weight_dtype = self.mlp[0].weight.dtype
|
||||
weight_dtype = self.mlp[0].bias.dtype
|
||||
if weight_dtype.is_floating_point:
|
||||
t_freq = t_freq.to(weight_dtype)
|
||||
t_emb = self.mlp(t_freq)
|
||||
@@ -420,7 +427,9 @@ class ZImageTransformerBlock(nn.Module):
|
||||
self.modulation = modulation
|
||||
if modulation:
|
||||
self.adaLN_modulation = nn.Sequential(
|
||||
nn.Linear(min(dim, ADALN_EMBED_DIM), 4 * dim, bias=True),
|
||||
ReplicatedLinear(
|
||||
min(dim, ADALN_EMBED_DIM), 4 * dim, bias=True, return_bias=False, quant_config=quant_config
|
||||
),
|
||||
)
|
||||
|
||||
def forward(
|
||||
@@ -473,14 +482,18 @@ class ZImageTransformerBlock(nn.Module):
|
||||
|
||||
|
||||
class FinalLayer(nn.Module):
|
||||
def __init__(self, hidden_size, out_channels):
|
||||
def __init__(self, hidden_size, out_channels, quant_config: "QuantizationConfig | None" = None):
|
||||
super().__init__()
|
||||
self.norm_final = nn.LayerNorm(hidden_size, elementwise_affine=False, eps=1e-6)
|
||||
self.linear = nn.Linear(hidden_size, out_channels, bias=True)
|
||||
self.linear = ReplicatedLinear(
|
||||
hidden_size, out_channels, bias=True, quant_config=quant_config, return_bias=False
|
||||
)
|
||||
|
||||
self.adaLN_modulation = nn.Sequential(
|
||||
nn.SiLU(),
|
||||
nn.Linear(min(hidden_size, ADALN_EMBED_DIM), hidden_size, bias=True),
|
||||
ReplicatedLinear(
|
||||
min(hidden_size, ADALN_EMBED_DIM), hidden_size, bias=True, quant_config=quant_config, return_bias=False
|
||||
),
|
||||
)
|
||||
|
||||
def forward(self, x, c):
|
||||
@@ -652,10 +665,18 @@ class ZImageTransformer2DModel(CachedTransformer):
|
||||
all_x_embedder = {}
|
||||
all_final_layer = {}
|
||||
for patch_idx, (patch_size, f_patch_size) in enumerate(zip(all_patch_size, all_f_patch_size)):
|
||||
x_embedder = nn.Linear(f_patch_size * patch_size * patch_size * in_channels, dim, bias=True)
|
||||
x_embedder = ReplicatedLinear(
|
||||
f_patch_size * patch_size * patch_size * in_channels,
|
||||
dim,
|
||||
bias=True,
|
||||
quant_config=quant_config,
|
||||
return_bias=False,
|
||||
)
|
||||
all_x_embedder[f"{patch_size}-{f_patch_size}"] = x_embedder
|
||||
|
||||
final_layer = FinalLayer(dim, patch_size * patch_size * f_patch_size * self.out_channels)
|
||||
final_layer = FinalLayer(
|
||||
dim, patch_size * patch_size * f_patch_size * self.out_channels, quant_config=quant_config
|
||||
)
|
||||
all_final_layer[f"{patch_size}-{f_patch_size}"] = final_layer
|
||||
|
||||
self.all_x_embedder = nn.ModuleDict(all_x_embedder)
|
||||
@@ -690,10 +711,10 @@ class ZImageTransformer2DModel(CachedTransformer):
|
||||
for layer_id in range(n_refiner_layers)
|
||||
]
|
||||
)
|
||||
self.t_embedder = TimestepEmbedder(min(dim, ADALN_EMBED_DIM), mid_size=1024)
|
||||
self.t_embedder = TimestepEmbedder(min(dim, ADALN_EMBED_DIM), mid_size=1024, quant_config=quant_config)
|
||||
self.cap_embedder = nn.Sequential(
|
||||
RMSNorm(cap_feat_dim, eps=norm_eps),
|
||||
nn.Linear(cap_feat_dim, dim, bias=True),
|
||||
ReplicatedLinear(cap_feat_dim, dim, bias=True, return_bias=False, quant_config=quant_config),
|
||||
)
|
||||
|
||||
self.x_pad_token = nn.Parameter(torch.empty((1, dim)))
|
||||
@@ -957,9 +978,9 @@ class ZImageTransformer2DModel(CachedTransformer):
|
||||
stacked_params_mapping = [
|
||||
# (param_name, shard_name, shard_id)
|
||||
# self-attn
|
||||
(".to_qkv", ".to_q", "q"),
|
||||
(".to_qkv", ".to_k", "k"),
|
||||
(".to_qkv", ".to_v", "v"),
|
||||
(".to_qkv.", ".to_q.", "q"),
|
||||
(".to_qkv.", ".to_k.", "k"),
|
||||
(".to_qkv.", ".to_v.", "v"),
|
||||
# ffn
|
||||
(".w13", ".w1", 0),
|
||||
(".w13", ".w3", 1),
|
||||
|
||||
@@ -28,6 +28,7 @@ from vllm.logger import init_logger
|
||||
|
||||
from .base import DiffusionQuantizationConfig
|
||||
from .fp8 import DiffusionFp8Config
|
||||
from .gguf import DiffusionGgufConfig
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.model_executor.layers.quantization.base_config import (
|
||||
@@ -40,6 +41,7 @@ logger = init_logger(__name__)
|
||||
# To add a new method, create a new config class and register it here
|
||||
_QUANT_CONFIG_REGISTRY: dict[str, type[DiffusionQuantizationConfig]] = {
|
||||
"fp8": DiffusionFp8Config,
|
||||
"gguf": DiffusionGgufConfig,
|
||||
}
|
||||
|
||||
SUPPORTED_QUANTIZATION_METHODS = list(_QUANT_CONFIG_REGISTRY.keys())
|
||||
@@ -108,6 +110,7 @@ def get_vllm_quant_config_for_layers(
|
||||
__all__ = [
|
||||
"DiffusionQuantizationConfig",
|
||||
"DiffusionFp8Config",
|
||||
"DiffusionGgufConfig",
|
||||
"get_diffusion_quant_config",
|
||||
"get_vllm_quant_config_for_layers",
|
||||
"SUPPORTED_QUANTIZATION_METHODS",
|
||||
|
||||
@@ -31,15 +31,14 @@ class DiffusionQuantizationConfig(ABC):
|
||||
# The underlying vLLM config instance
|
||||
_vllm_config: "QuantizationConfig | None" = None
|
||||
|
||||
@classmethod
|
||||
def get_name(cls) -> str:
|
||||
def get_name(self) -> str:
|
||||
"""Return the quantization method name (e.g., 'fp8', 'int8').
|
||||
|
||||
By default, delegates to the underlying vLLM config class.
|
||||
By default, delegates to the underlying vLLM config instance.
|
||||
"""
|
||||
if cls.quant_config_cls is not None:
|
||||
return cls.quant_config_cls.get_name()
|
||||
raise NotImplementedError("Subclass must set quant_config_cls or override get_name()")
|
||||
if self._vllm_config is not None:
|
||||
return self._vllm_config.get_name()
|
||||
raise NotImplementedError("Subclass must initialize _vllm_config or override get_name().")
|
||||
|
||||
def get_vllm_quant_config(self) -> "QuantizationConfig | None":
|
||||
"""Return the underlying vLLM QuantizationConfig for linear layers."""
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""GGUF quantization config for diffusion transformers."""
|
||||
|
||||
import gguf
|
||||
import torch
|
||||
from vllm import _custom_ops as ops
|
||||
from vllm.model_executor.layers.quantization.gguf import (
|
||||
UNQUANTIZED_TYPES,
|
||||
GGUFConfig,
|
||||
GGUFLinearMethod,
|
||||
LinearBase,
|
||||
QuantizeMethodBase,
|
||||
UnquantizedLinearMethod,
|
||||
is_layer_skipped_gguf,
|
||||
)
|
||||
|
||||
from .base import DiffusionQuantizationConfig
|
||||
|
||||
|
||||
def dequant_gemm_gguf(x: torch.Tensor, qweight: torch.Tensor, qweight_type: int) -> torch.Tensor:
|
||||
if qweight_type in UNQUANTIZED_TYPES:
|
||||
return x @ qweight.T
|
||||
block_size, type_size = gguf.GGML_QUANT_SIZES[qweight_type]
|
||||
shape = (qweight.shape[0], qweight.shape[1] // type_size * block_size)
|
||||
weight = ops.ggml_dequantize(qweight, qweight_type, *shape, x.dtype)
|
||||
return x @ weight.T
|
||||
|
||||
|
||||
class DiffusionGGUFLinearMethod(GGUFLinearMethod):
|
||||
def apply(
|
||||
self,
|
||||
layer: torch.nn.Module,
|
||||
x: torch.Tensor,
|
||||
bias: torch.Tensor | None = None,
|
||||
) -> torch.Tensor:
|
||||
# Dequantize + GEMM path: torch.matmul multiplies over the last
|
||||
# dimension and broadcasts leading dimensions, so no 2D flattening
|
||||
# is required here.
|
||||
shard_id = getattr(layer.qweight, "shard_id", None)
|
||||
|
||||
if shard_id:
|
||||
# dequantize shard weights respectively
|
||||
shard_id = ["q", "k", "v"] if "q" in shard_id else shard_id
|
||||
qweight = layer.qweight
|
||||
result = []
|
||||
for idx in shard_id:
|
||||
start, end, offset = layer.qweight.shard_offset_map[idx]
|
||||
qweight_type = layer.qweight_type.shard_weight_type[idx]
|
||||
result.append(dequant_gemm_gguf(x, qweight[start:end, :offset].contiguous(), qweight_type))
|
||||
out = torch.cat(result, axis=-1)
|
||||
else:
|
||||
qweight = layer.qweight
|
||||
qweight_type = layer.qweight_type.weight_type
|
||||
out = dequant_gemm_gguf(x, qweight, qweight_type)
|
||||
if bias is not None:
|
||||
out.add_(bias)
|
||||
return out
|
||||
|
||||
|
||||
class _GGUFConfig(GGUFConfig):
|
||||
def get_quant_method(self, layer: torch.nn.Module, prefix: str) -> "QuantizeMethodBase":
|
||||
if isinstance(layer, LinearBase):
|
||||
if is_layer_skipped_gguf(prefix, self.unquantized_modules, self.packed_modules_mapping):
|
||||
return UnquantizedLinearMethod()
|
||||
return DiffusionGGUFLinearMethod(self)
|
||||
return None
|
||||
|
||||
|
||||
class DiffusionGgufConfig(DiffusionQuantizationConfig):
|
||||
"""GGUF quantization config for diffusion transformers.
|
||||
|
||||
This is a thin wrapper around vLLM's GGUFConfig and also carries
|
||||
the GGUF model reference for loader use.
|
||||
|
||||
Args:
|
||||
gguf_model: GGUF model path or HF reference (repo/file or repo:quant_type)
|
||||
unquantized_modules: Optional list of module name patterns to skip GGUF
|
||||
quantization. Note: diffusion linear layers often use short prefixes
|
||||
(e.g., "to_qkv"), so these patterns are matched as substrings.
|
||||
"""
|
||||
|
||||
quant_config_cls = GGUFConfig
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
gguf_model: str | None = None,
|
||||
unquantized_modules: list[str] | None = None,
|
||||
) -> None:
|
||||
self.gguf_model = gguf_model
|
||||
self.unquantized_modules = unquantized_modules or []
|
||||
|
||||
self._vllm_config = _GGUFConfig(unquantized_modules=self.unquantized_modules)
|
||||
@@ -74,6 +74,7 @@ class OmniEngineArgs(EngineArgs):
|
||||
stage_connector_spec: dict[str, Any] = field(default_factory=dict)
|
||||
async_chunk: bool = False
|
||||
omni_kv_config: dict | None = None
|
||||
quantization_config: Any | None = None
|
||||
worker_type: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
@@ -230,6 +231,7 @@ class AsyncOmniEngineArgs(AsyncEngineArgs):
|
||||
stage_connector_spec: dict[str, Any] = field(default_factory=dict)
|
||||
async_chunk: bool = False
|
||||
omni_kv_config: dict | None = None
|
||||
quantization_config: Any | None = None
|
||||
worker_type: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
|
||||
@@ -6,6 +6,7 @@ diffusion models (e.g., Qwen-Image) through the same CLI interface.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
from typing import Any
|
||||
@@ -209,6 +210,15 @@ class OmniServeCommand(CLISubcommand):
|
||||
help="Ring Sequence Parallelism degree for diffusion models. "
|
||||
"Equivalent to setting DiffusionParallelConfig.ring_degree.",
|
||||
)
|
||||
omni_config_group.add_argument(
|
||||
"--quantization-config",
|
||||
type=json.loads,
|
||||
default=None,
|
||||
help=(
|
||||
"JSON string for diffusion quantization_config. "
|
||||
'Example: \'{"method":"gguf","gguf_model":"/path/to/model.gguf"}\'.'
|
||||
),
|
||||
)
|
||||
|
||||
# HSDP (Hybrid Sharded Data Parallel) parameters
|
||||
omni_config_group.add_argument(
|
||||
|
||||
@@ -285,6 +285,13 @@ class OmniBase:
|
||||
if lora_scale is not None:
|
||||
if not hasattr(cfg.engine_args, "lora_scale") or cfg.engine_args.lora_scale is None:
|
||||
cfg.engine_args.lora_scale = lora_scale
|
||||
quantization_config = kwargs.get("quantization_config")
|
||||
if quantization_config is not None:
|
||||
if (
|
||||
not hasattr(cfg.engine_args, "quantization_config")
|
||||
or cfg.engine_args.quantization_config is None
|
||||
):
|
||||
cfg.engine_args.quantization_config = quantization_config
|
||||
except Exception as e:
|
||||
logger.warning("Failed to inject LoRA config for stage: %s", e)
|
||||
|
||||
|
||||
Reference in new issue
Block a user