[Quantization] feat: add qwen-image gguf adapter (#1755)
Signed-off-by: David Chen <530634352@qq.com>
This commit is contained in:
1 parent
61cd532cff
commit
d6b55b9196
6 files changed
+345
-13
No files matched your search
@@ -6,7 +6,7 @@
|
||||
3. Keep user-facing knobs minimal and consistent across offline and online flows.
|
||||
|
||||
## Scope
|
||||
1. Models: Z-Image, and Flux2-klein.
|
||||
1. Models: Z-Image, Flux2-klein, and Qwen-Image family.
|
||||
2. Components: diffusion transformer weights, loader paths, and quantization configs.
|
||||
3. Modes: native GGUF (transformer-only weights).
|
||||
|
||||
@@ -124,19 +124,21 @@ x @ weight.T
|
||||
1. `DiffusersPipelineLoader.load_model` detects `quantization_config.method == "gguf"`.
|
||||
2. `gguf_model` is resolved as one of: local file, `repo/file.gguf`, or `repo:quant_type`.
|
||||
3. GGUF weights are routed through adapters in `vllm_omni/diffusion/model_loader/gguf_adapters/`.
|
||||
4. Name mapping is applied per-architecture (Z-Image, Flux2Klein).
|
||||
4. Name mapping is applied per-architecture (Z-Image, Flux2Klein, Qwen-Image).
|
||||
5. GGUF weights are loaded into transformer modules, remaining non-transformer weights come from the HF checkpoint.
|
||||
|
||||
## GGUF Adapter Design
|
||||
1. `GGUFAdapter` is an abstract base class for model-specific adapters.
|
||||
2. `Flux2KleinGGUFAdapter` implements Flux2-Klein remapping + qkv split + adaLN swap.
|
||||
3. `ZImageGGUFAdapter` implements Z-Image qkv + ffn shard handling and linear qweight routing.
|
||||
4. `get_gguf_adapter(...)` strictly selects by model class/config; unsupported models raise an error (no fallback adapter).
|
||||
4. `QwenImageGGUFAdapter` implements Qwen-Image fused attention projection remapping.
|
||||
5. `get_gguf_adapter(...)` strictly selects by model class/config; unsupported models raise an error (no fallback adapter).
|
||||
|
||||
Adapter paths:
|
||||
- Base: `vllm_omni/diffusion/model_loader/gguf_adapters/base.py`
|
||||
- Z-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/z_image.py`
|
||||
- Flux2-Klein: `vllm_omni/diffusion/model_loader/gguf_adapters/flux2_klein.py`
|
||||
- Qwen-Image: `vllm_omni/diffusion/model_loader/gguf_adapters/qwen_image.py`
|
||||
|
||||
## User Usage (Offline)
|
||||
|
||||
@@ -195,3 +197,31 @@ curl -X POST http://localhost:8000/v1/images/generations \
|
||||
"num_inference_steps": 4
|
||||
}'
|
||||
```
|
||||
|
||||
## Qwen-Image Example
|
||||
|
||||
### Offline
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--gguf-model QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf \
|
||||
--quantization gguf \
|
||||
--prompt "a red paper kite hanging from a pine tree in a winter courtyard" \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--seed 42 \
|
||||
--num_inference_steps 20 \
|
||||
--output outputs/qwen_image_q4km.png
|
||||
```
|
||||
|
||||
### Online
|
||||
```bash
|
||||
vllm serve Qwen/Qwen-Image \
|
||||
--omni \
|
||||
--port 8000 \
|
||||
--quantization-config '{"method":"gguf","gguf_model":"QuantStack/Qwen-Image-GGUF/Qwen_Image-Q4_K_M.gguf"}'
|
||||
```
|
||||
|
||||
Notes:
|
||||
1. The GGUF repo provides transformer weights only; the base model repo still provides tokenizer, text encoder, scheduler, and VAE.
|
||||
2. The same adapter path covers `QwenImagePipeline`, `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, and `QwenImageLayeredPipeline`.
|
||||
Reference in new issue
Block a user