[Diffusion] Add Diffusers quantization config bridge (#3837)

Signed-off-by: CAICAIIs <3360776475@qq.com>
This commit is contained in:
CAICAII authored and GitHub committed 2026-07-03 03:06:44 +00:00
1 parent 06329c6450
commit b443504d61
7 files changed
+816 -9

No files matched your search

@@ -9,7 +9,8 @@ vLLM-Omni supports running diffusion models with the diffusers backend, directly
The diffusers backend is a black-box adapter. Its primary focus is to serve diffusion models online.
Currently, the following features are NOT yet supported.
It is not guaranteed whether they will be supported in the future.
Community contributions are welcome to add these features when they can be
implemented with clear Diffusers or vLLM-Omni behavior.
- CFG parallel execution
- Sequence parallel execution
@@ -66,6 +67,35 @@ This is suitable for sampling parameters not available through the vLLM-Omni int
When a parameter is available in the vLLM-Omni interface, it will be adapted here.
But if that parameter is simultaneously set in both the vLLM-Omni interface and `diffusers_call_kwargs`, the **former** will take precedence (because it is set at request time).
### Quantization
Use `diffusers_load_kwargs` for Diffusers-native quantization options. If
`diffusers_load_kwargs["quantization_config"]` is provided as a dictionary, the
diffusers backend builds a Diffusers quantization config and lets Diffusers
validate it before calling `DiffusionPipeline.from_pretrained()`.
The diffusers backend also provides a small compatibility shortcut for
vLLM-Omni quantization configs when `diffusers_load_kwargs` does not already
contain `quantization_config`. Currently this maps online/dynamic `fp8` and
online/dynamic `int8` to Diffusers/TorchAO dynamic quantization for transformer
components such as `transformer` or `transformer_2`. This path requires
`torchao` to be installed.
For example, the CLI can request dynamic FP8 through the vLLM-Omni interface:
```bash
vllm serve "Qwen/Qwen-Image" \
--omni \
--diffusion-load-format diffusers \
--quantization-config '{"method": "fp8"}'
```
Other vLLM-Omni quantization methods, such as `gguf`, `modelopt`, `mxfp4`,
`mxfp8`, serialized checkpoints, static FP8 configs, and layer-name skip lists
such as `ignored_layers`, are intentionally not translated by this compatibility
shortcut. Use Diffusers-native configuration through `diffusers_load_kwargs` or
a native vLLM-Omni pipeline for those cases.
### Attention Backends
The diffusers backend converts