b766c4702c
Signed-off-by: juboyu <767868009@qq.com> Signed-off-by: JuboYu <767868009@qq.com> Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com> Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com> Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
863 B
863 B
Quantization for Diffusion Transformers
vLLM-Omni supports quantization of DiT linear layers to reduce memory usage and accelerate inference.
Supported Methods
| Method | Guide |
|---|---|
| FP8 | FP8 |
| Int8 | Int8 |
| GGUF | GGUF |
Device Compatibility for FP8
| GPU Generation | Example GPUs | FP8 Mode |
|---|---|---|
| Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
Kernel selection is automatic.
Device Compatibility for Int8
| Device Type | Generation | Example | Int8 Mode |
|---|---|---|---|
| NVIDIA GPU | Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
| Ascend NPU | Atlas A2/Atlas A3 | Atlas 800T A2/Atlas 900 A3 | Full W8A8 with native hardware |