0b884ebc6c
Signed-off-by: David Chen <530634352@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
489 B
489 B
Quantization for Diffusion Transformers
vLLM-Omni supports quantization of DiT linear layers to reduce memory usage and accelerate inference.
Supported Methods
| Method | Guide |
|---|---|
| FP8 | FP8 |
| GGUF | GGUF |
Device Compatibility
| GPU Generation | Example GPUs | FP8 Mode |
|---|---|---|
| Ada/Hopper (SM 89+) | RTX 4090, H100, H200 | Full W8A8 with native hardware |
Kernel selection is automatic.