[Quantization] Migrate GGUF diffusion model support to OOT plugin (#4769)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
1 parent
ff6f0da271
commit
b95905deb1
35 files changed
+272
-1614
No files matched your search
@@ -9,6 +9,15 @@ for the transformer.
|
||||
|
||||
GGUF is static quantization: the quantized weights are produced before serving.
|
||||
|
||||
!!! note
|
||||
GGUF support has migrated to OOT [vllm-gguf-plugin](https://github.com/vllm-project/vllm-gguf-plugin). Make sure you have GGUF plugin installed before serving a GGUF model.
|
||||
|
||||
Before serving a GGUF model, make sure to install the [vllm-gguf-plugin](https://github.com/vllm-project/vllm-gguf-plugin):
|
||||
|
||||
```bash
|
||||
uv pip install vllm-gguf-plugin>=0.0.3
|
||||
```
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
|
||||
Reference in new issue
Block a user