[Docs] Guide for using sleep mode and enable sleep mode (#660)

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
This commit is contained in:
knlnguyen1802andJiangyun Zhu authored and GitHub committed 2026-01-06 17:31:51 +08:00
1 parent b7e63e07d7
commit c32235c4e0
3 files changed
+62

No files matched your search

+39
View File
@@ -0,0 +1,39 @@
# Sleep Mode
vLLM-Omni’s **Sleep Mode** allows you to temporarily release most GPU memory used by a model—such as model weights and key-value (KV) caches (for autoregressive models)—**without stopping the server or unloading the Docker container**.
This feature is inherited from [vLLM’s Sleep Mode](https://blog.vllm.ai/2025/10/26/sleep-mode.html), which provides zero-reload model switching for multi-model serving.
It is especially useful in **RLHF**, **training**, or **cost-saving scenarios**, where GPU resources must be freed between inference workloads.
---
## Omni Model
Omni model inherit the feature from vLLM' Sleep Mode
This means:
- Support both Level 1 and Level 2 sleep, allow to release and reset both model weights and KV Cache
## Diffusion Model Extension
We added Sleep Mode support for **diffusion models**, which previously lacked this functionality.
In diffusion pipelines, this currently only offloads **model weight memory**, as these models typically do not use KV caches.
This means:
- Diffusion models can now enter Level 1 sleep.
- Pipeline states (e.g., noise schedulers, buffers) remain intact after waking.
- Useful for releasing VRAM between image generation or training cycles.
---
## Enable sleep mode
To enable sleep mode, set the `enable_sleep_mode` in `engine_args` to `True`
Example:
```python
omni = Omni(model=...,enable_sleep_mode=True)
```