anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Merge branch 'main' into codex-vae-spatial-decode-auto

Browse Source
This commit is contained in:
Hongsheng Liu authored and GitHub committed 2026-08-12 11:38:12 +08:00
2 parents af1825e07d a15b5b4bcd
commit 879d19c4c2
186 files changed
+10814 -1165
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/diffusion/lora.md
+105 -7
View File
@@ -1,14 +1,18 @@
# LoRA (Low-Rank Adaptation) Guide
LoRA (Low-Rank Adaptation) enables fine-tuning diffusion models by adding trainable low-rank matrices to existing model weights. vLLM-Omni currently supports PEFT-style LoRA adapters, allowing you to customize model behavior without modifying the base model weights.
LoRA (Low-Rank Adaptation) enables fine-tuning diffusion models by adding trainable low-rank matrices to existing model weights. vLLM-Omni supports two LoRA backends: **PEFT** for PEFT-style adapters, and **Distill** usually for few-steps inference. **PEFT** backend allows you to customize model behavior without modifying the base model weights. **Distill** backend fuses LoRA weights into the base model at initialization.
## Overview
LoRA adapters are lightweight, model-specific fine-tuning weights that can be dynamically loaded and applied to diffusion models. vLLM-Omni uses a unified LoRA handling mechanism similar to vLLM with LRU cache management.
LoRA adapters are lightweight, model-specific fine-tuning weights that can be applied to diffusion models in two ways:
- **PEFT backend** (`--lora-backend peft`, default): Loads a PEFT-format adapter folder via `DiffusionLoRAManager`. Adapters are cached (LRU) and activated per request via `LoRARequest`. It uses a unified LoRA handling mechanisms similar to vLLM with LRU cache management.
- **Distill backend** (`--lora-backend distill`): Calls `pipeline.load_lora_weights` once at initialization to fuse one or more concrete checkpoint files directly into the base weights. Typically used for distilled few-step LoRAs (e.g. Lightning, LightX2V).
## LoRA Adapter Format
LoRA adapters must be in **PEFT (Parameter-Efficient Fine-Tuning)** format. A typical LoRA adapter directory structure:
### PEFT (Parameter-Efficient Fine-Tuning) format (default)
A typical PEFT-format LoRA adapter directory structure:
```
lora_adapter/
@@ -25,19 +29,20 @@ The `adapter_config.json` file contains metadata about the LoRA adapter, includi
### Offline Inference
#### Pre-loaded LoRA
#### PEFT backend: pre-loaded LoRA
Load a LoRA adapter at initialization. This adapter is pre-loaded into the cache and can be activated by requests:
Load a PEFT-format LoRA adapter at initialization. The adapter is pre-loaded into the cache and can be activated per request:
```python
from vllm_omni import Omni
from vllm_omni.lora.request import LoRARequest
lora_path="/path/to/lora_adapter"
lora_path = "/path/to/lora_adapter"
omni = Omni(
model="stabilityai/stable-diffusion-3.5-medium",
lora_path=lora_path
lora_path=lora_path,
lora_backend="peft", # default, can be omitted
)
lora_request = LoRARequest(
@@ -53,6 +58,98 @@ outputs = omni.generate(
)
```
#### Distill backend: fuse distilled LoRA at init
For distilled few-step LoRAs, pass `lora_backend="distill"` together with one or more concrete `.safetensors` files. The weights are fused into the base model once at init; subsequent `generate()` calls do not need a `LoRARequest`.
##### Supported pipelines
For Qwen-Image and Wan pipelines, the distill backend calls `pipeline.load_lora_weights(...)` during worker initialization.
| Pipeline | Supported distilled LoRA repo | Notes |
|----------|--------------------------------|-------|
| `QwenImagePipeline` | `lightx2v/Qwen-Image-2512-Lightning` | Used with Qwen-Image-2512 Lightning-style few-step inference. |
| `Wan22Pipeline` | `lightx2v/Wan2.1-Distill-Loras`, `lightx2v/Wan2.2-Distill-Loras` | Wan2.1 uses one LoRA file. For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
| `Wan22I2VPipeline` | `lightx2v/Wan2.2-Distill-Loras` | For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
Other diffusion pipelines are not currently listed as supporting distilled LoRA. Use the PEFT backend for request-time adapters, or bake converted weights into a local Diffusers directory before serving.
Single-file example (Qwen-Image-Lightning):
```python
from vllm_omni import Omni
omni = Omni(
model="Qwen/Qwen-Image-2512",
lora_path="/path/to/Qwen-Image-2512-Lightning.safetensors",
lora_backend="distill",
)
outputs = omni.generate(prompt="A piece of cheesecake")
```
Multi-file example (Wan2.2 MoE, high + low noise):
```python
from vllm_omni import Omni
omni = Omni(
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
lora_path=[
"/path/to/wan2.2_high_noise_lora.safetensors", # -> transformer
"/path/to/wan2.2_low_noise_lora.safetensors", # -> transformer_2
],
lora_backend="distill",
)
```
The CLI examples under `examples/offline_inference/` accept the same flags, e.g.:
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--lora-backend distill \
--lora-path /path/to/high.safetensors /path/to/low.safetensors \
--prompt "A cat playing with yarn"
```
##### Online serving
Distilled LoRAs can also be fused when an online diffusion server starts. The
adapter remains active for the lifetime of that server, so requests do not pass
a per-request `lora` field.
```bash
vllm serve Qwen/Qwen-Image-2512 \
--omni \
--port 8091 \
--lora-backend distill \
--lora-path /path/to/Qwen-Image-2512-Lightning-4steps.safetensors
```
Then send a normal generation request with the sampling settings expected by
the distilled checkpoint:
```bash
curl -s http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "A Chinese female college student, around 20 years old, with a very short haircut that conveys a gentle, artistic vibe. Her hair naturally falls to partially cover her cheeks, projecting a tomboyish yet charming demeanor. She has cool-toned fair skin and delicate features, with a slightly shy yet subtly confident expression—her mouth crooked in a playful, youthful smirk. She wears an off-shoulder top, revealing one shoulder, with a well-proportioned figure. The image is framed as a close-up selfie: she dominates the foreground, while the background clearly shows her dormitory—a neatly made bed with white linens on the top bunk, a tidy study desk with organized stationery, and wooden cabinets and drawers. The photo is captured on a smartphone under soft, even ambient lighting, with natural tones, high clarity, and a bright, lively atmosphere full of youthful, everyday energy."}
],
"extra_body": {
"height": 1024,
"width": 1024,
"num_inference_steps": 4,
"true_cfg_scale": 1.0,
"seed": 42
}
}' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > qwen_image_distill_lora_output.png
```
For Wan2.2 MoE serving, pass the high-noise checkpoint first and the low-noise
checkpoint second to `--lora-path`.
!!! note "Server-side Path Requirement"
The LoRA adapter path (`local_path`) must be readable on the **server** machine. If your client and server are on different machines, ensure the LoRA adapter is accessible via a shared mount or copied to the server.
@@ -141,6 +238,7 @@ Notes:
- This route avoids runtime LoRA loading changes in vLLM-Omni when you choose to bake converted weights into a local Diffusers directory.
- Output quality and speed depend on the replacement checkpoints and sampling params you choose.
- If you only need to fuse distilled LoRAs into a Wan2.2 checkpoint at load time (without the full LightX2V convert + assemble pipeline), you can instead pass them directly via `--lora-backend distill --lora-path <high>.safetensors <low>.safetensors`. See the [Distill backend](#distill-backend-fuse-distilled-lora-at-init) section above.
## See Also
docs/user_guide/diffusion/startup_and_loading.md
+60
View File
@@ -0,0 +1,60 @@
# Diffusion Startup and Loading
Large diffusion models can take several minutes to load at startup. vLLM-Omni
loads safetensors shards in parallel to reduce this initialization time.
Multi-thread weight loading is enabled by default with four threads. No
configuration is needed for the default behavior.
## Configuration
| Parameter | CLI flag | Default | Description |
| --- | --- | --- | --- |
| `enable_multithread_weight_load` | `--disable-multithread-weight-load` | `True` | Pass the flag to disable multi-thread loading |
| `num_weight_load_threads` | `--num-weight-load-threads` | `4` | Number of parallel weight-loading threads |
!!! tip
The default balances startup speed and disk I/O contention. Fast NVMe
storage may benefit from more threads, while network storage or hard disks
may not.
## Online Serving
```bash
# Default: multi-thread loading with four threads
vllm serve Qwen/Qwen-Image --omni --port 8091
# Increase the thread count
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni \
--num-weight-load-threads 8
# Disable multi-thread loading
vllm serve Qwen/Qwen-Image --omni --disable-multithread-weight-load
```
## Offline Inference
```python
from vllm_omni import Omni
# Default: multi-thread loading with four threads
omni = Omni(model="Qwen/Qwen-Image")
# Increase the thread count
omni = Omni(
model="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
num_weight_load_threads=8,
)
```
## Reference Benchmarks
The following measurements were collected on NVIDIA H800 hardware. Treat them
as reference results rather than a guarantee for other storage or hardware
configurations.
| Model | Sequential loading | Multi-thread loading | Speedup |
| --- | ---: | ---: | ---: |
| **Qwen/Qwen-Image** (53.7 GiB) | 168 s | 27 s | **6.2x** |
| **Wan-AI/Wan2.2-I2V-A14B-Diffusers** (64.5 GiB) | 283 s | 56 s | **5.1x** |