[Doc] format lora related docs for the user's end (#1009)
Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk> Signed-off-by: Andy Zhou <46011930+AndyZhou952@users.noreply.github.com>
This commit is contained in:
Andy Zhou authored and GitHub committed
1 parent
caed95fb45
commit
a2d5b61703
16 files changed
+314
-687
No files matched your search
@@ -1,108 +0,0 @@
|
||||
# LoRA Inference(Diffusion)
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/lora_inference>.
|
||||
|
||||
|
||||
This directory contains examples for using LoRA (Low-Rank Adaptation) adapters with vLLM-omni diffusion models for offline inference.
|
||||
The example uses the `stabilityai/stable-diffusion-3.5-medium` as the default model, but you can replace it with other models in vLLM-omni.
|
||||
|
||||
## Overview
|
||||
|
||||
Similar to vLLM, vLLM-omni uses a unified LoRA handling mechanism:
|
||||
|
||||
- **Pre-loaded LoRA**: Loaded at initialization via `--lora-path` (pre-loaded into cache)
|
||||
- **Per-request LoRA**: Loaded on-demand. In the example, the LoRA is loaded via `--lora-request-path` in each request
|
||||
|
||||
Both approaches use the same underlying mechanism - all LoRA adapters are handled uniformly through `set_active_adapter()`. If no LoRA request is provided in a request, all adapters are deactivated.
|
||||
|
||||
## Usage
|
||||
|
||||
### Pre-loaded LoRA (via --lora-path)
|
||||
|
||||
Load a LoRA adapter at initialization. This adapter is pre-loaded into the cache and can be activated by requests:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_preloaded.png
|
||||
```
|
||||
|
||||
**Note**: When using `--lora-path`, the adapter is loaded at init time with a stable ID derived from the adapter path. This example activates it automatically for the request.
|
||||
|
||||
### Per-request LoRA (via --lora-request-path)
|
||||
|
||||
Load a LoRA adapter on-demand for each request:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-request-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_per_request.png
|
||||
```
|
||||
|
||||
### No LoRA
|
||||
|
||||
If no LoRA request is provided, we will use the base model without any LoRA adapters:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_no_lora.png
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
### LoRA Parameters
|
||||
|
||||
- `--lora-path`: Path to LoRA adapter folder to pre-load at initialization (loads into cache with a stable ID derived from the path)
|
||||
- `--lora-request-path`: Path to LoRA adapter folder for per-request loading
|
||||
- `--lora-request-id`: Integer ID for the LoRA adapter (optional). If not provided and `--lora-request-path` is set, will derive a stable ID from the path.
|
||||
- `--lora-scale`: Scale factor for LoRA weights (default: 1.0). Higher values increase the influence of the LoRA adapter.
|
||||
|
||||
### Standard Parameters
|
||||
|
||||
- `--prompt`: Text prompt for image generation (required)
|
||||
- `--seed`: Random seed for reproducibility (default: 42)
|
||||
- `--height`: Image height in pixels (default: 1024)
|
||||
- `--width`: Image width in pixels (default: 1024)
|
||||
- `--num-inference-steps`: Number of denoising steps (default: 50)
|
||||
- `--output`: Output file path (default: `lora_output.png`)
|
||||
|
||||
## How LoRA Works
|
||||
|
||||
All LoRA adapters are handled uniformly:
|
||||
|
||||
1. **Initialization**: If `--lora-path` is provided, the adapter is loaded into cache with a stable ID derived from the adapter path
|
||||
2. **Per-request**: If `--lora-request-path` is provided, the adapter is loaded/activated for that request
|
||||
3. **No LoRA**: If no LoRA request is provided (`req.lora_request` is None), all adapters are deactivated
|
||||
|
||||
The system uses LRU cache management - adapters are cached and evicted when the cache is full (unless pinned).
|
||||
|
||||
## LoRA Adapter Format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "lora_inference.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/lora_inference/lora_inference.py"
|
||||
``````
|
||||
@@ -108,6 +108,36 @@ Key arguments:
|
||||
|
||||
> ℹ️ Qwen-Image currently publishes best-effort presets at `1328x1328`, `1664x928`, `928x1664`, `1472x1140`, `1140x1472`, `1584x1056`, and `1056x1584`. Adjust `--height/--width` accordingly for the most reliable outcomes.
|
||||
|
||||
## LoRA
|
||||
|
||||
This example supports Peft-compatible LoRA (Low-Rank Adaptation) adapters for diffusion models. Pass `--lora-path` to use a LoRA adapter and optionally `--lora-scale` (default 1.0); omit it to use the base model only.
|
||||
|
||||
### Basic usage with LoRA
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model Tongyi-MAI/Z-Image-Turbo \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--output output.png
|
||||
```
|
||||
|
||||
### LoRA parameters
|
||||
|
||||
- `--lora-path`: Path to LoRA adapter folder (PEFT format). Loaded at initialization and used for generation.
|
||||
- `--lora-scale`: Scale factor for LoRA weights (default: 1.0). Higher values increase the influence of the LoRA adapter.
|
||||
|
||||
### LoRA adapter format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Web UI Demo
|
||||
|
||||
Launch the gradio demo:
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# LoRA Inference(Diffusion)
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/lora_inference>.
|
||||
|
||||
|
||||
This example shows how to use **per-request LoRA** with vLLM-Omni diffusion models via the OpenAI-compatible Chat Completions API.
|
||||
|
||||
> Note: The LoRA adapter path must be readable on the **server** machine (usually a local path or a mounted directory).
|
||||
> Note: This example uses `/v1/chat/completions`. LoRA payloads for other OpenAI endpoints are not implemented here.
|
||||
|
||||
## Start Server
|
||||
|
||||
```bash
|
||||
# Pick a diffusion model (examples)
|
||||
# export MODEL=stabilityai/stable-diffusion-3.5-medium
|
||||
# export MODEL=Qwen/Qwen-Image
|
||||
|
||||
bash run_server.sh
|
||||
```
|
||||
|
||||
## Call API (curl)
|
||||
|
||||
```bash
|
||||
# Required: local LoRA folder on the server
|
||||
export LORA_PATH=/path/to/lora_adapter
|
||||
|
||||
# Optional
|
||||
export SERVER=http://localhost:8091
|
||||
export PROMPT="A piece of cheesecake"
|
||||
export LORA_NAME=my_lora
|
||||
export LORA_SCALE=1.0
|
||||
# Optional: if omitted, the server derives a stable id from LORA_PATH.
|
||||
# export LORA_INT_ID=123
|
||||
|
||||
bash run_curl_lora_inference.sh
|
||||
```
|
||||
|
||||
## Call API (Python)
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora_adapter \
|
||||
--lora-name my_lora \
|
||||
--lora-scale 1.0 \
|
||||
--output output.png
|
||||
```
|
||||
|
||||
## LoRA Format
|
||||
|
||||
LoRA adapters should be in PEFT format, for example:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "openai_chat_client.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/lora_inference/openai_chat_client.py"
|
||||
``````
|
||||
??? abstract "run_curl_lora_inference.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/lora_inference/run_curl_lora_inference.sh"
|
||||
``````
|
||||
??? abstract "run_server.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/lora_inference/run_server.sh"
|
||||
``````
|
||||
@@ -61,6 +61,59 @@ python gradio_demo.py
|
||||
# Visit http://localhost:7860
|
||||
```
|
||||
|
||||
## LoRA
|
||||
|
||||
This example supports Peft-compatible LoRA (Low-Rank Adaptation) adapters for diffusion models. The LoRA adapter path must be readable on the **server** machine (usually a local path or a mounted directory).
|
||||
|
||||
### Using Python Client with LoRA
|
||||
|
||||
```bash
|
||||
python openai_chat_client.py \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora_adapter \
|
||||
--lora-name my_lora \
|
||||
--lora-scale 1.0 \
|
||||
--output output.png
|
||||
```
|
||||
|
||||
### Using curl with LoRA (Images API)
|
||||
|
||||
The `/v1/images/generations` endpoint supports a `lora` field in the request body:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/images/generations \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": "A piece of cheesecake",
|
||||
"size": "1024x1024",
|
||||
"seed": 42,
|
||||
"lora": {
|
||||
"name": "my_lora",
|
||||
"local_path": "/path/to/lora_adapter",
|
||||
"scale": 1.0
|
||||
}
|
||||
}' | jq -r '.data[0].b64_json' | base64 -d > output.png
|
||||
```
|
||||
|
||||
### LoRA Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
| ------------- | ------ | ---------------------------------------------------------------- |
|
||||
| `name` | str | LoRA adapter name (optional, defaults to path stem) |
|
||||
| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) |
|
||||
| `scale` | float | LoRA scale factor (default: 1.0) |
|
||||
| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) |
|
||||
|
||||
### LoRA Adapter Format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Request Format
|
||||
|
||||
### Simple Text Generation
|
||||
|
||||
Reference in new issue
Block a user