[Doc] format lora related docs for the user's end (#1009)
Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk> Signed-off-by: Andy Zhou <46011930+AndyZhou952@users.noreply.github.com>
This commit is contained in:
1 parent
caed95fb45
commit
a2d5b61703
16 files changed
+314
-687
No files matched your search
@@ -1,98 +0,0 @@
|
||||
# LoRA Inference(Diffusion)
|
||||
|
||||
This directory contains examples for using LoRA (Low-Rank Adaptation) adapters with vLLM-omni diffusion models for offline inference.
|
||||
The example uses the `stabilityai/stable-diffusion-3.5-medium` as the default model, but you can replace it with other models in vLLM-omni.
|
||||
|
||||
## Overview
|
||||
|
||||
Similar to vLLM, vLLM-omni uses a unified LoRA handling mechanism:
|
||||
|
||||
- **Pre-loaded LoRA**: Loaded at initialization via `--lora-path` (pre-loaded into cache)
|
||||
- **Per-request LoRA**: Loaded on-demand. In the example, the LoRA is loaded via `--lora-request-path` in each request
|
||||
|
||||
Both approaches use the same underlying mechanism - all LoRA adapters are handled uniformly through `set_active_adapter()`. If no LoRA request is provided in a request, all adapters are deactivated.
|
||||
|
||||
## Usage
|
||||
|
||||
### Pre-loaded LoRA (via --lora-path)
|
||||
|
||||
Load a LoRA adapter at initialization. This adapter is pre-loaded into the cache and can be activated by requests:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_preloaded.png
|
||||
```
|
||||
|
||||
**Note**: When using `--lora-path`, the adapter is loaded at init time with a stable ID derived from the adapter path. This example activates it automatically for the request.
|
||||
|
||||
### Per-request LoRA (via --lora-request-path)
|
||||
|
||||
Load a LoRA adapter on-demand for each request:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-request-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_per_request.png
|
||||
```
|
||||
|
||||
### No LoRA
|
||||
|
||||
If no LoRA request is provided, we will use the base model without any LoRA adapters:
|
||||
|
||||
```bash
|
||||
python -m examples.offline_inference.lora_inference.lora_inference \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--num-inference-steps 50 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output output_no_lora.png
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
### LoRA Parameters
|
||||
|
||||
- `--lora-path`: Path to LoRA adapter folder to pre-load at initialization (loads into cache with a stable ID derived from the path)
|
||||
- `--lora-request-path`: Path to LoRA adapter folder for per-request loading
|
||||
- `--lora-request-id`: Integer ID for the LoRA adapter (optional). If not provided and `--lora-request-path` is set, will derive a stable ID from the path.
|
||||
- `--lora-scale`: Scale factor for LoRA weights (default: 1.0). Higher values increase the influence of the LoRA adapter.
|
||||
|
||||
### Standard Parameters
|
||||
|
||||
- `--prompt`: Text prompt for image generation (required)
|
||||
- `--seed`: Random seed for reproducibility (default: 42)
|
||||
- `--height`: Image height in pixels (default: 1024)
|
||||
- `--width`: Image width in pixels (default: 1024)
|
||||
- `--num-inference-steps`: Number of denoising steps (default: 50)
|
||||
- `--output`: Output file path (default: `lora_output.png`)
|
||||
|
||||
## How LoRA Works
|
||||
|
||||
All LoRA adapters are handled uniformly:
|
||||
|
||||
1. **Initialization**: If `--lora-path` is provided, the adapter is loaded into cache with a stable ID derived from the adapter path
|
||||
2. **Per-request**: If `--lora-request-path` is provided, the adapter is loaded/activated for that request
|
||||
3. **No LoRA**: If no LoRA request is provided (`req.lora_request` is None), all adapters are deactivated
|
||||
|
||||
The system uses LRU cache management - adapters are cached and evicted when the cache is full (unless pinned).
|
||||
|
||||
## LoRA Adapter Format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
@@ -1,147 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
from vllm_omni.lora.utils import stable_lora_int_id
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Generate images with LoRA adapters.")
|
||||
parser.add_argument("--model", default="stabilityai/stable-diffusion-3.5-medium", help="Model name or path.")
|
||||
parser.add_argument("--prompt", required=True, help="Text prompt for image generation.")
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed for deterministic results.")
|
||||
parser.add_argument("--height", type=int, default=1024, help="Height of generated image.")
|
||||
parser.add_argument("--width", type=int, default=1024, help="Width of generated image.")
|
||||
parser.add_argument(
|
||||
"--num-inference-steps",
|
||||
type=int,
|
||||
default=50,
|
||||
help="Number of denoising steps for the diffusion sampler.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default="lora_output.png",
|
||||
help="Path to save the generated image (PNG).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-path",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to LoRA adapter folder to pre-load at initialization (PEFT format). "
|
||||
"Note: pre-loading populates the cache; you still need to pass a lora_request to activate it.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-request-path",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to LoRA adapter folder for per-request activation (dynamic LoRA). "
|
||||
"If --lora-request-id is not provided, a stable ID will be derived from this path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-request-id",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Integer ID for the LoRA adapter (for dynamic LoRA). "
|
||||
"If not provided and --lora-request-path is set, will derive a stable ID from the path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-scale",
|
||||
type=float,
|
||||
default=1.0,
|
||||
help="Scale factor for LoRA weights (default: 1.0).",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
model = args.model
|
||||
|
||||
omni_kwargs = {}
|
||||
|
||||
if args.lora_path:
|
||||
omni_kwargs["lora_path"] = args.lora_path
|
||||
print(f"Using static LoRA from: {args.lora_path}")
|
||||
|
||||
omni = Omni(model=model, **omni_kwargs)
|
||||
|
||||
lora_request = None
|
||||
if args.lora_request_path:
|
||||
if args.lora_request_id is None:
|
||||
lora_request_id = stable_lora_int_id(args.lora_request_path)
|
||||
else:
|
||||
lora_request_id = args.lora_request_id
|
||||
|
||||
lora_name = Path(args.lora_request_path).stem
|
||||
lora_request = LoRARequest(
|
||||
lora_name=lora_name,
|
||||
lora_int_id=lora_request_id,
|
||||
lora_path=args.lora_request_path,
|
||||
)
|
||||
print(f"Using per-request LoRA: name={lora_name}, id={lora_request_id}, scale={args.lora_scale}")
|
||||
elif args.lora_path:
|
||||
# pre-loaded LoRA
|
||||
lora_request_id = stable_lora_int_id(args.lora_path)
|
||||
lora_request = LoRARequest(
|
||||
lora_name="preloaded",
|
||||
lora_int_id=lora_request_id,
|
||||
lora_path=args.lora_path,
|
||||
)
|
||||
print(f"Activating pre-loaded LoRA: id={lora_request_id}, scale={args.lora_scale}")
|
||||
|
||||
sampling_params = OmniDiffusionSamplingParams(
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
num_inference_steps=args.num_inference_steps,
|
||||
)
|
||||
|
||||
if lora_request:
|
||||
sampling_params.lora_request = lora_request
|
||||
sampling_params.lora_scale = args.lora_scale
|
||||
|
||||
outputs = omni.generate(args.prompt, sampling_params)
|
||||
|
||||
if not outputs or len(outputs) == 0:
|
||||
raise ValueError("No output generated from omni.generate()")
|
||||
|
||||
if isinstance(outputs, list):
|
||||
first_output = outputs[0]
|
||||
else:
|
||||
first_output = outputs
|
||||
|
||||
images = None
|
||||
if hasattr(first_output, "images") and first_output.images:
|
||||
images = first_output.images
|
||||
elif hasattr(first_output, "request_output") and first_output.request_output:
|
||||
req_out = first_output.request_output
|
||||
if isinstance(req_out, list) and len(req_out) > 0:
|
||||
req_out = req_out[0]
|
||||
if hasattr(req_out, "images") and req_out.images:
|
||||
images = req_out.images
|
||||
|
||||
if not images:
|
||||
raise ValueError("No images found in request_output")
|
||||
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
suffix = output_path.suffix or ".png"
|
||||
stem = output_path.stem or "lora_output"
|
||||
if len(images) <= 1:
|
||||
images[0].save(output_path)
|
||||
print(f"Saved generated image to {output_path}")
|
||||
else:
|
||||
for idx, img in enumerate(images):
|
||||
save_path = output_path.parent / f"{stem}_{idx}{suffix}"
|
||||
img.save(save_path)
|
||||
print(f"Saved generated image to {save_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -133,6 +133,36 @@ python text_to_image.py \
|
||||
|
||||
> ℹ️ Qwen-Image currently publishes best-effort presets at `1328x1328`, `1664x928`, `928x1664`, `1472x1140`, `1140x1472`, `1584x1056`, and `1056x1584`. Adjust `--height/--width` accordingly for the most reliable outcomes.
|
||||
|
||||
## LoRA
|
||||
|
||||
This example supports Peft-compatible LoRA (Low-Rank Adaptation) adapters for diffusion models. Pass `--lora-path` to use a LoRA adapter and optionally `--lora-scale` (default 1.0); omit it to use the base model only.
|
||||
|
||||
### Basic usage with LoRA
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model Tongyi-MAI/Z-Image-Turbo \
|
||||
--prompt "A piece of cheesecake" \
|
||||
--lora-path /path/to/lora/ \
|
||||
--lora-scale 1.0 \
|
||||
--output output.png
|
||||
```
|
||||
|
||||
### LoRA parameters
|
||||
|
||||
- `--lora-path`: Path to LoRA adapter folder (PEFT format). Loaded at initialization and used for generation.
|
||||
- `--lora-scale`: Scale factor for LoRA weights (default: 1.0). Higher values increase the influence of the LoRA adapter.
|
||||
|
||||
### LoRA adapter format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
├── adapter_config.json
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Web UI Demo
|
||||
|
||||
Launch the gradio demo:
|
||||
|
||||
@@ -12,6 +12,8 @@ import torch
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig, logger
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.lora.request import LoRARequest
|
||||
from vllm_omni.lora.utils import stable_lora_int_id
|
||||
from vllm_omni.outputs import OmniRequestOutput
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
@@ -160,6 +162,28 @@ def parse_args() -> argparse.Namespace:
|
||||
default=1,
|
||||
help="Number of GPUs used for tensor parallelism (TP) inside the DiT.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-path",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to LoRA adapter folder (PEFT format). Loaded at initialization and used for generation.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--lora-scale",
|
||||
type=float,
|
||||
default=1.0,
|
||||
help="Scale factor for LoRA weights (default: 1.0).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--vae_use_slicing",
|
||||
action="store_true",
|
||||
help="Enable VAE slicing for memory optimization.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--vae_use_tiling",
|
||||
action="store_true",
|
||||
help="Enable VAE tiling for memory optimization.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--vae-patch-parallel-size",
|
||||
type=int,
|
||||
@@ -241,6 +265,12 @@ def main():
|
||||
# Check if profiling is requested via environment variable
|
||||
profiler_enabled = bool(os.getenv("VLLM_TORCH_PROFILER_DIR"))
|
||||
|
||||
# Prepare LoRA kwargs for Omni initialization
|
||||
lora_args: dict[str, Any] = {}
|
||||
if args.lora_path:
|
||||
lora_args["lora_path"] = args.lora_path
|
||||
print(f"Using LoRA from: {args.lora_path}")
|
||||
|
||||
# Build quantization kwargs: use quantization_config dict when
|
||||
# ignored_layers is specified so the list flows through OmniDiffusionConfig
|
||||
quant_kwargs: dict[str, Any] = {}
|
||||
@@ -253,18 +283,19 @@ def main():
|
||||
elif args.quantization:
|
||||
quant_kwargs["quantization"] = args.quantization
|
||||
|
||||
# Initialize Omni with model-specific settings
|
||||
omni_kwargs = {
|
||||
"model": args.model,
|
||||
"enable_layerwise_offload": args.enable_layerwise_offload,
|
||||
"layerwise_num_gpu_layers": args.layerwise_num_gpu_layers,
|
||||
"vae_use_slicing": args.vae_use_slicing,
|
||||
"vae_use_tiling": args.vae_use_tiling,
|
||||
"cache_backend": cache_backend,
|
||||
"cache_backend": args.cache_backend,
|
||||
"cache_config": cache_config,
|
||||
"enable_cache_dit_summary": args.enable_cache_dit_summary,
|
||||
"parallel_config": parallel_config,
|
||||
"enforce_eager": args.enforce_eager,
|
||||
"enable_cpu_offload": args.enable_cpu_offload,
|
||||
**lora_args,
|
||||
**quant_kwargs,
|
||||
}
|
||||
if use_nextstep:
|
||||
@@ -292,8 +323,20 @@ def main():
|
||||
)
|
||||
print(f" CPU offload: {args.enable_cpu_offload}")
|
||||
print(f" Image size: {args.width}x{args.height}")
|
||||
if args.lora_path:
|
||||
print(f" LoRA: scale={args.lora_scale}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
# Build LoRA request when --lora-path is set
|
||||
lora_request = None
|
||||
if args.lora_path:
|
||||
lora_request_id = stable_lora_int_id(args.lora_path)
|
||||
lora_request = LoRARequest(
|
||||
lora_name=Path(args.lora_path).stem,
|
||||
lora_int_id=lora_request_id,
|
||||
lora_path=args.lora_path,
|
||||
)
|
||||
|
||||
generation_start = time.perf_counter()
|
||||
|
||||
extra_args = {
|
||||
@@ -302,6 +345,10 @@ def main():
|
||||
"use_norm": args.use_norm,
|
||||
}
|
||||
|
||||
if lora_request:
|
||||
extra_args["lora_request"] = lora_request
|
||||
extra_args["lora_scale"] = args.lora_scale
|
||||
|
||||
outputs = omni.generate(
|
||||
{
|
||||
"prompt": args.prompt,
|
||||
|
||||
Reference in new issue
Block a user