[Doc] Sync and fix. (#2110)
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
This commit is contained in:
Alicia authored and GitHub committed
1 parent
9bf8f8ce79
commit
b0efb6ede0
23 files changed
+1382
-141
No files matched your search
@@ -3,14 +3,50 @@
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_image>.
|
||||
|
||||
|
||||
This folder provides several entrypoints for experimenting with `Qwen/Qwen-Image` `Qwen/Qwen-Image-2512` `Tongyi-MAI/Z-Image-Turbo` `stepfun-ai/NextStep-1.1` using vLLM-Omni, note that NextStep-1.1 has different architecture so we treat it differently regarding running arguments and pipeline.
|
||||
Generate images from text prompts using vLLM-Omni's diffusion pipeline entrypoints.
|
||||
|
||||
- `text_to_image.py`: command-line script for single image generation with advanced options.
|
||||
- `web_demo.py`: lightweight Gradio UI for interactive prompt/seed/CFG exploration.
|
||||
- `gradio_demo.py`: lightweight Gradio UI for interactive prompt/seed/CFG exploration.
|
||||
|
||||
Note that when you pass in multiple independent prompts, they will be processed sequentially. Batching requests is currently not supported.
|
||||
## Table of Contents
|
||||
|
||||
## Basic Usage
|
||||
- [Overview](#overview)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Key Arguments](#key-arguments)
|
||||
- [More CLI Examples](#more-cli-examples)
|
||||
- [Web UI Demo](#web-ui-demo)
|
||||
|
||||
## Overview
|
||||
|
||||
This folder provides several entrypoints for experimenting with text-to-image diffusion models using vLLM-Omni. Note that `NextStep-1.1` has a different architecture, so it is treated differently regarding running arguments and pipeline.
|
||||
|
||||
### Supported Models
|
||||
|
||||
| Model | Image Shape | Peak VRAM (GiB) * | Model Weights (GiB) |
|
||||
| ----- | ----------- | ----------- | ----------------- |
|
||||
| `Qwen/Qwen-Image` | 1024 x 1024 | 60.0 | 53.7 |
|
||||
| `Qwen/Qwen-Image-2512` |1024 x 1024 | 60.0 | 53.7 |
|
||||
| `Tongyi-MAI/Z-Image-Turbo` | 1024 x 1024 | 24.8 | 19.2 |
|
||||
| `stepfun-ai/NextStep-1.1` | 512 x 512 | 71.8 | 28.1 |
|
||||
| `meituan-longcat/LongCat-Image` | 1024 x 1024 | 71.2 | 27.3 |
|
||||
| `AIDC-AI/Ovis-Image-7B` | 1024 x 1024 | 71.8 | 17.1 |
|
||||
| `OmniGen2/OmniGen2` | 1024 x 1024 | 20.1 | 14.7 |
|
||||
| `stabilityai/stable-diffusion-3.5-medium` | 1024 x 1024 | 20.1 | 15.6 |
|
||||
| `black-forest-labs/FLUX.1-dev` | 1024 x 1024 | 77.6 | 31.4 |
|
||||
| `black-forest-labs/FLUX.2-klein-4B` | 1024 x 1024 | 72.7 | 14.9 |
|
||||
| `black-forest-labs/FLUX.2-klein-9B` | 1024 x 1024 | 37.1 | 32.3 |
|
||||
| `black-forest-labs/FLUX.2-dev` | 1024 x 1024 | 65.7 | >80 (CPU offload required) |
|
||||
|
||||
!!! info
|
||||
*Peak VRAM: based on basic single-card usage, batch size =1, without any acceleration/optimization features. FLUX.2-dev requires `--enable-cpu-offload` on a single 80 GiB GPU.
|
||||
|
||||
Default model: `Qwen/Qwen-Image`
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Python API
|
||||
|
||||
Single-prompt generation:
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
@@ -23,78 +59,74 @@ if __name__ == "__main__":
|
||||
images[0].save("coffee.png")
|
||||
```
|
||||
|
||||
Or put more than one prompt in a request.
|
||||
### Local CLI Usage
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
prompts = [
|
||||
"a cup of coffee on a table",
|
||||
"a toy dinosaur on a sandy beach",
|
||||
"a fox waking up in bed and yawning",
|
||||
]
|
||||
outputs = omni.generate(prompts)
|
||||
for i, output in enumerate(outputs):
|
||||
image = output.request_output.images[0].save(f"{i}.jpg")
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "a cup of coffee on the table" \
|
||||
--output coffee.png
|
||||
```
|
||||
|
||||
!!! info
|
||||
## Key Arguments
|
||||
|
||||
However, it is not currently recommended to do so
|
||||
because not all models support batch inference,
|
||||
and batch requesting mostly does not provide significant performance improvement (despite the impression that it does).
|
||||
This feature is primarily for the sake of interface compatibility with vLLM and to allow for future improvements.
|
||||
**Common arguments:**
|
||||
|
||||
!!! info
|
||||
| Argument | Type | Default | Description |
|
||||
| -------- | ---- | ------- | ----------- |
|
||||
| `--prompt` | str | `"a cup of coffee on the table"` | Text description for image generation |
|
||||
| `--seed` | int | `142` | Integer seed for deterministic sampling |
|
||||
| `--negative-prompt` | str | `None` | Negative prompt for classifier-free conditional guidance |
|
||||
| `--cfg-scale` | float | `4.0` | True CFG scale (model-specific guidance strength) |
|
||||
| `--guidance-scale` | float | `1.0` | Classifier-free guidance scale |
|
||||
| `--num-images-per-prompt` | int | `1` | Number of images per prompt (saved as `output`, `output_1`, ...) |
|
||||
| `--num-inference-steps` | int | `50` | Diffusion sampling steps (more steps = higher quality, slower) |
|
||||
| `--height` | int | `1024` | Output image height in pixels |
|
||||
| `--width` | int | `1024` | Output image width in pixels |
|
||||
| `--output` | str | `"qwen_image_output.png"` | Path to save the generated image |
|
||||
| `--vae-use-slicing` | flag | off | Enable VAE slicing for memory optimization |
|
||||
| `--vae-use-tiling` | flag | off | Enable VAE tiling for memory optimization |
|
||||
| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
|
||||
| `--enable-cpu-offload` | flag | off | Enable CPU offloading for diffusion models |
|
||||
| `--lora-path` | str | — | Path to PEFT LoRA adapter folder |
|
||||
| `--lora-scale` | float | `1.0` | Scale factor for LoRA weights |
|
||||
|
||||
For diffusion pipelines, the stage config field `stage_args.[].engine_args.max_num_seqs` is 1 by default, and the input
|
||||
list is sliced into single-item requests before feeding into the diffusion pipeline. For models that do internally support
|
||||
batched inputs, you can [modify this configuration](https://github.com/vllm-project/vllm-omni/tree/main/configuration/stage_configs.md) to let the model accept a longer batch of prompts.
|
||||
**NextStep-1.1 specific arguments:**
|
||||
|
||||
Apart from string prompt, vLLM-Omni also supports dictionary prompts in the same style as vLLM.
|
||||
This is useful for models that support negative prompts.
|
||||
| Argument | Type | Default | Description |
|
||||
| -------- | ---- | ------- | ----------- |
|
||||
| `--guidance-scale-2` | float | `1.0` | Secondary guidance scale (e.g. image-level CFG) |
|
||||
| `--timesteps-shift` | float | `1.0` | Timesteps shift parameter for sampling |
|
||||
| `--cfg-schedule` | str | `"constant"` | CFG schedule type: `"constant"` or `"linear"` |
|
||||
| `--use-norm` | flag | off | Apply layer normalization to sampled tokens |
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
> If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage.
|
||||
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
outputs = omni.generate([
|
||||
{
|
||||
"prompt": "a cup of coffee on a table"，
|
||||
"negative_prompt": "low resolution"
|
||||
},
|
||||
{
|
||||
"prompt": "a toy dinosaur on a sandy beach"，
|
||||
"negative_prompt": "cinematic, realistic"
|
||||
}
|
||||
])
|
||||
for i, output in enumerate(outputs):
|
||||
image = output.request_output.images[0].save(f"{i}.jpg")
|
||||
```
|
||||
> Qwen-Image currently publishes best-effort presets at `1328x1328`, `1664x928`, `928x1664`, `1472x1140`, `1140x1472`, `1584x1056`, and `1056x1584`. Adjust `--height/--width` accordingly for the most reliable outcomes.
|
||||
|
||||
## Local CLI Usage
|
||||
## More CLI Examples
|
||||
|
||||
### Qwen/Tongyi Models
|
||||
### Tongyi Models
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model Tongyi-MAI/Z-Image-Turbo \
|
||||
--prompt "a cup of coffee on the table" \
|
||||
--seed 42 \
|
||||
--cfg-scale 4.0 \
|
||||
--guidance-scale 0.0 \
|
||||
--num-images-per-prompt 1 \
|
||||
--num-inference-steps 50 \
|
||||
--num-inference-steps 9 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--output outputs/coffee.png
|
||||
```
|
||||
|
||||
`Tongyi-MAI/Z-Image-Turbo` is a distilled version of Z-Image. Distilled diffusion models usually require less number of inference steps (4~9), and Classifier-Free Guidance (CFG) is usually NOT applied. Similar distilled models are `black-forest-labs/FLUX.2-klein-4B` and `black-forest-labs/FLUX.2-klein-9B`.
|
||||
|
||||
### NextStep Models
|
||||
|
||||
NextStep-1.1 can have extra arguments
|
||||
NextStep-1.1 supports extra arguments for dual-level CFG control:
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
--model stepfun-ai/NextStep-1.1 \
|
||||
@@ -109,38 +141,101 @@ python text_to_image.py \
|
||||
--seed 42
|
||||
```
|
||||
|
||||
### Key Arguments
|
||||
### FLUX.2-dev Models
|
||||
|
||||
**Common arguments:**
|
||||
To run FLUX.2-dev on a single GPU, `--enable-cpu-offload` is required because the model weights exceed 80 GiB:
|
||||
|
||||
- `--prompt`: text description (string).
|
||||
- `--seed`: integer seed for deterministic sampling.
|
||||
- `--cfg-scale`: true CFG scale (model-specific guidance strength).
|
||||
- `--num-images-per-prompt`: number of images to generate per prompt (saves as `output`, `output_1`, ...).
|
||||
- `--num-inference-steps`: diffusion sampling steps (more steps = higher quality, slower).
|
||||
- `--height/--width`: output resolution (defaults 1024x1024).
|
||||
- `--output`: path to save the generated PNG.
|
||||
- `--vae-use-slicing`: enable VAE slicing for memory optimization.
|
||||
- `--vae-use-tiling`: enable VAE tiling for memory optimization.
|
||||
- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion_acceleration.md#using-cfg-parallel).
|
||||
- `--enable-cpu-offload`: enable CPU offloading for diffusion models.
|
||||
- `--guidance-scale`: classifier-free guidance scale.
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model black-forest-labs/FLUX.2-dev \
|
||||
--prompt "a lovely bunny holding a sign that says 'vllm-omni'" \
|
||||
--seed 42 \
|
||||
--tensor-parallel-size 1 \
|
||||
--num-images-per-prompt 1 \
|
||||
--num-inference-steps 50 \
|
||||
--guidance-scale 4.0 \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--enable-cpu-offload \
|
||||
--output flux2-dev.png
|
||||
```
|
||||
|
||||
**NextStep-1.1 specific:**
|
||||
- `--guidance-scale-2`: secondary guidance scale, e.g. image-level CFG (default: 1.0).
|
||||
- `--timesteps-shift`: timesteps shift parameter for sampling (default: 1.0).
|
||||
- `--cfg-schedule`: CFG schedule type, "constant" or "linear" (default: "constant").
|
||||
- `--use-norm`: apply layer normalization to sampled tokens.
|
||||
### Batch Requests (Multiple Prompts)
|
||||
|
||||
> ℹ️ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage.
|
||||
You can pass multiple prompts in a single `generate` call.
|
||||
|
||||
> ℹ️ Qwen-Image currently publishes best-effort presets at `1328x1328`, `1664x928`, `928x1664`, `1472x1140`, `1140x1472`, `1584x1056`, and `1056x1584`. Adjust `--height/--width` accordingly for the most reliable outcomes.
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
## LoRA
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
prompts = [
|
||||
"a cup of coffee on a table",
|
||||
"a toy dinosaur on a sandy beach",
|
||||
"a fox waking up in bed and yawning",
|
||||
]
|
||||
outputs = omni.generate(prompts)
|
||||
for i, output in enumerate(outputs):
|
||||
output.request_output.images[0].save(f"{i}.jpg")
|
||||
```
|
||||
|
||||
This example supports Peft-compatible LoRA (Low-Rank Adaptation) adapters for diffusion models. Pass `--lora-path` to use a LoRA adapter and optionally `--lora-scale` (default 1.0); omit it to use the base model only.
|
||||
!!! info
|
||||
|
||||
### Basic usage with LoRA
|
||||
Not all models support batch inference, and batch requesting mostly does not provide significant
|
||||
performance improvement. This feature is primarily for interface compatibility with vLLM and to
|
||||
allow for future improvements.
|
||||
|
||||
!!! info
|
||||
|
||||
For diffusion pipelines, the stage config field `stage_args.[].runtime.max_batch_size` is 1 by
|
||||
default, and the input list is sliced into single-item requests before feeding into the diffusion
|
||||
pipeline. For models that do internally support batched inputs, you can
|
||||
[modify this configuration](https://github.com/vllm-project/vllm-omni/tree/main/configuration/stage_configs.md) to let the model accept a
|
||||
longer batch of prompts.
|
||||
|
||||
### Negative Prompts
|
||||
|
||||
vLLM-Omni supports dictionary prompts for models that accept negative prompts:
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="Qwen/Qwen-Image")
|
||||
outputs = omni.generate([
|
||||
{
|
||||
"prompt": "a cup of coffee on a table",
|
||||
"negative_prompt": "low resolution"
|
||||
},
|
||||
{
|
||||
"prompt": "a toy dinosaur on a sandy beach",
|
||||
"negative_prompt": "cinematic, realistic"
|
||||
}
|
||||
])
|
||||
for i, output in enumerate(outputs):
|
||||
output.request_output.images[0].save(f"{i}.jpg")
|
||||
```
|
||||
|
||||
You can also pass a negative prompt via the CLI argument `--negative-prompt`:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||||
--model Qwen/Qwen-Image \
|
||||
--prompt "a cup of coffee on a table" \
|
||||
--negative-prompt "low resolution, blurry" \
|
||||
--output coffee.png
|
||||
```
|
||||
|
||||
### Advanced Features
|
||||
|
||||
#### CFG Parallel
|
||||
|
||||
Set `--cfg-parallel-size 2` to enable CFG Parallel for faster inference on multi-GPU setups.
|
||||
See more examples in the [diffusion acceleration user guide](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion_acceleration.md#using-cfg-parallel).
|
||||
|
||||
#### LoRA
|
||||
|
||||
This example supports PEFT-compatible LoRA (Low-Rank Adaptation) adapters for diffusion models. Pass `--lora-path` to use a LoRA adapter and optionally `--lora-scale` (default `1.0`); omit it to use the base model only.
|
||||
|
||||
```bash
|
||||
python text_to_image.py \
|
||||
@@ -151,14 +246,7 @@ python text_to_image.py \
|
||||
--output output.png
|
||||
```
|
||||
|
||||
### LoRA parameters
|
||||
|
||||
- `--lora-path`: Path to LoRA adapter folder (PEFT format). Loaded at initialization and used for generation.
|
||||
- `--lora-scale`: Scale factor for LoRA weights (default: 1.0). Higher values increase the influence of the LoRA adapter.
|
||||
|
||||
### LoRA adapter format
|
||||
|
||||
LoRA adapters must be in PEFT (Parameter-Efficient Fine-Tuning) format. A typical LoRA adapter directory structure:
|
||||
LoRA adapters must be in PEFT format. A typical adapter directory structure:
|
||||
|
||||
```
|
||||
lora_adapter/
|
||||
@@ -168,13 +256,13 @@ lora_adapter/
|
||||
|
||||
## Web UI Demo
|
||||
|
||||
Launch the gradio demo:
|
||||
Launch the Gradio demo:
|
||||
|
||||
```bash
|
||||
python gradio_demo.py --port 7862
|
||||
```
|
||||
|
||||
Then open `http://localhost:7862/` on your local browser to interact with the web UI.
|
||||
Then open `http://localhost:7862/` in your local browser to interact with the web UI.
|
||||
|
||||
## Example materials
|
||||
|
||||
|
||||
Reference in new issue
Block a user