e49fbd8a2d
Signed-off-by: Semmer <semmer@live.cn> Signed-off-by: jader <yjader@foxmail.com> Signed-off-by: asukaqaq-s <1311722138@qq.com> Co-authored-by: asukaqaq-s <1311722138@qq.com> Co-authored-by: jader <yjader@foxmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
131 lines
3.7 KiB
Markdown
131 lines
3.7 KiB
Markdown
# Qwen-Image for text-to-image serving with optional continuous batching on 1x A100 80GB
|
|
|
|
## Summary
|
|
|
|
- Vendor: Qwen
|
|
- Model: `Qwen/Qwen-Image`
|
|
- Task: Text-to-image generation
|
|
- Mode: Online serving with optional step-wise continuous batching
|
|
- Maintainer: Community
|
|
|
|
## When to use this recipe
|
|
|
|
Use this recipe when you want a known-good starting point for serving
|
|
`Qwen/Qwen-Image` on a single 80 GB A100, validate the normal online-serving
|
|
path, and optionally replay the benefit of step-wise continuous batching with
|
|
the benchmark assets already bundled in this repository.
|
|
|
|
## References
|
|
|
|
- Upstream or canonical docs:
|
|
[`docs/user_guide/examples/online_serving/text_to_image.md`](../../docs/user_guide/examples/online_serving/text_to_image.md)
|
|
- Related example under `examples/`:
|
|
[`examples/online_serving/text_to_image/README.md`](../../examples/online_serving/text_to_image/README.md)
|
|
- Related benchmark:
|
|
[`benchmarks/diffusion/diffusion_benchmark_serving.py`](../../benchmarks/diffusion/diffusion_benchmark_serving.py)
|
|
|
|
## Hardware Support
|
|
|
|
This recipe currently documents one CUDA GPU serving configuration. Extend it
|
|
with more hardware sections as community validation lands.
|
|
|
|
## GPU
|
|
|
|
### 1x A100 80GB
|
|
|
|
#### Environment
|
|
|
|
- OS: Linux
|
|
- Python: 3.10+
|
|
- Driver / runtime: NVIDIA CUDA environment with an A100 80 GB GPU
|
|
- vLLM version: Match the repository requirements for your checkout
|
|
- vLLM-Omni version or commit: Use the commit you are deploying from
|
|
|
|
#### Command
|
|
|
|
Start the baseline server:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen-Image --omni --port 8091
|
|
```
|
|
|
|
To enable the step-wise runtime without batching:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen-Image --omni \
|
|
--port 8091 \
|
|
--step-execution \
|
|
--max-num-seqs 1
|
|
```
|
|
|
|
To enable experimental compatible-request batching:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen-Image --omni \
|
|
--port 8091 \
|
|
--step-execution \
|
|
--max-num-seqs 8
|
|
```
|
|
|
|
You can also use the example launcher and pass the extra flags through:
|
|
|
|
```bash
|
|
bash examples/online_serving/text_to_image/run_server.sh --step-execution --max-num-seqs 8
|
|
```
|
|
|
|
#### Verification
|
|
|
|
Run the existing client example after the server is ready:
|
|
|
|
```bash
|
|
python examples/online_serving/text_to_image/openai_chat_client.py \
|
|
--server http://localhost:8091 \
|
|
--prompt "A ceramic teapot on a wooden table" \
|
|
--output /tmp/qwen_image_recipe.png
|
|
```
|
|
|
|
For a direct API smoke test:
|
|
|
|
```bash
|
|
curl http://localhost:8091/v1/images/generations \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Qwen/Qwen-Image",
|
|
"prompt": "A ceramic teapot on a wooden table",
|
|
"size": "1024x1024",
|
|
"num_inference_steps": 20,
|
|
"seed": 42
|
|
}'
|
|
```
|
|
|
|
To replay the batching benefit with matched warmup:
|
|
|
|
```bash
|
|
python benchmarks/diffusion/diffusion_benchmark_serving.py \
|
|
--backend vllm-omni \
|
|
--dataset vbench \
|
|
--task t2i \
|
|
--model Qwen/Qwen-Image \
|
|
--num-prompts 10 \
|
|
--height 1024 \
|
|
--width 1024 \
|
|
--num-inference-steps 50 \
|
|
--seed 42 \
|
|
--port 8089 \
|
|
--max-concurrency 8 \
|
|
--warmup-requests 8 \
|
|
--warmup-concurrency 8 \
|
|
--warmup-num-inference-steps 3 \
|
|
--disable-tqdm
|
|
```
|
|
|
|
Run that once against `--max-num-seqs 1`, then rerun it against `--max-num-seqs 8`
|
|
and compare the output JSON or terminal metrics.
|
|
|
|
#### Notes
|
|
|
|
- Memory usage: keep headroom for first-request compile and image decode overhead.
|
|
- Key flags: `--step-execution` enables the step-wise runtime; `--max-num-seqs` controls how many compatible requests may stay active together.
|
|
- Keep `--max-num-seqs 1` when you want the more conservative path, when traffic is mostly single-request, or when you are debugging correctness before measuring throughput.
|
|
- Current batching is still shape-sensitive: different step progress can co-batch, but different resolutions do not yet co-batch.
|