anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Benchmark and Bugfix for GLM-Image (#3024)

Browse Source 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Jared WenandHongsheng Liu authored and GitHub committed 2026-04-25 20:49:19 +08:00
1 parent 6b52db9e2b
commit 6e35bfbbae
18 files changed
+1836 -381
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/examples/offline_inference/glm_image.md
+70 -137
View File
@@ -1,154 +1,87 @@
# GLM-Image Multistage End-to-End Inference
# GLM-Image Offline Inference
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/glm_image>.
This example demonstrates how to run GLM-Image with the vLLM-Omni multistage architecture.
GLM-Image is a 2-stage image generation model (AR + Diffusion) supported by vLLM-Omni's
declarative config system. The pipeline topology and stage structure are declared in
`vllm_omni/model_executor/models/glm_image/pipeline.py`; deployment knobs live in
`vllm_omni/deploy/glm_image.yaml`.
## Architecture
GLM-Image uses a 2-stage pipeline:
```
┌─────────────────────────────────────────────────────────────┐
│ GLM-Image Pipeline │
├─────────────────────────────────────────────────────────────┤
│ │
│ Stage 0 (AR Model) Stage 1 (Diffusion) │
│ ┌─────────────────┐ ┌─────────────────────┐ │
│ │ vLLM-optimized │ │ GlmImagePipeline │ │
│ │ GlmImageFor │ prior │ ┌───────────────┐ │ │
│ │ Conditional │──tokens───►│ │ DiT Denoiser │ │ │
│ │ Generation │ │ └───────────────┘ │ │
│ │ (9B AR model) │ │ │ │ │
│ └─────────────────┘ │ ▼ │ │
│ ▲ │ ┌───────────────┐ │ │
│ │ │ │ VAE Decode │──┼──► Image
│ Text/Image │ └───────────────┘ │ │
│ Input └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Stage 0 (AR Model) Stage 1 (Diffusion)
┌───────────────────┐ ┌─────────────────────┐
│ vLLM-optimized │ prior │ GlmImagePipeline │
│ GlmImageFor │──tokens──►│ ┌───────────────┐ │
│ Conditional │ │ │ DiT Denoiser │ │
│ Generation │ │ └───────┬───────┘ │
│ (9B AR model) │ │ ▼ │
└───────────────────┘ │ ┌───────────────┐ │
▲ │ │ VAE Decode │──┼──► Image
│ │ └───────────────┘ │
Text / Image └─────────────────────┘
Input
```
## Features
## Text-to-Image
- **vLLM-optimized AR**: Uses PagedAttention and tensor parallelism for faster prior token generation
- **Flexible deployment**: AR and Diffusion stages can run on different GPUs
- **Text-to-Image**: Generate images from text descriptions
- **Image-to-Image**: Edit existing images with text prompts
```python
from vllm_omni.entrypoints.omni import Omni
## Usage
### Text-to-Image
```bash
python end2end.py \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "A beautiful sunset over the ocean with sailing boats" \
--height 1024 \
--width 1024 \
--output output_t2i.png
if __name__ == "__main__":
omni = Omni(model="zai-org/GLM-Image")
outputs = omni.generate(
"A photorealistic mountain landscape at sunset",
sampling_params={
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 1.5,
"seed": 42,
},
)
outputs[0].request_output.images[0].save("output.png")
```
### Image-to-Image (Image Editing)
## Image-to-Image (Image Editing)
```bash
python end2end.py \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "Transform this scene into a winter wonderland" \
--image input.png \
--output output_i2i.png
```python
from vllm_omni.entrypoints.omni import Omni
if __name__ == "__main__":
omni = Omni(model="zai-org/GLM-Image")
outputs = omni.generate(
{
"prompt": "Convert this image to watercolor style",
"multi_modal_data": {
"image": "input.png",
},
},
sampling_params={
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 1.5,
"seed": 42,
},
)
outputs[0].request_output.images[0].save("output.png")
```
### With Custom Parameters
## Generation Parameters
```bash
python end2end.py \
--model-path /path/to/glm-image \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "A photorealistic cat sitting on a window sill" \
--height 1024 \
--width 1024 \
--num-inference-steps 50 \
--guidance-scale 1.5 \
--seed 42 \
--output output.png
```
| Parameter | Type | Default | Description |
| --------------------- | ----- | ------- | ----------------------------------- |
| `height` | int | 1024 | Image height in pixels |
| `width` | int | 1024 | Image width in pixels |
| `num_inference_steps` | int | 50 | Number of diffusion denoising steps |
| `guidance_scale` | float | 1.5 | Classifier-free guidance scale |
| `seed` | int | None | Optional random seed |
| `negative_prompt` | str | None | Negative prompt |
## Shell Scripts
## VRAM Requirements
### Run Text-to-Image
```bash
./run_t2i.sh
```
### Run Image-to-Image
```bash
./run_i2i.sh --image /path/to/input.png
```
## Stage Configuration
The stage config (`glm_image.yaml`) defines:
- **Stage 0 (AR)**: Uses `GPUARWorker` with vLLM engine
- Model: `GlmImageForConditionalGeneration`
- Output: `token_ids` (prior tokens)
- **Stage 1 (Diffusion)**: Uses diffusion engine
- Model: `GlmImagePipeline`
- Output: Generated image
See `vllm_omni/model_executor/stage_configs/glm_image.yaml` for full configuration.
## Comparison with Single-Stage
| Aspect | Single-Stage (transformers) | Multistage (vLLM) |
| ----------- | --------------------------- | ------------------- |
| AR Model | transformers native | vLLM PagedAttention |
| Memory | Higher (no KV cache opt) | Lower (optimized) |
| Throughput | Lower | Higher |
| Flexibility | Single GPU | Multi-GPU support |
## Troubleshooting
### OOM Error
Try reducing memory usage:
```bash
# In glm_image.yaml, adjust:
gpu_memory_utilization: 0.5 # Reduce from 0.6
```
### Slow Initialization
The first run loads model weights. Subsequent runs are faster:
```bash
--stage-init-timeout 900 # Increase timeout for slow storage
```
## Requirements
- vLLM-Omni with GLM-Image support
- CUDA-capable GPU (recommended: H100/A100 with 80GB)
- GLM-Image model weights
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/glm_image/end2end.py"
``````
??? abstract "run_i2i.sh"
``````sh
--8<-- "examples/offline_inference/glm_image/run_i2i.sh"
``````
??? abstract "run_t2i.sh"
``````sh
--8<-- "examples/offline_inference/glm_image/run_t2i.sh"
``````
| Stage | VRAM |
| :---------------- | :--------------------- |
| Stage-0 (AR) | **~18 GiB + KV Cache** |
| Stage-1 (DiT+VAE) | **~20 GiB** |
| Total | **~38 GiB + KV Cache** |
docs/user_guide/examples/online_serving/glm_image.md
+75 -107
View File
@@ -1,99 +1,96 @@
# GLM-Image Online Serving
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/glm_image>.
GLM-Image is a 2-stage image generation model (AR + Diffusion) supported by vLLM-Omni's
declarative config system. The pipeline topology and stage structure are declared in
`vllm_omni/model_executor/models/glm_image/pipeline.py`; deployment knobs (GPU placement,
memory, sampling params) live in `vllm_omni/deploy/glm_image.yaml`.
This example demonstrates how to deploy GLM-Image for online image generation using vLLM-Omni.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
## Run examples (GLM-Image)
**Note**: These examples work with the default configuration on **2× NVIDIA A100 (80GB)** or equivalent. Stage 0 (AR) and Stage 1 (Diffusion) each use one GPU by default. For single-GPU setups, modify the stage configuration to share the same device.
### Launch the Server
## Start Server
```bash
# Use default configuration
vllm serve zai-org/GLM-Image --omni --port 8091
```
Or use the convenience script:
The config system auto-detects the pipeline from the model's `model_index.json` — no
manual `--stage-configs-path` or `--deploy-config` needed.
By default, stage 0 (AR) runs on GPU 0 and stage 1 (Diffusion) on GPU 1. To colocate
both stages on a single GPU, override per stage:
```bash
cd examples/online_serving/glm_image
bash run_server.sh
vllm serve zai-org/GLM-Image --omni --port 8091 \
--stage-0-devices 0 --stage-1-devices 0
```
If you have a custom stage configs file:
```bash
vllm serve zai-org/GLM-Image --omni --port 8091 --stage-configs-path /path/to/glm_image.yaml
```
### Send Requests
Get into the glm_image folder:
```bash
cd examples/online_serving/glm_image
```
Send request via Python:
```bash
python openai_chat_client.py --prompt "A cute cat sitting on a window sill"
```
The Python client supports the following command-line arguments:
- `--prompt` (or `-p`): Text prompt for generation (default: `A beautiful sunset over the ocean with sailing boats`)
- `--output` (or `-o`): Output file path (default: `glm_image_output.png`)
- `--server` (or `-s`): Server URL (default: `http://localhost:8091`)
- `--image` (or `-i`): Input image path (for image-to-image editing)
- `--height`: Image height in pixels (default: 1024)
- `--width`: Image width in pixels (default: 1024)
- `--steps`: Number of inference steps (default: 50)
- `--guidance-scale`: Classifier-free guidance scale (default: 1.5)
- `--seed`: Random seed (default: 42)
- `--negative`: Negative prompt
## Modality Control
GLM-Image supports **text-to-image** and **image-to-image** modes.
The default yaml configuration deploys AR on GPU 0 and DiT on GPU 1. You can use the default configuration file: [`glm_image.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/glm_image.yaml)
| Mode | Input | Output | Description |
| -------------- | ------------ | ------ | ---------------------------------- |
| Text-to-Image | Text | Image | Generate images from text prompts |
| Image-to-Image | Image + Text | Image | Edit images with text instructions |
## API Calls
### Text-to-Image
```bash
python openai_chat_client.py \
--prompt "A photorealistic mountain landscape at sunset" \
--height 1024 \
--width 1024 \
--output landscape.png
# Or use the curl script:
bash run_curl_text_to_image.sh "A futuristic city skyline at night"
curl -s http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "A photorealistic mountain landscape at sunset"}
],
"extra_body": {
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 1.5,
"seed": 42
}
}' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png
```
### Image-to-Image (Image Editing)
```bash
python openai_chat_client.py \
--prompt "Convert this image to watercolor style" \
--image input.png \
--output watercolor.png
curl -s http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Convert this image to watercolor style"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,$(base64 -w0 input.png)}"}
]
}
],
"extra_body": {
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 1.5
}
}' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png
```
# Or use the curl script:
bash run_curl_image_edit.sh input.png "Convert to watercolor style"
### Using the OpenAI Python SDK
```python
from openai import OpenAI
import base64
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.chat.completions.create(
model="zai-org/GLM-Image",
messages=[{"role": "user", "content": "A beautiful sunset over the ocean"}],
extra_body={
"height": 1024,
"width": 1024,
"num_inference_steps": 50,
"guidance_scale": 1.5,
"seed": 42,
},
)
img_url = response.choices[0].message.content[0].image_url.url
_, b64_data = img_url.split(",", 1)
with open("output.png", "wb") as f:
f.write(base64.b64decode(b64_data))
```
For general-purpose request methods (curl, OpenAI SDK, Python `requests`), see
@@ -116,7 +113,7 @@ dimensions and count, use `size` and `n` rather than `height` or `width`.
| `width` | int | 1024 | Image width in pixels |
| `num_inference_steps` | int | 50 | Number of diffusion denoising steps |
| `guidance_scale` | float | 1.5 | Classifier-free guidance scale |
| `seed` | int | None | Optional random seed; `/v1/images/*` generates one server-side if omitted |
| `seed` | int | None | Optional random seed |
| `negative_prompt` | str | None | Negative prompt |
## Response Format
@@ -150,13 +147,12 @@ dimensions and count, use `size` and `n` rather than `height` or `width`.
## Extract Image
```bash
# From a saved JSON response
cat response.json | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png
```
## Architecture
GLM-Image uses a 2-stage multistage pipeline:
GLM-Image uses a 2-stage pipeline:
```
Stage 0 (AR Model) Stage 1 (Diffusion)
@@ -181,41 +177,13 @@ Stage 0 (AR Model) Stage 1 (Diffusion)
| Stage-1 (DiT+VAE) | **~20 GiB** |
| Total | **~38 GiB + KV Cache** |
## File Description
| File | Description |
| --------------------------- | ------------------------------------- |
| `run_server.sh` | Server startup script |
| `run_curl_text_to_image.sh` | Text-to-image curl example |
| `run_curl_image_edit.sh` | Image-to-image (editing) curl example |
| `openai_chat_client.py` | Python client (t2i + i2i) |
## FAQ
- If you encounter OOM errors, adjust `gpu_memory_utilization` in the stage config:
- If you encounter OOM errors, adjust `gpu_memory_utilization` in the deploy config:
```yaml
# In glm_image.yaml, reduce from default 0.6:
# In vllm_omni/deploy/glm_image.yaml, reduce from default 0.6:
gpu_memory_utilization: 0.5
```
- The first request may be slow due to model warmup. Subsequent requests will be faster.
## Example materials
??? abstract "openai_chat_client.py"
``````py
--8<-- "examples/online_serving/glm_image/openai_chat_client.py"
``````
??? abstract "run_curl_image_edit.sh"
``````sh
--8<-- "examples/online_serving/glm_image/run_curl_image_edit.sh"
``````
??? abstract "run_curl_text_to_image.sh"
``````sh
--8<-- "examples/online_serving/glm_image/run_curl_text_to_image.sh"
``````
??? abstract "run_server.sh"
``````sh
--8<-- "examples/online_serving/glm_image/run_server.sh"
``````