vllm-omni/examples/online_serving/glm_image/README.md

# GLM-Image Online Serving

This example demonstrates how to deploy GLM-Image for online image generation using vLLM-Omni.

## 🛠️ Installation

Please refer to [README.md](../../../README.md)

## Run examples (GLM-Image)

**Note**: These examples work with the default configuration on **2× NVIDIA A100 (80GB)** or equivalent. Stage 0 (AR) and Stage 1 (Diffusion) each use one GPU by default. For single-GPU setups, modify the stage configuration to share the same device.

### Launch the Server

```bash
# Use default configuration
vllm serve zai-org/GLM-Image --omni --port 8091
```

Or use the convenience script:

```bash
cd examples/online_serving/glm_image
bash run_server.sh
```

If you have a custom stage configs file:

```bash
vllm serve zai-org/GLM-Image --omni --port 8091 --stage-configs-path /path/to/glm_image.yaml
```

### Send Requests

Get into the glm_image folder:

```bash
cd examples/online_serving/glm_image
```

Send request via Python:

```bash
python openai_chat_client.py --prompt "A cute cat sitting on a window sill"
```

The Python client supports the following command-line arguments:

- `--prompt` (or `-p`): Text prompt for generation (default: `A beautiful sunset over the ocean with sailing boats`)
- `--output` (or `-o`): Output file path (default: `glm_image_output.png`)
- `--server` (or `-s`): Server URL (default: `http://localhost:8091`)
- `--image` (or `-i`): Input image path (for image-to-image editing)
- `--height`: Image height in pixels (default: 1024)
- `--width`: Image width in pixels (default: 1024)
- `--steps`: Number of inference steps (default: 50)
- `--guidance-scale`: Classifier-free guidance scale (default: 1.5)
- `--seed`: Random seed (default: 42)
- `--negative`: Negative prompt

## Modality Control

GLM-Image supports **text-to-image** and **image-to-image** modes.

The default yaml configuration deploys AR on GPU 0 and DiT on GPU 1. You can use the default configuration file: [`glm_image.yaml`](../../../vllm_omni/model_executor/stage_configs/glm_image.yaml)

| Mode           | Input        | Output | Description                        |
| -------------- | ------------ | ------ | ---------------------------------- |
| Text-to-Image  | Text         | Image  | Generate images from text prompts  |
| Image-to-Image | Image + Text | Image  | Edit images with text instructions |

### Text-to-Image

```bash
python openai_chat_client.py \
    --prompt "A photorealistic mountain landscape at sunset" \
    --height 1024 \
    --width 1024 \
    --output landscape.png

# Or use the curl script:
bash run_curl_text_to_image.sh "A futuristic city skyline at night"
```

### Image-to-Image (Image Editing)

```bash
python openai_chat_client.py \
    --prompt "Convert this image to watercolor style" \
    --image input.png \
    --output watercolor.png

# Or use the curl script:
bash run_curl_image_edit.sh input.png "Convert to watercolor style"
```

For general-purpose request methods (curl, OpenAI SDK, Python `requests`), see
the [Text-to-Image](../text_to_image/README.md) and
[Image-to-Image](../image_to_image/README.md) guides.

## Generation Parameters

When using `/v1/chat/completions`, pass these inside `extra_body` in the curl
JSON, or via the `extra_body` keyword argument in the OpenAI Python SDK.
When using the dedicated `/v1/images/generations` or `/v1/images/edits`
endpoints, pass the supported generation controls as top-level fields directly.
For image dimensions and count, use `size` and `n` rather than `height` or
`width`.

| Parameter             | Type  | Default | Description                         |
| --------------------- | ----- | ------- | ----------------------------------- |
| `height`              | int   | 1024    | Image height in pixels              |
| `width`               | int   | 1024    | Image width in pixels               |
| `num_inference_steps` | int   | 50      | Number of diffusion denoising steps |
| `guidance_scale`      | float | 1.5     | Classifier-free guidance scale      |
| `seed`                | int   | None    | Optional random seed; `/v1/images/*` generates one server-side if omitted |
| `negative_prompt`     | str   | None    | Negative prompt                     |

## Response Format

```json
{
  "id": "chatcmpl-xxx",
  "created": 1234567890,
  "model": "zai-org/GLM-Image",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,..."
            }
          }
        ]
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {}
}
```

## Extract Image

```bash
# From a saved JSON response
cat response.json | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png
```

## Architecture

GLM-Image uses a 2-stage multistage pipeline:

```
Stage 0 (AR Model)                Stage 1 (Diffusion)
┌───────────────────┐            ┌─────────────────────┐
│ vLLM-optimized    │  prior     │  GlmImagePipeline   │
│ GlmImageFor       │──tokens──►│  ┌───────────────┐  │
│ Conditional       │            │  │ DiT Denoiser  │  │
│ Generation        │            │  └───────┬───────┘  │
│ (9B AR model)     │            │          ▼          │
└───────────────────┘            │  ┌───────────────┐  │
        ▲                        │  │  VAE Decode   │──┼──► Image
        │                        │  └───────────────┘  │
   Text / Image                  └─────────────────────┘
     Input
```

## VRAM Requirements

| Stage             | VRAM                   |
| :---------------- | :--------------------- |
| Stage-0 (AR)      | **~18 GiB + KV Cache** |
| Stage-1 (DiT+VAE) | **~20 GiB**            |
| Total             | **~38 GiB + KV Cache** |

## File Description

| File                        | Description                           |
| --------------------------- | ------------------------------------- |
| `run_server.sh`             | Server startup script                 |
| `run_curl_text_to_image.sh` | Text-to-image curl example            |
| `run_curl_image_edit.sh`    | Image-to-image (editing) curl example |
| `openai_chat_client.py`     | Python client (t2i + i2i)             |

## FAQ

- If you encounter OOM errors, adjust `gpu_memory_utilization` in the stage config:

```yaml
# In glm_image.yaml, reduce from default 0.6:
gpu_memory_utilization: 0.5
```

- The first request may be slow due to model warmup. Subsequent requests will be faster.

- If you encounter `Transformers does not recognize this architecture` error, your have to upgrade `transformers` package to `5.3.0` or above:

```
pip install --upgrade transformers
```