anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[ConfigRefactor] GLM-Image (#2977)

Browse Source 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: Jared Wen <w13431838023@gmail.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
This commit is contained in:
Jared WenandSYLAR authored and GitHub committed 2026-04-22 15:09:28 +08:00
1 parent e1cdd0c23b
commit 554233208f
16 files changed
+142 -1491
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/glm_image/README.md
-145
View File
@@ -1,145 +0,0 @@
# GLM-Image Multistage End-to-End Inference
This example demonstrates how to run GLM-Image with the vLLM-Omni multistage architecture.
## Architecture
GLM-Image uses a 2-stage pipeline:
```
┌─────────────────────────────────────────────────────────────┐
│ GLM-Image Pipeline │
├─────────────────────────────────────────────────────────────┤
│ │
│ Stage 0 (AR Model) Stage 1 (Diffusion) │
│ ┌─────────────────┐ ┌─────────────────────┐ │
│ │ vLLM-optimized │ │ GlmImagePipeline │ │
│ │ GlmImageFor │ prior │ ┌───────────────┐ │ │
│ │ Conditional │──tokens───►│ │ DiT Denoiser │ │ │
│ │ Generation │ │ └───────────────┘ │ │
│ │ (9B AR model) │ │ │ │ │
│ └─────────────────┘ │ ▼ │ │
│ ▲ │ ┌───────────────┐ │ │
│ │ │ │ VAE Decode │──┼──► Image
│ Text/Image │ └───────────────┘ │ │
│ Input └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
## Features
- **vLLM-optimized AR**: Uses PagedAttention and tensor parallelism for faster prior token generation
- **Flexible deployment**: AR and Diffusion stages can run on different GPUs
- **Text-to-Image**: Generate images from text descriptions
- **Image-to-Image**: Edit existing images with text prompts
## Usage
### Text-to-Image
```bash
python end2end.py \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "A beautiful sunset over the ocean with sailing boats" \
--height 1024 \
--width 1024 \
--output output_t2i.png
```
### Image-to-Image (Image Editing)
```bash
python end2end.py \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "Transform this scene into a winter wonderland" \
--image input.png \
--output output_i2i.png
```
### With Custom Parameters
```bash
python end2end.py \
--model-path /path/to/glm-image \
--config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \
--prompt "A photorealistic cat sitting on a window sill" \
--height 1024 \
--width 1024 \
--num-inference-steps 50 \
--guidance-scale 1.5 \
--seed 42 \
--output output.png
```
## Shell Scripts
### Run Text-to-Image
```bash
./run_t2i.sh
```
### Run Image-to-Image
```bash
./run_i2i.sh --image /path/to/input.png
```
## Stage Configuration
The stage config (`glm_image.yaml`) defines:
- **Stage 0 (AR)**: Uses `GPUARWorker` with vLLM engine
- Model: `GlmImageForConditionalGeneration`
- Output: `token_ids` (prior tokens)
- **Stage 1 (Diffusion)**: Uses diffusion engine
- Model: `GlmImagePipeline`
- Output: Generated image
See `vllm_omni/model_executor/stage_configs/glm_image.yaml` for full configuration.
## Comparison with Single-Stage
| Aspect | Single-Stage (transformers) | Multistage (vLLM) |
| ----------- | --------------------------- | ------------------- |
| AR Model | transformers native | vLLM PagedAttention |
| Memory | Higher (no KV cache opt) | Lower (optimized) |
| Throughput | Lower | Higher |
| Flexibility | Single GPU | Multi-GPU support |
## Troubleshooting
### OOM Error
Try reducing memory usage:
```bash
# In glm_image.yaml, adjust:
gpu_memory_utilization: 0.5 # Reduce from 0.6
```
### Slow Initialization
The first run loads model weights. Subsequent runs are faster:
```bash
--stage-init-timeout 900 # Increase timeout for slow storage
```
### `Transformers does not recognize this architecture` Error
Your have to upgrade `transformers` package to `5.3.0` or above:
```
pip install --upgrade transformers
```
## Requirements
- vLLM-Omni with GLM-Image support
- CUDA-capable GPU (recommended: H100/A100 with 80GB)
- GLM-Image model weights
- `transformers` v5.3.0 or above
examples/offline_inference/glm_image/end2end.py
-525
View File
@@ -1,525 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
End-to-end offline inference example for GLM-Image with multistage architecture.
This script tests the multistage pipeline where:
- Stage 0 (AR): vLLM-optimized GlmImageForConditionalGeneration generates prior_token_ids
- Stage 1 (Diffusion): GlmImagePipeline performs DiT denoising + VAE decode
Usage (text-to-image):
python end2end.py \
--model-path /path/to/glm-image \
--config-path /path/to/glm_image.yaml \
--prompt "A beautiful sunset over the ocean" \
--output output_t2i.png
Usage (image-to-image / image edit):
python end2end.py \
--model-path /path/to/glm-image \
--config-path /path/to/glm_image.yaml \
--prompt "Make it look like winter" \
--image input.png \
--output output_i2i.png
Usage (with custom parameters):
python end2end.py \
--model-path /path/to/glm-image \
--config-path /path/to/glm_image.yaml \
--prompt "A cat sitting on a window sill" \
--height 1024 \
--width 1024 \
--num-inference-steps 50 \
--guidance-scale 1.5 \
--seed 42
For more options, run:
python end2end.py --help
"""
import argparse
import os
import time
from pathlib import Path
from PIL import Image
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
# Default stage config path (relative to vllm_omni package)
DEFAULT_CONFIG_PATH = "vllm_omni/model_executor/stage_configs/glm_image.yaml"
SEED = 42
# GLM-Image special tokens
GLM_IMAGE_EOS_TOKEN_ID = 16385 # eos_token_id from generation_config.json
GLM_IMAGE_VISION_VOCAB_SIZE = 16512 # top_k should be vision_vocab_size
def compute_max_tokens(height: int, width: int, factor: int = 32, is_i2i: bool = False) -> int:
"""
Compute max_new_tokens for GLM-Image AR generation.
GLM-Image generation differs by mode:
- text-to-image (t2i): small preview + large target + EOS
- image-to-image (i2i): large target + EOS
Args:
height: Target image height in pixels
width: Target image width in pixels
factor: Downsampling factor (32 for GLM-Image AR output)
is_i2i: Whether the request is image-to-image mode
Returns:
Total number of tokens to generate for the specified mode
"""
# Large image tokens (target resolution)
token_h = height // factor
token_w = width // factor
large_tokens = token_h * token_w
# Small preview tokens (half resolution in each dimension)
import math
ratio = token_h / token_w if token_w > 0 else 1.0
small_token_h = max(1, int(math.sqrt(ratio) * (factor // 2)))
small_token_w = max(1, int(math.sqrt(1 / ratio) * (factor // 2)))
small_tokens = small_token_h * small_token_w
if is_i2i:
return large_tokens + 1
return small_tokens + large_tokens + 1
def load_image(image_path: str) -> Image.Image:
"""Load an image from file path."""
if not os.path.exists(image_path):
raise FileNotFoundError(f"Image file not found: {image_path}")
return Image.open(image_path).convert("RGB")
def save_image(image: Image.Image, output_path: str) -> None:
"""Save an image to file path."""
output_dir = os.path.dirname(output_path)
if output_dir:
os.makedirs(output_dir, exist_ok=True)
image.save(output_path)
print(f"Image saved to: {output_path}")
def build_prompt_for_t2i(
prompt: str,
height: int = 1024,
width: int = 1024,
) -> dict:
"""
Build prompt dict for text-to-image generation.
Args:
prompt: Text description for image generation
height: Target image height
width: Target image width
Returns:
Dict containing prompt and generation parameters
"""
return {
"prompt": prompt,
"height": height,
"width": width,
# Pass target dimensions to AR processor for proper grid token generation
"mm_processor_kwargs": {
"target_h": height,
"target_w": width,
},
}
def build_prompt_for_i2i(
prompt: str,
image: Image.Image,
height: int | None = None,
width: int | None = None,
) -> dict:
"""
Build prompt dict for image-to-image generation.
Args:
prompt: Text description for image editing
image: Source image for editing
height: Target image height (default: use source image size)
width: Target image width (default: use source image size)
Returns:
Dict containing prompt, image, and generation parameters
"""
# Use source image dimensions if not specified
if height is None:
height = image.height
if width is None:
width = image.width
return {
"prompt": prompt,
"multi_modal_data": {
"image": image,
},
"height": height,
"width": width,
# Pass target dimensions to AR processor for proper grid token generation
"mm_processor_kwargs": {
"target_h": height,
"target_w": width,
},
}
def main(args: argparse.Namespace) -> None:
"""Main entry point for GLM-Image end-to-end inference."""
print("=" * 60)
print("GLM-Image Multistage End-to-End Inference")
print("=" * 60)
# Validate arguments
if not args.prompt:
raise ValueError("--prompt is required")
# Determine config path
config_path = args.config_path
if config_path is None:
# Try to find default config
if os.path.exists(DEFAULT_CONFIG_PATH):
config_path = DEFAULT_CONFIG_PATH
else:
# Try relative to script location
script_dir = Path(__file__).parent.parent.parent.parent
config_path = script_dir / "vllm_omni/model_executor/stage_configs/glm_image.yaml"
if not config_path.exists():
raise FileNotFoundError(
f"Stage config not found. Please specify --config-path. Tried: {DEFAULT_CONFIG_PATH}"
)
config_path = str(config_path)
print(f"Model path: {args.model_path}")
print(f"Config path: {config_path}")
print(f"Prompt: {args.prompt}")
# Load source image for image-to-image mode
source_image = None
if args.image:
print(f"Source image: {args.image}")
source_image = load_image(args.image)
print(f" Image size: {source_image.width}x{source_image.height}")
# Build prompt based on mode
if source_image is not None:
# Image-to-image mode
prompt_dict = build_prompt_for_i2i(
prompt=args.prompt,
image=source_image,
height=args.height,
width=args.width,
)
mode = "image-to-image"
else:
# Text-to-image mode
prompt_dict = build_prompt_for_t2i(
prompt=args.prompt,
height=args.height or 1024,
width=args.width or 1024,
)
mode = "text-to-image"
print(f"Mode: {mode}")
print(f"Target size: {prompt_dict.get('height', 1024)}x{prompt_dict.get('width', 1024)}")
# Add generation parameters to prompt
prompt_dict["seed"] = args.seed
prompt_dict["num_inference_steps"] = args.num_inference_steps
prompt_dict["guidance_scale"] = args.guidance_scale
if args.negative_prompt:
prompt_dict["negative_prompt"] = args.negative_prompt
# Build cache-dit config if requested
cache_config = None
if args.cache_backend == "cache_dit":
cache_config = {
"Fn_compute_blocks": 1,
"Bn_compute_blocks": 0,
"max_warmup_steps": 4,
"residual_diff_threshold": 0.24,
"max_continuous_cached_steps": 3,
"enable_taylorseer": False,
"taylorseer_order": 1,
"scm_steps_mask_policy": None,
"scm_steps_policy": "dynamic",
}
# Initialize Omni with multistage config
print("\nInitializing Omni with multistage pipeline...")
print(f" Cache backend: {args.cache_backend or 'None (no acceleration)'}")
start_time = time.time()
omni = Omni(
model=args.model_path,
stage_configs_path=config_path,
log_stats=args.enable_stats,
stage_init_timeout=args.stage_init_timeout,
cache_backend=args.cache_backend,
cache_config=cache_config,
enable_cache_dit_summary=getattr(args, "enable_cache_dit_summary", False),
enable_diffusion_pipeline_profiler=args.enable_diffusion_pipeline_profiler,
)
init_time = time.time() - start_time
print(f"Initialization completed in {init_time:.2f}s")
# Prepare prompts (support batch generation)
prompts = [prompt_dict for _ in range(args.num_prompts)]
# No explicit sampling_params for diffusion - parameters are in prompt_dict
# For multistage, the AR stage may need sampling params
from vllm import SamplingParams
# Compute max_tokens dynamically based on target image size
target_height = prompt_dict.get("height", 1024)
target_width = prompt_dict.get("width", 1024)
is_i2i = source_image is not None
calculated_max_tokens = compute_max_tokens(target_height, target_width, is_i2i=is_i2i)
# Use calculated value unless user explicitly specified a different value
# Default args.max_tokens is 16384 (very large), so prefer calculated value
effective_max_tokens = calculated_max_tokens if args.max_tokens == 16384 else args.max_tokens
if args.verbose:
print(
f"AR max_tokens: {effective_max_tokens} "
f"(calculated: {calculated_max_tokens}, arg: {args.max_tokens}, mode: {'i2i' if is_i2i else 't2i'})"
)
# IMPORTANT: GLM-Image AR model requires these exact sampling parameters
# from generation_config.json for proper image token generation.
# - temperature=0.9, top_p=0.75, top_k=16512 (vision_vocab_size)
# - stop_token_ids=[16385] (eos_token_id) is CRITICAL to stop generation
ar_sampling_params = SamplingParams(
temperature=0.9, # From generation_config.json
top_p=0.75, # From generation_config.json
top_k=GLM_IMAGE_VISION_VOCAB_SIZE, # 16512, vision vocabulary size
max_tokens=effective_max_tokens,
stop_token_ids=[GLM_IMAGE_EOS_TOKEN_ID], # 16385, CRITICAL for stopping
seed=args.seed,
detokenize=False,
# Keep target size available in runner/model for deterministic M-RoPE
# decode grids in t2i (no mm_features available in this path).
extra_args={
"target_h": int(target_height),
"target_w": int(target_width),
},
)
# For diffusion stage, sampling_params contains diffusion-specific parameters
# These are passed as kwargs to the diffusion engine
diffusion_sampling_params = OmniDiffusionSamplingParams(
num_inference_steps=args.num_inference_steps,
guidance_scale=args.guidance_scale,
height=prompt_dict.get("height", 1024),
width=prompt_dict.get("width", 1024),
seed=args.seed,
)
# For multistage, we need sampling_params for each stage
# Stage 0 (AR): SamplingParams for vLLM
# Stage 1 (Diffusion): dict with diffusion kwargs
sampling_params_list = [ar_sampling_params, diffusion_sampling_params]
# Run generation
print(f"\nGenerating {args.num_prompts} image(s)...")
gen_start_time = time.time()
output_dir = os.path.dirname(args.output) if args.output else "outputs"
if output_dir:
os.makedirs(output_dir, exist_ok=True)
output_count = 0
for stage_outputs in omni.generate(prompts, sampling_params_list, py_generator=True):
output = stage_outputs.request_output
if stage_outputs.final_output_type == "image":
request_id = output.request_id
# Get generated images
images = output.images if hasattr(output, "images") else []
if not images and hasattr(output, "multimodal_output"):
images = output.multimodal_output.get("images", [])
# Save each generated image
for idx, img in enumerate(images):
if args.num_prompts == 1 and len(images) == 1:
output_path = args.output
else:
base, ext = os.path.splitext(args.output)
output_path = f"{base}_{request_id}_{idx}{ext}"
if isinstance(img, Image.Image):
save_image(img, output_path)
else:
print(f"Warning: Unexpected image type for request {request_id}: {type(img)}")
output_count += 1
elif stage_outputs.final_output_type == "text":
# AR stage output (intermediate, for debugging)
if args.verbose:
print(f"AR output for request {output.request_id}:")
print(f" Token count: {len(output.outputs[0].token_ids)}")
gen_time = time.time() - gen_start_time
print(f"\nGeneration completed in {gen_time:.2f}s")
print(f"Generated {output_count} image(s)")
# Cleanup
omni.close()
print("\nDone!")
def parse_args() -> argparse.Namespace:
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="GLM-Image Multistage End-to-End Inference",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
# Required arguments
parser.add_argument(
"--model-path",
type=str,
default="zai-org/GLM-Image",
help="Path to GLM-Image model directory or HuggingFace model ID",
)
parser.add_argument(
"--prompt",
type=str,
required=True,
help="Text prompt for image generation",
)
# Optional arguments
parser.add_argument(
"--config-path",
type=str,
default=None,
help="Path to stage config YAML file (default: auto-detect)",
)
parser.add_argument(
"--image",
type=str,
default=None,
help="Path to source image for image-to-image mode",
)
parser.add_argument(
"--output",
type=str,
default="output_glm_image.png",
help="Output image path (default: output_glm_image.png)",
)
parser.add_argument(
"--negative-prompt",
type=str,
default=None,
help="Negative prompt for classifier-free guidance",
)
# Generation parameters
parser.add_argument(
"--height",
type=int,
default=None,
help="Output image height (default: 1024 for t2i, source size for i2i)",
)
parser.add_argument(
"--width",
type=int,
default=None,
help="Output image width (default: 1024 for t2i, source size for i2i)",
)
parser.add_argument(
"--num-inference-steps",
type=int,
default=50,
help="Number of diffusion denoising steps (default: 50)",
)
parser.add_argument(
"--guidance-scale",
type=float,
default=1.5,
help="Classifier-free guidance scale (default: 1.5)",
)
parser.add_argument(
"--seed",
type=int,
default=SEED,
help=f"Random seed for reproducibility (default: {SEED})",
)
parser.add_argument(
"--max-tokens",
type=int,
default=16384,
help="Maximum tokens for AR generation (default: 16384)",
)
# Batch processing
parser.add_argument(
"--num-prompts",
type=int,
default=1,
help="Number of images to generate (default: 1)",
)
# Cache acceleration
parser.add_argument(
"--cache-backend",
type=str,
default=None,
choices=["cache_dit"],
help="Cache backend for DiT acceleration. Default: None (no cache).",
)
parser.add_argument(
"--enable-cache-dit-summary",
action="store_true",
help="Enable cache-dit summary logging after diffusion forward passes.",
)
# Runtime options
parser.add_argument(
"--enable-stats",
action="store_true",
default=False,
help="Enable statistics logging",
)
parser.add_argument(
"--stage-init-timeout",
type=int,
default=600,
help="Timeout for stage initialization in seconds (default: 600)",
)
parser.add_argument(
"--verbose",
"-v",
action="store_true",
default=False,
help="Enable verbose output",
)
parser.add_argument(
"--enable-diffusion-pipeline-profiler",
action="store_true",
help="Enable diffusion pipeline profiler to display stage durations.",
)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
main(args)
examples/offline_inference/glm_image/run_i2i.sh
-93
View File
@@ -1,93 +0,0 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# Run GLM-Image image-to-image (editing) with multistage pipeline
set -e
# Default values
MODEL_PATH="${MODEL_PATH:-/path/to/glm-image}"
CONFIG_PATH="${CONFIG_PATH:-vllm_omni/model_executor/stage_configs/glm_image.yaml}"
PROMPT="${PROMPT:-Transform this image into an oil painting style}"
INPUT_IMAGE=""
OUTPUT="${OUTPUT:-output_i2i.png}"
NUM_STEPS="${NUM_STEPS:-50}"
GUIDANCE="${GUIDANCE:-1.5}"
SEED="${SEED:-42}"
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
--model-path)
MODEL_PATH="$2"
shift 2
;;
--config-path)
CONFIG_PATH="$2"
shift 2
;;
--prompt)
PROMPT="$2"
shift 2
;;
--image)
INPUT_IMAGE="$2"
shift 2
;;
--output)
OUTPUT="$2"
shift 2
;;
--num-steps)
NUM_STEPS="$2"
shift 2
;;
--guidance)
GUIDANCE="$2"
shift 2
;;
--seed)
SEED="$2"
shift 2
;;
*)
echo "Unknown option: $1"
exit 1
;;
esac
done
# Check if input image is provided
if [ -z "${INPUT_IMAGE}" ]; then
echo "Error: --image is required for image-to-image mode"
echo "Usage: ./run_i2i.sh --image /path/to/input.png [--prompt \"edit instruction\"]"
exit 1
fi
if [ ! -f "${INPUT_IMAGE}" ]; then
echo "Error: Input image not found: ${INPUT_IMAGE}"
exit 1
fi
echo "=============================================="
echo "GLM-Image Image-to-Image Generation"
echo "=============================================="
echo "Model: ${MODEL_PATH}"
echo "Config: ${CONFIG_PATH}"
echo "Input: ${INPUT_IMAGE}"
echo "Prompt: ${PROMPT}"
echo "Output: ${OUTPUT}"
echo "Steps: ${NUM_STEPS}"
echo "Guidance: ${GUIDANCE}"
echo "Seed: ${SEED}"
echo "=============================================="
python end2end.py \
--model-path "${MODEL_PATH}" \
--config-path "${CONFIG_PATH}" \
--prompt "${PROMPT}" \
--image "${INPUT_IMAGE}" \
--output "${OUTPUT}" \
--num-inference-steps "${NUM_STEPS}" \
--guidance-scale "${GUIDANCE}" \
--seed "${SEED}" \
--verbose
examples/offline_inference/glm_image/run_t2i.sh
-87
View File
@@ -1,87 +0,0 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# Run GLM-Image text-to-image generation with multistage pipeline
set -e
# Default values
MODEL_PATH="${MODEL_PATH:-/path/to/glm-image}"
CONFIG_PATH="${CONFIG_PATH:-vllm_omni/model_executor/stage_configs/glm_image.yaml}"
PROMPT="${PROMPT:-A beautiful mountain landscape with snow-capped peaks and a clear blue lake}"
OUTPUT="${OUTPUT:-output_t2i.png}"
HEIGHT="${HEIGHT:-1024}"
WIDTH="${WIDTH:-1024}"
NUM_STEPS="${NUM_STEPS:-50}"
GUIDANCE="${GUIDANCE:-1.5}"
SEED="${SEED:-42}"
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
--model-path)
MODEL_PATH="$2"
shift 2
;;
--config-path)
CONFIG_PATH="$2"
shift 2
;;
--prompt)
PROMPT="$2"
shift 2
;;
--output)
OUTPUT="$2"
shift 2
;;
--height)
HEIGHT="$2"
shift 2
;;
--width)
WIDTH="$2"
shift 2
;;
--num-steps)
NUM_STEPS="$2"
shift 2
;;
--guidance)
GUIDANCE="$2"
shift 2
;;
--seed)
SEED="$2"
shift 2
;;
*)
echo "Unknown option: $1"
exit 1
;;
esac
done
echo "=============================================="
echo "GLM-Image Text-to-Image Generation"
echo "=============================================="
echo "Model: ${MODEL_PATH}"
echo "Config: ${CONFIG_PATH}"
echo "Prompt: ${PROMPT}"
echo "Output: ${OUTPUT}"
echo "Size: ${WIDTH}x${HEIGHT}"
echo "Steps: ${NUM_STEPS}"
echo "Guidance: ${GUIDANCE}"
echo "Seed: ${SEED}"
echo "=============================================="
python end2end.py \
--model-path "${MODEL_PATH}" \
--config-path "${CONFIG_PATH}" \
--prompt "${PROMPT}" \
--output "${OUTPUT}" \
--height "${HEIGHT}" \
--width "${WIDTH}" \
--num-inference-steps "${NUM_STEPS}" \
--guidance-scale "${GUIDANCE}" \
--seed "${SEED}" \
--verbose
examples/online_serving/glm_image/README.md
-204
View File
@@ -1,204 +0,0 @@
# GLM-Image Online Serving
This example demonstrates how to deploy GLM-Image for online image generation using vLLM-Omni.
## 🛠️ Installation
Please refer to [README.md](../../../README.md)
## Run examples (GLM-Image)
**Note**: These examples work with the default configuration on **2× NVIDIA A100 (80GB)** or equivalent. Stage 0 (AR) and Stage 1 (Diffusion) each use one GPU by default. For single-GPU setups, modify the stage configuration to share the same device.
### Launch the Server
```bash
# Use default configuration
vllm serve zai-org/GLM-Image --omni --port 8091
```
Or use the convenience script:
```bash
cd examples/online_serving/glm_image
bash run_server.sh
```
If you have a custom stage configs file:
```bash
vllm serve zai-org/GLM-Image --omni --port 8091 --stage-configs-path /path/to/glm_image.yaml
```
### Send Requests
Get into the glm_image folder:
```bash
cd examples/online_serving/glm_image
```
Send request via Python:
```bash
python openai_chat_client.py --prompt "A cute cat sitting on a window sill"
```
The Python client supports the following command-line arguments:
- `--prompt` (or `-p`): Text prompt for generation (default: `A beautiful sunset over the ocean with sailing boats`)
- `--output` (or `-o`): Output file path (default: `glm_image_output.png`)
- `--server` (or `-s`): Server URL (default: `http://localhost:8091`)
- `--image` (or `-i`): Input image path (for image-to-image editing)
- `--height`: Image height in pixels (default: 1024)
- `--width`: Image width in pixels (default: 1024)
- `--steps`: Number of inference steps (default: 50)
- `--guidance-scale`: Classifier-free guidance scale (default: 1.5)
- `--seed`: Random seed (default: 42)
- `--negative`: Negative prompt
## Modality Control
GLM-Image supports **text-to-image** and **image-to-image** modes.
The default yaml configuration deploys AR on GPU 0 and DiT on GPU 1. You can use the default configuration file: [`glm_image.yaml`](../../../vllm_omni/model_executor/stage_configs/glm_image.yaml)
| Mode | Input | Output | Description |
| -------------- | ------------ | ------ | ---------------------------------- |
| Text-to-Image | Text | Image | Generate images from text prompts |
| Image-to-Image | Image + Text | Image | Edit images with text instructions |
### Text-to-Image
```bash
python openai_chat_client.py \
--prompt "A photorealistic mountain landscape at sunset" \
--height 1024 \
--width 1024 \
--output landscape.png
# Or use the curl script:
bash run_curl_text_to_image.sh "A futuristic city skyline at night"
```
### Image-to-Image (Image Editing)
```bash
python openai_chat_client.py \
--prompt "Convert this image to watercolor style" \
--image input.png \
--output watercolor.png
# Or use the curl script:
bash run_curl_image_edit.sh input.png "Convert to watercolor style"
```
For general-purpose request methods (curl, OpenAI SDK, Python `requests`), see
the [Text-to-Image](../text_to_image/README.md) and
[Image-to-Image](../image_to_image/README.md) guides.
## Generation Parameters
When using `/v1/chat/completions`, pass these inside `extra_body` in the curl
JSON, or via the `extra_body` keyword argument in the OpenAI Python SDK.
When using the dedicated `/v1/images/generations` or `/v1/images/edits`
endpoints, pass the supported generation controls as top-level fields directly.
For image dimensions and count, use `size` and `n` rather than `height` or
`width`.
| Parameter | Type | Default | Description |
| --------------------- | ----- | ------- | ----------------------------------- |
| `height` | int | 1024 | Image height in pixels |
| `width` | int | 1024 | Image width in pixels |
| `num_inference_steps` | int | 50 | Number of diffusion denoising steps |
| `guidance_scale` | float | 1.5 | Classifier-free guidance scale |
| `seed` | int | None | Optional random seed; `/v1/images/*` generates one server-side if omitted |
| `negative_prompt` | str | None | Negative prompt |
## Response Format
```json
{
"id": "chatcmpl-xxx",
"created": 1234567890,
"model": "zai-org/GLM-Image",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": [
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,..."
}
}
]
},
"finish_reason": "stop"
}
],
"usage": {}
}
```
## Extract Image
```bash
# From a saved JSON response
cat response.json | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png
```
## Architecture
GLM-Image uses a 2-stage multistage pipeline:
```
Stage 0 (AR Model) Stage 1 (Diffusion)
┌───────────────────┐ ┌─────────────────────┐
│ vLLM-optimized │ prior │ GlmImagePipeline │
│ GlmImageFor │──tokens──►│ ┌───────────────┐ │
│ Conditional │ │ │ DiT Denoiser │ │
│ Generation │ │ └───────┬───────┘ │
│ (9B AR model) │ │ ▼ │
└───────────────────┘ │ ┌───────────────┐ │
▲ │ │ VAE Decode │──┼──► Image
│ │ └───────────────┘ │
Text / Image └─────────────────────┘
Input
```
## VRAM Requirements
| Stage | VRAM |
| :---------------- | :--------------------- |
| Stage-0 (AR) | **~18 GiB + KV Cache** |
| Stage-1 (DiT+VAE) | **~20 GiB** |
| Total | **~38 GiB + KV Cache** |
## File Description
| File | Description |
| --------------------------- | ------------------------------------- |
| `run_server.sh` | Server startup script |
| `run_curl_text_to_image.sh` | Text-to-image curl example |
| `run_curl_image_edit.sh` | Image-to-image (editing) curl example |
| `openai_chat_client.py` | Python client (t2i + i2i) |
## FAQ
- If you encounter OOM errors, adjust `gpu_memory_utilization` in the stage config:
```yaml
# In glm_image.yaml, reduce from default 0.6:
gpu_memory_utilization: 0.5
```
- The first request may be slow due to model warmup. Subsequent requests will be faster.
- If you encounter `Transformers does not recognize this architecture` error, your have to upgrade `transformers` package to `5.3.0` or above:
```
pip install --upgrade transformers
```
examples/online_serving/glm_image/openai_chat_client.py
-172
View File
@@ -1,172 +0,0 @@
#!/usr/bin/env python3
"""
GLM-Image OpenAI-compatible chat client for text-to-image and image-to-image.
Usage:
# Text-to-image
python openai_chat_client.py --prompt "A cute cat" --output output.png
# Image-to-image (image editing)
python openai_chat_client.py --prompt "Convert to watercolor style" --image input.png --output output.png
"""
import argparse
import base64
from pathlib import Path
import requests
def generate_image(
prompt: str,
server_url: str = "http://localhost:8091",
image_path: str | None = None,
height: int = 1024,
width: int = 1024,
steps: int = 50,
guidance_scale: float = 1.5,
seed: int | None = None,
negative_prompt: str | None = None,
) -> bytes | None:
"""Generate or edit an image using the chat completions API.
Args:
prompt: Text description or editing instruction
server_url: Server URL
image_path: Path to input image (for image-to-image editing)
height: Image height in pixels
width: Image width in pixels
steps: Number of inference steps
guidance_scale: Classifier-free guidance scale
seed: Random seed for reproducibility
negative_prompt: Negative prompt
Returns:
Image bytes or None if failed
"""
# Build message content
content: list[dict] = [{"type": "text", "text": prompt}]
if image_path:
img_path = Path(image_path)
if not img_path.exists():
print(f"Error: Image file not found: {image_path}")
return None
b64_data = base64.b64encode(img_path.read_bytes()).decode("utf-8")
suffix = img_path.suffix.lstrip(".").lower()
mime = {"jpg": "jpeg", "jpeg": "jpeg", "png": "png", "webp": "webp"}.get(suffix, "png")
content.append(
{
"type": "image_url",
"image_url": {"url": f"data:image/{mime};base64,{b64_data}"},
}
)
messages = [{"role": "user", "content": content}]
# Build request payload
extra_body: dict = {
"height": height,
"width": width,
"num_inference_steps": steps,
"guidance_scale": guidance_scale,
}
if seed is not None:
extra_body["seed"] = seed
if negative_prompt:
extra_body["negative_prompt"] = negative_prompt
payload = {"messages": messages, "extra_body": extra_body}
# Send request
try:
mode = "image-to-image" if image_path else "text-to-image"
print(f"Sending {mode} request to {server_url}...")
response = requests.post(
f"{server_url}/v1/chat/completions",
headers={"Content-Type": "application/json"},
json=payload,
timeout=600,
)
response.raise_for_status()
data = response.json()
# Extract image from response
choices = data.get("choices", [])
for choice in choices:
choice_content = choice.get("message", {}).get("content")
if isinstance(choice_content, list):
for item in choice_content:
if isinstance(item, dict) and "image_url" in item:
img_url = item["image_url"].get("url", "")
if img_url.startswith("data:image"):
_, b64 = img_url.split(",", 1)
return base64.b64decode(b64)
print(f"Unexpected response format: {data}")
return None
except Exception as e:
print(f"Error: {e}")
return None
def main():
parser = argparse.ArgumentParser(description="GLM-Image chat client")
parser.add_argument(
"--prompt",
"-p",
default="A beautiful sunset over the ocean with sailing boats",
help="Text prompt",
)
parser.add_argument("--output", "-o", default="glm_image_output.png", help="Output file")
parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL")
# Image-to-image
parser.add_argument(
"--image",
"-i",
type=str,
help="Input image path (for image-to-image editing)",
)
# Generation parameters
parser.add_argument("--height", type=int, default=1024, help="Image height")
parser.add_argument("--width", type=int, default=1024, help="Image width")
parser.add_argument("--steps", type=int, default=50, help="Inference steps")
parser.add_argument("--guidance-scale", type=float, default=1.5, help="CFG guidance scale")
parser.add_argument("--seed", type=int, default=42, help="Random seed")
parser.add_argument("--negative", help="Negative prompt")
args = parser.parse_args()
mode = "image-to-image" if args.image else "text-to-image"
print(f"Mode: {mode}")
print(f"Prompt: {args.prompt}")
if args.image:
print(f"Input image: {args.image}")
image_bytes = generate_image(
prompt=args.prompt,
server_url=args.server,
image_path=args.image,
height=args.height,
width=args.width,
steps=args.steps,
guidance_scale=args.guidance_scale,
seed=args.seed,
negative_prompt=args.negative,
)
if image_bytes:
output_path = Path(args.output)
output_path.write_bytes(image_bytes)
print(f"Image saved to: {output_path}")
print(f"Size: {len(image_bytes) / 1024:.1f} KB")
else:
print("Failed to generate image")
exit(1)
if __name__ == "__main__":
main()
examples/online_serving/glm_image/run_curl_image_edit.sh
-61
View File
@@ -1,61 +0,0 @@
#!/bin/bash
# GLM-Image image-edit (image-to-image) curl example
set -euo pipefail
if [[ $# -lt 2 ]]; then
echo "Usage: $0 <input_image> \"<edit_prompt>\" [output_file]" >&2
exit 1
fi
INPUT_IMG=$1
PROMPT=$2
SERVER="${SERVER:-http://localhost:8091}"
CURRENT_TIME=$(date +%Y%m%d%H%M%S)
OUTPUT="${3:-glm_image_i2i_${CURRENT_TIME}.png}"
if [[ ! -f "$INPUT_IMG" ]]; then
echo "Input image not found: $INPUT_IMG" >&2
exit 1
fi
# base64 encode (macOS uses -i, Linux uses -w0)
if [[ "$(uname)" == "Darwin" ]]; then
IMG_B64=$(base64 < "$INPUT_IMG" | tr -d '\n')
else
IMG_B64=$(base64 -w0 "$INPUT_IMG")
fi
REQUEST_JSON=$(
jq -n --arg prompt "$PROMPT" --arg img "$IMG_B64" '{
messages: [{
role: "user",
content: [
{"type": "text", "text": $prompt},
{"type": "image_url", "image_url": {"url": ("data:image/png;base64," + $img)}}
]
}],
extra_body: {
height: 1024,
width: 1024,
num_inference_steps: 50,
guidance_scale: 1.5,
seed: 42
}
}'
)
echo "Generating edited image..."
echo "Server: $SERVER"
echo "Prompt: $PROMPT"
echo "Input : $INPUT_IMG"
echo "Output: $OUTPUT"
curl -s "$SERVER/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "$REQUEST_JSON" \
| jq -r '.choices[0].message.content[0].image_url.url' \
| cut -d',' -f2- \
| base64 -d > "$OUTPUT"
echo "Image saved to: $OUTPUT"
examples/online_serving/glm_image/run_curl_text_to_image.sh
-30
View File
@@ -1,30 +0,0 @@
#!/bin/bash
# GLM-Image text-to-image curl example
set -euo pipefail
PROMPT="${1:-A beautiful sunset over the ocean with sailing boats}"
SERVER="${SERVER:-http://localhost:8091}"
OUTPUT="${OUTPUT:-glm_image_t2i_output.png}"
echo "Generating image..."
echo "Server: $SERVER"
echo "Prompt: $PROMPT"
echo "Output: $OUTPUT"
curl -s "$SERVER/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{
\"messages\": [
{\"role\": \"user\", \"content\": \"$PROMPT\"}
],
\"extra_body\": {
\"height\": 1024,
\"width\": 1024,
\"num_inference_steps\": 50,
\"guidance_scale\": 1.5,
\"seed\": 42
}
}" | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > "$OUTPUT"
echo "Image saved to: $OUTPUT"
examples/online_serving/glm_image/run_server.sh
-12
View File
@@ -1,12 +0,0 @@
#!/bin/bash
# GLM-Image online serving startup script
MODEL="${MODEL:-zai-org/GLM-Image}"
PORT="${PORT:-8091}"
echo "Starting GLM-Image server..."
echo "Model: $MODEL"
echo "Port: $PORT"
vllm serve "$MODEL" --omni \
--port "$PORT"
vllm_omni/config/pipeline_registry.py
+4
View File
@@ -43,6 +43,10 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = {
"vllm_omni.model_executor.models.qwen3_tts.pipeline",
"QWEN3_TTS_PIPELINE",
),
"glm_image": (
"vllm_omni.model_executor.models.glm_image.pipeline",
"GLM_IMAGE_PIPELINE",
),
"voxcpm2": (
"vllm_omni.model_executor.models.voxcpm2.pipeline",
"VOXCPM2_PIPELINE",
vllm_omni/config/stage_config.py
+39
View File
@@ -169,6 +169,12 @@ class StagePipelineConfig:
prompt_expand_func: str | None = None
cfg_kv_collect_func: str | None = None
omni_kv_config: dict[str, Any] | None = None
# Model subdirectory indirections: for multi-component HF repos where the
# stage's config/tokenizer lives in a subdirectory (e.g. GLM-Image's AR
# config is in ``vision_language_encoder/``). Consumed at stage-init time
# by ``stage_init_utils._resolve_model_tokenizer_paths``.
model_subdir: str | None = None
tokenizer_subdir: str | None = None
extras: dict[str, Any] = field(default_factory=dict)
@@ -185,6 +191,11 @@ class PipelineConfig:
# matches ``hf_config.architectures[*]`` against this tuple to route
# to the correct pipeline. Leave empty for models with unique model_type.
hf_architectures: tuple[str, ...] = ()
# Diffusers pipeline class name: for models that ship a ``model_index.json``
# (no root ``config.json``), the ``_class_name`` field is matched against
# this value to auto-detect the pipeline. Only needed for diffusers-style
# multi-component repos (e.g. GLM-Image). ``None`` = not a diffusers model.
diffusers_class_name: str | None = None
def get_stage(self, stage_id: int) -> StagePipelineConfig | None:
"""Look up a stage by its ID."""
@@ -702,6 +713,13 @@ def _build_engine_args(
engine_args["engine_output_type"] = ps.engine_output_type
if next_stage_proc:
engine_args["custom_process_next_stage_input_func"] = next_stage_proc
# Subdirectory indirections from StagePipelineConfig (structural, not
# deployment knobs). Deploy YAML ``engine_extras`` can still override
# these per-stage if needed.
if ps.model_subdir:
engine_args["model_subdir"] = ps.model_subdir
if ps.tokenizer_subdir:
engine_args["tokenizer_subdir"] = ps.tokenizer_subdir
# Pipeline-wide top-level DeployConfig settings, applied to every stage.
for name in _PIPELINE_WIDE_ENGINE_FIELDS:
@@ -1378,6 +1396,27 @@ class StageConfigFactory:
except Exception as e:
logger.debug(f"Failed to auto-detect model type for {model}: {e}")
# Fallback for diffusers-style models: check model_index.json.
# Some models (e.g. GLM-Image) have no root config.json but ship a
# model_index.json with _class_name that maps to a pipeline key via
# PipelineConfig.diffusers_class_name.
try:
from vllm.transformers_utils.config import get_hf_file_to_dict
model_index = get_hf_file_to_dict("model_index.json", model, revision=None)
if model_index and "_class_name" in model_index:
class_name = model_index["_class_name"]
for pipeline_cfg in _PIPELINE_REGISTRY.values():
if pipeline_cfg.diffusers_class_name == class_name:
logger.info(
"Detected pipeline %r from model_index.json (_class_name=%r)",
pipeline_cfg.model_type,
class_name,
)
return pipeline_cfg.model_type, None
except Exception:
pass
# Final fallback: some models (e.g. CosyVoice3) ship an empty
# config.json and rely on naming conventions. Match the model path
# basename against registered pipeline keys — longest match wins
vllm_omni/deploy/glm_image.yaml
+38
View File
@@ -0,0 +1,38 @@
# GLM-Image deploy: AR (stage 0) + Diffusion (stage 1).
# Topology declared in vllm_omni/model_executor/models/glm_image/pipeline.py.
#
# Fields omitted from a stage fall back to StageDeployConfig dataclass
# defaults (see vllm_omni/config/stage_config.py). Pipeline-wide settings
# (trust_remote_code, distributed_executor_backend, etc.) use DeployConfig
# defaults unless overridden here.
async_chunk: false
stages:
# Stage 0: AR Model (GlmImageForConditionalGeneration)
# Generates prior_token_ids for conditioning the diffusion process.
- stage_id: 0
max_num_seqs: 1
gpu_memory_utilization: 0.6
enforce_eager: false
max_num_batched_tokens: 32768
devices: "0"
default_sampling_params:
temperature: 0.9
top_p: 0.75
top_k: 16512
stop_token_ids: [16385]
seed: 42
detokenize: false
# Stage 1: Diffusion (DiT + VAE)
# Receives prior_token_ids from AR, performs denoising + VAE decode.
- stage_id: 1
max_num_seqs: 1
enforce_eager: true
devices: "1"
default_sampling_params:
seed: 42
num_inference_steps: 50
guidance_scale: 1.5
height: 1024
width: 1024
vllm_omni/model_executor/models/glm_image/__init__.py
+14 -1
View File
@@ -1,3 +1,16 @@
from .glm_image_ar import GlmImageForConditionalGeneration
def __getattr__(name: str):
"""Lazy import to avoid importing transformers.models.glm_image at module init.
The AR model depends on ``transformers.models.glm_image`` which is only
available when the model weights are present. Importing it eagerly in
``__init__.py`` breaks the pipeline registry lookup for environments that
don't have this custom transformers extension installed.
"""
if name == "GlmImageForConditionalGeneration":
from .glm_image_ar import GlmImageForConditionalGeneration
return GlmImageForConditionalGeneration
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
__all__ = ["GlmImageForConditionalGeneration"]
vllm_omni/model_executor/models/glm_image/pipeline.py
+47
View File
@@ -0,0 +1,47 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""GLM-Image pipeline topologies (frozen).
Two-stage (default):
Stage 0: AR — multimodal understanding + token_ids generation
Stage 1: DiT — diffusion image generation
"""
from vllm_omni.config.stage_config import (
PipelineConfig,
StageExecutionType,
StagePipelineConfig,
)
GLM_IMAGE_PIPELINE = PipelineConfig(
model_type="glm_image",
model_arch="GlmImageForConditionalGeneration",
hf_architectures=("GlmImageForConditionalGeneration",),
diffusers_class_name="GlmImagePipeline",
stages=(
StagePipelineConfig(
stage_id=0,
model_stage="ar",
execution_type=StageExecutionType.LLM_AR,
requires_multimodal_data=True,
input_sources=(),
final_output=False,
owns_tokenizer=True,
model_arch="GlmImageForConditionalGeneration",
engine_output_type="token_ids",
model_subdir="vision_language_encoder",
tokenizer_subdir="processor",
),
StagePipelineConfig(
stage_id=1,
model_stage="dit",
execution_type=StageExecutionType.DIFFUSION,
input_sources=(0,),
requires_multimodal_data=True,
final_output=True,
final_output_type="image",
model_arch="GlmImagePipeline",
custom_process_input_func="vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion",
omni_kv_config={"need_recv_cache": False},
),
),
)
vllm_omni/model_executor/stage_configs/glm_image.yaml
-74
View File
@@ -1,74 +0,0 @@
# Stage config for running GLM-Image with 2-stage architecture
# Stage 0: AR Model (vLLM implementation) - generates prior_token_ids
# Stage 1: Diffusion (DiT + VAE) - denoising and image decoding
stage_args:
# Stage 0: AR Model (GlmImageForConditionalGeneration)
# This stage uses the vLLM-optimized AR model to generate prior tokens
# for conditioning the diffusion process.
- stage_id: 0
stage_type: llm
runtime:
process: true
devices: "0"
requires_multimodal_data: true # Required for i2i mode to receive source images
engine_args:
model_stage: ar
max_num_seqs: 1
model_arch: GlmImageForConditionalGeneration
model_subdir: vision_language_encoder # AR model config.json is in this subdirectory
tokenizer_subdir: processor # Use processor's tokenizer (not ByT5 from tokenizer/)
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
gpu_memory_utilization: 0.6
enforce_eager: false
trust_remote_code: true
engine_output_type: token_ids # Output prior_token_ids for diffusion stage
distributed_executor_backend: "mp"
enable_prefix_caching: false
max_num_batched_tokens: 32768
final_output: false # AR is not the final output
is_comprehension: true
default_sampling_params:
temperature: 0.9 # From model's generation_config.json
top_p: 0.75 # From model's generation_config.json
top_k: 16512 # vision_vocab_size from generation_config.json
stop_token_ids: [16385] # eos_token_id from generation_config.json
seed: 42
detokenize: false
# Stage 1: Diffusion (DiT + VAE)
# This stage receives prior_token_ids from AR and performs denoising + VAE decode
- stage_id: 1
stage_type: diffusion
runtime:
process: true
devices: "1" # Can use different GPU, or same GPU if memory allows
requires_multimodal_data: true # Required for i2i mode to pass condition images
engine_args:
model_stage: dit
max_num_seqs: 1
model_arch: GlmImagePipeline # Required for diffusion model class resolution
# Diffusion-specific parameters
num_gpus: 1
enforce_eager: true
trust_remote_code: true
distributed_executor_backend: "mp"
engine_input_source: [0] # Input from AR stage
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion
final_output: true
final_output_type: image
default_sampling_params:
# Diffusion-specific parameters only (no LLM params like temperature/top_p/top_k)
seed: 42
num_inference_steps: 50
guidance_scale: 1.5
height: 1024
width: 1024
# Top-level runtime config
runtime:
enabled: true
edges:
- from: 0 # AR → Diffusion: trigger after AR completes
to: 1
vllm_omni/model_executor/stage_configs/glm_image_muilticonnector.yaml
-87
View File
@@ -1,87 +0,0 @@
# Stage config for running GLM-Image with 2-stage architecture (MultiConnector version)
# Stage 0: AR Model (vLLM implementation) - generates prior_token_ids
# Stage 1: Diffusion (DiT + VAE) - denoising and image decoding
#
# This config uses OmniConnectors for inter-stage communication,
# enabling efficient tensor transfer between stages on different processes/nodes.
stage_args:
# Stage 0: AR Model (GlmImageForConditionalGeneration)
# This stage uses the vLLM-optimized AR model to generate prior tokens
# for conditioning the diffusion process.
- stage_id: 0
stage_type: llm
runtime:
process: true
devices: "0"
requires_multimodal_data: true # Required for i2i mode to receive source images
engine_args:
model_stage: ar
max_num_seqs: 1
model_arch: GlmImageForConditionalGeneration
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
gpu_memory_utilization: 0.6
enforce_eager: false
trust_remote_code: true
engine_output_type: token_ids # Output prior_token_ids for diffusion stage
distributed_executor_backend: "mp"
enable_prefix_caching: false
max_num_batched_tokens: 32768
hf_config_name: vision_language_encoder # Subfolder in model path
final_output: false # AR is not the final output
is_comprehension: true
default_sampling_params:
temperature: 0.9 # From model's generation_config.json
top_p: 0.75 # From model's generation_config.json
top_k: 16512 # vision_vocab_size from generation_config.json
stop_token_ids: [16385] # eos_token_id from generation_config.json
seed: 42
detokenize: false
# Stage 1: Diffusion (DiT + VAE)
# This stage receives prior_token_ids from AR and performs denoising + VAE decode
- stage_id: 1
stage_type: diffusion
runtime:
process: true
devices: "1" # Use separate GPU for diffusion
requires_multimodal_data: true # Required for i2i mode to pass condition images
engine_args:
model_stage: dit
max_num_seqs: 1
# Diffusion-specific parameters
num_gpus: 1
enforce_eager: true
trust_remote_code: true
distributed_executor_backend: "mp"
engine_input_source: [0] # Input from AR stage
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion
final_output: true
final_output_type: image
default_sampling_params:
seed: 42
num_inference_steps: 50
guidance_scale: 1.5
height: 1024
width: 1024
# Top-level runtime config with MultiConnector support
runtime:
enabled: true
edges:
- from: 0 # AR → Diffusion
to: 1
# OmniConnector configuration for efficient inter-stage tensor transfer
connectors:
- type: tensor_transfer
source_stage: 0
target_stage: 1
# Transfer prior_token_ids efficiently between stages
fields:
- name: prior_token_ids
dtype: int64
- name: prior_token_image_ids
dtype: int64
optional: true