diff --git a/examples/offline_inference/glm_image/README.md b/examples/offline_inference/glm_image/README.md deleted file mode 100644 index c3c7c2916..000000000 --- a/examples/offline_inference/glm_image/README.md +++ /dev/null @@ -1,145 +0,0 @@ -# GLM-Image Multistage End-to-End Inference - -This example demonstrates how to run GLM-Image with the vLLM-Omni multistage architecture. - -## Architecture - -GLM-Image uses a 2-stage pipeline: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ GLM-Image Pipeline │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ Stage 0 (AR Model) Stage 1 (Diffusion) │ -│ ┌─────────────────┐ ┌─────────────────────┐ │ -│ │ vLLM-optimized │ │ GlmImagePipeline │ │ -│ │ GlmImageFor │ prior │ ┌───────────────┐ │ │ -│ │ Conditional │──tokens───►│ │ DiT Denoiser │ │ │ -│ │ Generation │ │ └───────────────┘ │ │ -│ │ (9B AR model) │ │ │ │ │ -│ └─────────────────┘ │ ▼ │ │ -│ ▲ │ ┌───────────────┐ │ │ -│ │ │ │ VAE Decode │──┼──► Image -│ Text/Image │ └───────────────┘ │ │ -│ Input └─────────────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Features - -- **vLLM-optimized AR**: Uses PagedAttention and tensor parallelism for faster prior token generation -- **Flexible deployment**: AR and Diffusion stages can run on different GPUs -- **Text-to-Image**: Generate images from text descriptions -- **Image-to-Image**: Edit existing images with text prompts - -## Usage - -### Text-to-Image - -```bash -python end2end.py \ - --config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ - --prompt "A beautiful sunset over the ocean with sailing boats" \ - --height 1024 \ - --width 1024 \ - --output output_t2i.png -``` - -### Image-to-Image (Image Editing) - -```bash -python end2end.py \ - --config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ - --prompt "Transform this scene into a winter wonderland" \ - --image input.png \ - --output output_i2i.png -``` - -### With Custom Parameters - -```bash -python end2end.py \ - --model-path /path/to/glm-image \ - --config-path ../../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ - --prompt "A photorealistic cat sitting on a window sill" \ - --height 1024 \ - --width 1024 \ - --num-inference-steps 50 \ - --guidance-scale 1.5 \ - --seed 42 \ - --output output.png -``` - -## Shell Scripts - -### Run Text-to-Image - -```bash -./run_t2i.sh -``` - -### Run Image-to-Image - -```bash -./run_i2i.sh --image /path/to/input.png -``` - -## Stage Configuration - -The stage config (`glm_image.yaml`) defines: - -- **Stage 0 (AR)**: Uses `GPUARWorker` with vLLM engine - - - Model: `GlmImageForConditionalGeneration` - - Output: `token_ids` (prior tokens) - -- **Stage 1 (Diffusion)**: Uses diffusion engine - - Model: `GlmImagePipeline` - - Output: Generated image - -See `vllm_omni/model_executor/stage_configs/glm_image.yaml` for full configuration. - -## Comparison with Single-Stage - -| Aspect | Single-Stage (transformers) | Multistage (vLLM) | -| ----------- | --------------------------- | ------------------- | -| AR Model | transformers native | vLLM PagedAttention | -| Memory | Higher (no KV cache opt) | Lower (optimized) | -| Throughput | Lower | Higher | -| Flexibility | Single GPU | Multi-GPU support | - -## Troubleshooting - -### OOM Error - -Try reducing memory usage: - -```bash -# In glm_image.yaml, adjust: -gpu_memory_utilization: 0.5 # Reduce from 0.6 -``` - -### Slow Initialization - -The first run loads model weights. Subsequent runs are faster: - -```bash ---stage-init-timeout 900 # Increase timeout for slow storage -``` - -### `Transformers does not recognize this architecture` Error - -Your have to upgrade `transformers` package to `5.3.0` or above: - -``` -pip install --upgrade transformers -``` - -## Requirements - -- vLLM-Omni with GLM-Image support -- CUDA-capable GPU (recommended: H100/A100 with 80GB) -- GLM-Image model weights -- `transformers` v5.3.0 or above diff --git a/examples/offline_inference/glm_image/end2end.py b/examples/offline_inference/glm_image/end2end.py deleted file mode 100644 index 7ae9478ba..000000000 --- a/examples/offline_inference/glm_image/end2end.py +++ /dev/null @@ -1,525 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -""" -End-to-end offline inference example for GLM-Image with multistage architecture. - -This script tests the multistage pipeline where: -- Stage 0 (AR): vLLM-optimized GlmImageForConditionalGeneration generates prior_token_ids -- Stage 1 (Diffusion): GlmImagePipeline performs DiT denoising + VAE decode - -Usage (text-to-image): - python end2end.py \ - --model-path /path/to/glm-image \ - --config-path /path/to/glm_image.yaml \ - --prompt "A beautiful sunset over the ocean" \ - --output output_t2i.png - -Usage (image-to-image / image edit): - python end2end.py \ - --model-path /path/to/glm-image \ - --config-path /path/to/glm_image.yaml \ - --prompt "Make it look like winter" \ - --image input.png \ - --output output_i2i.png - -Usage (with custom parameters): - python end2end.py \ - --model-path /path/to/glm-image \ - --config-path /path/to/glm_image.yaml \ - --prompt "A cat sitting on a window sill" \ - --height 1024 \ - --width 1024 \ - --num-inference-steps 50 \ - --guidance-scale 1.5 \ - --seed 42 - -For more options, run: - python end2end.py --help -""" - -import argparse -import os -import time -from pathlib import Path - -from PIL import Image - -from vllm_omni.entrypoints.omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -# Default stage config path (relative to vllm_omni package) -DEFAULT_CONFIG_PATH = "vllm_omni/model_executor/stage_configs/glm_image.yaml" - -SEED = 42 - -# GLM-Image special tokens -GLM_IMAGE_EOS_TOKEN_ID = 16385 # eos_token_id from generation_config.json -GLM_IMAGE_VISION_VOCAB_SIZE = 16512 # top_k should be vision_vocab_size - - -def compute_max_tokens(height: int, width: int, factor: int = 32, is_i2i: bool = False) -> int: - """ - Compute max_new_tokens for GLM-Image AR generation. - - GLM-Image generation differs by mode: - - text-to-image (t2i): small preview + large target + EOS - - image-to-image (i2i): large target + EOS - - Args: - height: Target image height in pixels - width: Target image width in pixels - factor: Downsampling factor (32 for GLM-Image AR output) - is_i2i: Whether the request is image-to-image mode - - Returns: - Total number of tokens to generate for the specified mode - """ - # Large image tokens (target resolution) - token_h = height // factor - token_w = width // factor - large_tokens = token_h * token_w - - # Small preview tokens (half resolution in each dimension) - import math - - ratio = token_h / token_w if token_w > 0 else 1.0 - small_token_h = max(1, int(math.sqrt(ratio) * (factor // 2))) - small_token_w = max(1, int(math.sqrt(1 / ratio) * (factor // 2))) - small_tokens = small_token_h * small_token_w - - if is_i2i: - return large_tokens + 1 - return small_tokens + large_tokens + 1 - - -def load_image(image_path: str) -> Image.Image: - """Load an image from file path.""" - if not os.path.exists(image_path): - raise FileNotFoundError(f"Image file not found: {image_path}") - return Image.open(image_path).convert("RGB") - - -def save_image(image: Image.Image, output_path: str) -> None: - """Save an image to file path.""" - output_dir = os.path.dirname(output_path) - if output_dir: - os.makedirs(output_dir, exist_ok=True) - image.save(output_path) - print(f"Image saved to: {output_path}") - - -def build_prompt_for_t2i( - prompt: str, - height: int = 1024, - width: int = 1024, -) -> dict: - """ - Build prompt dict for text-to-image generation. - - Args: - prompt: Text description for image generation - height: Target image height - width: Target image width - - Returns: - Dict containing prompt and generation parameters - """ - return { - "prompt": prompt, - "height": height, - "width": width, - # Pass target dimensions to AR processor for proper grid token generation - "mm_processor_kwargs": { - "target_h": height, - "target_w": width, - }, - } - - -def build_prompt_for_i2i( - prompt: str, - image: Image.Image, - height: int | None = None, - width: int | None = None, -) -> dict: - """ - Build prompt dict for image-to-image generation. - - Args: - prompt: Text description for image editing - image: Source image for editing - height: Target image height (default: use source image size) - width: Target image width (default: use source image size) - - Returns: - Dict containing prompt, image, and generation parameters - """ - # Use source image dimensions if not specified - if height is None: - height = image.height - if width is None: - width = image.width - - return { - "prompt": prompt, - "multi_modal_data": { - "image": image, - }, - "height": height, - "width": width, - # Pass target dimensions to AR processor for proper grid token generation - "mm_processor_kwargs": { - "target_h": height, - "target_w": width, - }, - } - - -def main(args: argparse.Namespace) -> None: - """Main entry point for GLM-Image end-to-end inference.""" - print("=" * 60) - print("GLM-Image Multistage End-to-End Inference") - print("=" * 60) - - # Validate arguments - if not args.prompt: - raise ValueError("--prompt is required") - - # Determine config path - config_path = args.config_path - if config_path is None: - # Try to find default config - if os.path.exists(DEFAULT_CONFIG_PATH): - config_path = DEFAULT_CONFIG_PATH - else: - # Try relative to script location - script_dir = Path(__file__).parent.parent.parent.parent - config_path = script_dir / "vllm_omni/model_executor/stage_configs/glm_image.yaml" - if not config_path.exists(): - raise FileNotFoundError( - f"Stage config not found. Please specify --config-path. Tried: {DEFAULT_CONFIG_PATH}" - ) - config_path = str(config_path) - - print(f"Model path: {args.model_path}") - print(f"Config path: {config_path}") - print(f"Prompt: {args.prompt}") - - # Load source image for image-to-image mode - source_image = None - if args.image: - print(f"Source image: {args.image}") - source_image = load_image(args.image) - print(f" Image size: {source_image.width}x{source_image.height}") - - # Build prompt based on mode - if source_image is not None: - # Image-to-image mode - prompt_dict = build_prompt_for_i2i( - prompt=args.prompt, - image=source_image, - height=args.height, - width=args.width, - ) - mode = "image-to-image" - else: - # Text-to-image mode - prompt_dict = build_prompt_for_t2i( - prompt=args.prompt, - height=args.height or 1024, - width=args.width or 1024, - ) - mode = "text-to-image" - - print(f"Mode: {mode}") - print(f"Target size: {prompt_dict.get('height', 1024)}x{prompt_dict.get('width', 1024)}") - - # Add generation parameters to prompt - prompt_dict["seed"] = args.seed - prompt_dict["num_inference_steps"] = args.num_inference_steps - prompt_dict["guidance_scale"] = args.guidance_scale - - if args.negative_prompt: - prompt_dict["negative_prompt"] = args.negative_prompt - - # Build cache-dit config if requested - cache_config = None - if args.cache_backend == "cache_dit": - cache_config = { - "Fn_compute_blocks": 1, - "Bn_compute_blocks": 0, - "max_warmup_steps": 4, - "residual_diff_threshold": 0.24, - "max_continuous_cached_steps": 3, - "enable_taylorseer": False, - "taylorseer_order": 1, - "scm_steps_mask_policy": None, - "scm_steps_policy": "dynamic", - } - - # Initialize Omni with multistage config - print("\nInitializing Omni with multistage pipeline...") - print(f" Cache backend: {args.cache_backend or 'None (no acceleration)'}") - start_time = time.time() - - omni = Omni( - model=args.model_path, - stage_configs_path=config_path, - log_stats=args.enable_stats, - stage_init_timeout=args.stage_init_timeout, - cache_backend=args.cache_backend, - cache_config=cache_config, - enable_cache_dit_summary=getattr(args, "enable_cache_dit_summary", False), - enable_diffusion_pipeline_profiler=args.enable_diffusion_pipeline_profiler, - ) - - init_time = time.time() - start_time - print(f"Initialization completed in {init_time:.2f}s") - - # Prepare prompts (support batch generation) - prompts = [prompt_dict for _ in range(args.num_prompts)] - - # No explicit sampling_params for diffusion - parameters are in prompt_dict - # For multistage, the AR stage may need sampling params - from vllm import SamplingParams - - # Compute max_tokens dynamically based on target image size - target_height = prompt_dict.get("height", 1024) - target_width = prompt_dict.get("width", 1024) - is_i2i = source_image is not None - calculated_max_tokens = compute_max_tokens(target_height, target_width, is_i2i=is_i2i) - - # Use calculated value unless user explicitly specified a different value - # Default args.max_tokens is 16384 (very large), so prefer calculated value - effective_max_tokens = calculated_max_tokens if args.max_tokens == 16384 else args.max_tokens - - if args.verbose: - print( - f"AR max_tokens: {effective_max_tokens} " - f"(calculated: {calculated_max_tokens}, arg: {args.max_tokens}, mode: {'i2i' if is_i2i else 't2i'})" - ) - - # IMPORTANT: GLM-Image AR model requires these exact sampling parameters - # from generation_config.json for proper image token generation. - # - temperature=0.9, top_p=0.75, top_k=16512 (vision_vocab_size) - # - stop_token_ids=[16385] (eos_token_id) is CRITICAL to stop generation - ar_sampling_params = SamplingParams( - temperature=0.9, # From generation_config.json - top_p=0.75, # From generation_config.json - top_k=GLM_IMAGE_VISION_VOCAB_SIZE, # 16512, vision vocabulary size - max_tokens=effective_max_tokens, - stop_token_ids=[GLM_IMAGE_EOS_TOKEN_ID], # 16385, CRITICAL for stopping - seed=args.seed, - detokenize=False, - # Keep target size available in runner/model for deterministic M-RoPE - # decode grids in t2i (no mm_features available in this path). - extra_args={ - "target_h": int(target_height), - "target_w": int(target_width), - }, - ) - - # For diffusion stage, sampling_params contains diffusion-specific parameters - # These are passed as kwargs to the diffusion engine - diffusion_sampling_params = OmniDiffusionSamplingParams( - num_inference_steps=args.num_inference_steps, - guidance_scale=args.guidance_scale, - height=prompt_dict.get("height", 1024), - width=prompt_dict.get("width", 1024), - seed=args.seed, - ) - - # For multistage, we need sampling_params for each stage - # Stage 0 (AR): SamplingParams for vLLM - # Stage 1 (Diffusion): dict with diffusion kwargs - sampling_params_list = [ar_sampling_params, diffusion_sampling_params] - - # Run generation - print(f"\nGenerating {args.num_prompts} image(s)...") - gen_start_time = time.time() - - output_dir = os.path.dirname(args.output) if args.output else "outputs" - if output_dir: - os.makedirs(output_dir, exist_ok=True) - - output_count = 0 - for stage_outputs in omni.generate(prompts, sampling_params_list, py_generator=True): - output = stage_outputs.request_output - if stage_outputs.final_output_type == "image": - request_id = output.request_id - - # Get generated images - images = output.images if hasattr(output, "images") else [] - if not images and hasattr(output, "multimodal_output"): - images = output.multimodal_output.get("images", []) - - # Save each generated image - for idx, img in enumerate(images): - if args.num_prompts == 1 and len(images) == 1: - output_path = args.output - else: - base, ext = os.path.splitext(args.output) - output_path = f"{base}_{request_id}_{idx}{ext}" - - if isinstance(img, Image.Image): - save_image(img, output_path) - else: - print(f"Warning: Unexpected image type for request {request_id}: {type(img)}") - - output_count += 1 - - elif stage_outputs.final_output_type == "text": - # AR stage output (intermediate, for debugging) - if args.verbose: - print(f"AR output for request {output.request_id}:") - print(f" Token count: {len(output.outputs[0].token_ids)}") - - gen_time = time.time() - gen_start_time - print(f"\nGeneration completed in {gen_time:.2f}s") - print(f"Generated {output_count} image(s)") - - # Cleanup - omni.close() - print("\nDone!") - - -def parse_args() -> argparse.Namespace: - """Parse command line arguments.""" - parser = argparse.ArgumentParser( - description="GLM-Image Multistage End-to-End Inference", - formatter_class=argparse.RawDescriptionHelpFormatter, - epilog=__doc__, - ) - - # Required arguments - parser.add_argument( - "--model-path", - type=str, - default="zai-org/GLM-Image", - help="Path to GLM-Image model directory or HuggingFace model ID", - ) - parser.add_argument( - "--prompt", - type=str, - required=True, - help="Text prompt for image generation", - ) - - # Optional arguments - parser.add_argument( - "--config-path", - type=str, - default=None, - help="Path to stage config YAML file (default: auto-detect)", - ) - parser.add_argument( - "--image", - type=str, - default=None, - help="Path to source image for image-to-image mode", - ) - parser.add_argument( - "--output", - type=str, - default="output_glm_image.png", - help="Output image path (default: output_glm_image.png)", - ) - parser.add_argument( - "--negative-prompt", - type=str, - default=None, - help="Negative prompt for classifier-free guidance", - ) - - # Generation parameters - parser.add_argument( - "--height", - type=int, - default=None, - help="Output image height (default: 1024 for t2i, source size for i2i)", - ) - parser.add_argument( - "--width", - type=int, - default=None, - help="Output image width (default: 1024 for t2i, source size for i2i)", - ) - parser.add_argument( - "--num-inference-steps", - type=int, - default=50, - help="Number of diffusion denoising steps (default: 50)", - ) - parser.add_argument( - "--guidance-scale", - type=float, - default=1.5, - help="Classifier-free guidance scale (default: 1.5)", - ) - parser.add_argument( - "--seed", - type=int, - default=SEED, - help=f"Random seed for reproducibility (default: {SEED})", - ) - parser.add_argument( - "--max-tokens", - type=int, - default=16384, - help="Maximum tokens for AR generation (default: 16384)", - ) - - # Batch processing - parser.add_argument( - "--num-prompts", - type=int, - default=1, - help="Number of images to generate (default: 1)", - ) - - # Cache acceleration - parser.add_argument( - "--cache-backend", - type=str, - default=None, - choices=["cache_dit"], - help="Cache backend for DiT acceleration. Default: None (no cache).", - ) - parser.add_argument( - "--enable-cache-dit-summary", - action="store_true", - help="Enable cache-dit summary logging after diffusion forward passes.", - ) - - # Runtime options - parser.add_argument( - "--enable-stats", - action="store_true", - default=False, - help="Enable statistics logging", - ) - parser.add_argument( - "--stage-init-timeout", - type=int, - default=600, - help="Timeout for stage initialization in seconds (default: 600)", - ) - parser.add_argument( - "--verbose", - "-v", - action="store_true", - default=False, - help="Enable verbose output", - ) - parser.add_argument( - "--enable-diffusion-pipeline-profiler", - action="store_true", - help="Enable diffusion pipeline profiler to display stage durations.", - ) - - return parser.parse_args() - - -if __name__ == "__main__": - args = parse_args() - main(args) diff --git a/examples/offline_inference/glm_image/run_i2i.sh b/examples/offline_inference/glm_image/run_i2i.sh deleted file mode 100755 index f81b157b0..000000000 --- a/examples/offline_inference/glm_image/run_i2i.sh +++ /dev/null @@ -1,93 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: Apache-2.0 -# Run GLM-Image image-to-image (editing) with multistage pipeline - -set -e - -# Default values -MODEL_PATH="${MODEL_PATH:-/path/to/glm-image}" -CONFIG_PATH="${CONFIG_PATH:-vllm_omni/model_executor/stage_configs/glm_image.yaml}" -PROMPT="${PROMPT:-Transform this image into an oil painting style}" -INPUT_IMAGE="" -OUTPUT="${OUTPUT:-output_i2i.png}" -NUM_STEPS="${NUM_STEPS:-50}" -GUIDANCE="${GUIDANCE:-1.5}" -SEED="${SEED:-42}" - -# Parse command line arguments -while [[ $# -gt 0 ]]; do - case $1 in - --model-path) - MODEL_PATH="$2" - shift 2 - ;; - --config-path) - CONFIG_PATH="$2" - shift 2 - ;; - --prompt) - PROMPT="$2" - shift 2 - ;; - --image) - INPUT_IMAGE="$2" - shift 2 - ;; - --output) - OUTPUT="$2" - shift 2 - ;; - --num-steps) - NUM_STEPS="$2" - shift 2 - ;; - --guidance) - GUIDANCE="$2" - shift 2 - ;; - --seed) - SEED="$2" - shift 2 - ;; - *) - echo "Unknown option: $1" - exit 1 - ;; - esac -done - -# Check if input image is provided -if [ -z "${INPUT_IMAGE}" ]; then - echo "Error: --image is required for image-to-image mode" - echo "Usage: ./run_i2i.sh --image /path/to/input.png [--prompt \"edit instruction\"]" - exit 1 -fi - -if [ ! -f "${INPUT_IMAGE}" ]; then - echo "Error: Input image not found: ${INPUT_IMAGE}" - exit 1 -fi - -echo "==============================================" -echo "GLM-Image Image-to-Image Generation" -echo "==============================================" -echo "Model: ${MODEL_PATH}" -echo "Config: ${CONFIG_PATH}" -echo "Input: ${INPUT_IMAGE}" -echo "Prompt: ${PROMPT}" -echo "Output: ${OUTPUT}" -echo "Steps: ${NUM_STEPS}" -echo "Guidance: ${GUIDANCE}" -echo "Seed: ${SEED}" -echo "==============================================" - -python end2end.py \ - --model-path "${MODEL_PATH}" \ - --config-path "${CONFIG_PATH}" \ - --prompt "${PROMPT}" \ - --image "${INPUT_IMAGE}" \ - --output "${OUTPUT}" \ - --num-inference-steps "${NUM_STEPS}" \ - --guidance-scale "${GUIDANCE}" \ - --seed "${SEED}" \ - --verbose diff --git a/examples/offline_inference/glm_image/run_t2i.sh b/examples/offline_inference/glm_image/run_t2i.sh deleted file mode 100755 index 5d249960b..000000000 --- a/examples/offline_inference/glm_image/run_t2i.sh +++ /dev/null @@ -1,87 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: Apache-2.0 -# Run GLM-Image text-to-image generation with multistage pipeline - -set -e - -# Default values -MODEL_PATH="${MODEL_PATH:-/path/to/glm-image}" -CONFIG_PATH="${CONFIG_PATH:-vllm_omni/model_executor/stage_configs/glm_image.yaml}" -PROMPT="${PROMPT:-A beautiful mountain landscape with snow-capped peaks and a clear blue lake}" -OUTPUT="${OUTPUT:-output_t2i.png}" -HEIGHT="${HEIGHT:-1024}" -WIDTH="${WIDTH:-1024}" -NUM_STEPS="${NUM_STEPS:-50}" -GUIDANCE="${GUIDANCE:-1.5}" -SEED="${SEED:-42}" - -# Parse command line arguments -while [[ $# -gt 0 ]]; do - case $1 in - --model-path) - MODEL_PATH="$2" - shift 2 - ;; - --config-path) - CONFIG_PATH="$2" - shift 2 - ;; - --prompt) - PROMPT="$2" - shift 2 - ;; - --output) - OUTPUT="$2" - shift 2 - ;; - --height) - HEIGHT="$2" - shift 2 - ;; - --width) - WIDTH="$2" - shift 2 - ;; - --num-steps) - NUM_STEPS="$2" - shift 2 - ;; - --guidance) - GUIDANCE="$2" - shift 2 - ;; - --seed) - SEED="$2" - shift 2 - ;; - *) - echo "Unknown option: $1" - exit 1 - ;; - esac -done - -echo "==============================================" -echo "GLM-Image Text-to-Image Generation" -echo "==============================================" -echo "Model: ${MODEL_PATH}" -echo "Config: ${CONFIG_PATH}" -echo "Prompt: ${PROMPT}" -echo "Output: ${OUTPUT}" -echo "Size: ${WIDTH}x${HEIGHT}" -echo "Steps: ${NUM_STEPS}" -echo "Guidance: ${GUIDANCE}" -echo "Seed: ${SEED}" -echo "==============================================" - -python end2end.py \ - --model-path "${MODEL_PATH}" \ - --config-path "${CONFIG_PATH}" \ - --prompt "${PROMPT}" \ - --output "${OUTPUT}" \ - --height "${HEIGHT}" \ - --width "${WIDTH}" \ - --num-inference-steps "${NUM_STEPS}" \ - --guidance-scale "${GUIDANCE}" \ - --seed "${SEED}" \ - --verbose diff --git a/examples/online_serving/glm_image/README.md b/examples/online_serving/glm_image/README.md deleted file mode 100644 index 7ba4b501c..000000000 --- a/examples/online_serving/glm_image/README.md +++ /dev/null @@ -1,204 +0,0 @@ -# GLM-Image Online Serving - -This example demonstrates how to deploy GLM-Image for online image generation using vLLM-Omni. - -## 🛠️ Installation - -Please refer to [README.md](../../../README.md) - -## Run examples (GLM-Image) - -**Note**: These examples work with the default configuration on **2× NVIDIA A100 (80GB)** or equivalent. Stage 0 (AR) and Stage 1 (Diffusion) each use one GPU by default. For single-GPU setups, modify the stage configuration to share the same device. - -### Launch the Server - -```bash -# Use default configuration -vllm serve zai-org/GLM-Image --omni --port 8091 -``` - -Or use the convenience script: - -```bash -cd examples/online_serving/glm_image -bash run_server.sh -``` - -If you have a custom stage configs file: - -```bash -vllm serve zai-org/GLM-Image --omni --port 8091 --stage-configs-path /path/to/glm_image.yaml -``` - -### Send Requests - -Get into the glm_image folder: - -```bash -cd examples/online_serving/glm_image -``` - -Send request via Python: - -```bash -python openai_chat_client.py --prompt "A cute cat sitting on a window sill" -``` - -The Python client supports the following command-line arguments: - -- `--prompt` (or `-p`): Text prompt for generation (default: `A beautiful sunset over the ocean with sailing boats`) -- `--output` (or `-o`): Output file path (default: `glm_image_output.png`) -- `--server` (or `-s`): Server URL (default: `http://localhost:8091`) -- `--image` (or `-i`): Input image path (for image-to-image editing) -- `--height`: Image height in pixels (default: 1024) -- `--width`: Image width in pixels (default: 1024) -- `--steps`: Number of inference steps (default: 50) -- `--guidance-scale`: Classifier-free guidance scale (default: 1.5) -- `--seed`: Random seed (default: 42) -- `--negative`: Negative prompt - -## Modality Control - -GLM-Image supports **text-to-image** and **image-to-image** modes. - -The default yaml configuration deploys AR on GPU 0 and DiT on GPU 1. You can use the default configuration file: [`glm_image.yaml`](../../../vllm_omni/model_executor/stage_configs/glm_image.yaml) - -| Mode | Input | Output | Description | -| -------------- | ------------ | ------ | ---------------------------------- | -| Text-to-Image | Text | Image | Generate images from text prompts | -| Image-to-Image | Image + Text | Image | Edit images with text instructions | - -### Text-to-Image - -```bash -python openai_chat_client.py \ - --prompt "A photorealistic mountain landscape at sunset" \ - --height 1024 \ - --width 1024 \ - --output landscape.png - -# Or use the curl script: -bash run_curl_text_to_image.sh "A futuristic city skyline at night" -``` - -### Image-to-Image (Image Editing) - -```bash -python openai_chat_client.py \ - --prompt "Convert this image to watercolor style" \ - --image input.png \ - --output watercolor.png - -# Or use the curl script: -bash run_curl_image_edit.sh input.png "Convert to watercolor style" -``` - -For general-purpose request methods (curl, OpenAI SDK, Python `requests`), see -the [Text-to-Image](../text_to_image/README.md) and -[Image-to-Image](../image_to_image/README.md) guides. - -## Generation Parameters - -When using `/v1/chat/completions`, pass these inside `extra_body` in the curl -JSON, or via the `extra_body` keyword argument in the OpenAI Python SDK. -When using the dedicated `/v1/images/generations` or `/v1/images/edits` -endpoints, pass the supported generation controls as top-level fields directly. -For image dimensions and count, use `size` and `n` rather than `height` or -`width`. - -| Parameter | Type | Default | Description | -| --------------------- | ----- | ------- | ----------------------------------- | -| `height` | int | 1024 | Image height in pixels | -| `width` | int | 1024 | Image width in pixels | -| `num_inference_steps` | int | 50 | Number of diffusion denoising steps | -| `guidance_scale` | float | 1.5 | Classifier-free guidance scale | -| `seed` | int | None | Optional random seed; `/v1/images/*` generates one server-side if omitted | -| `negative_prompt` | str | None | Negative prompt | - -## Response Format - -```json -{ - "id": "chatcmpl-xxx", - "created": 1234567890, - "model": "zai-org/GLM-Image", - "choices": [ - { - "index": 0, - "message": { - "role": "assistant", - "content": [ - { - "type": "image_url", - "image_url": { - "url": "data:image/png;base64,..." - } - } - ] - }, - "finish_reason": "stop" - } - ], - "usage": {} -} -``` - -## Extract Image - -```bash -# From a saved JSON response -cat response.json | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png -``` - -## Architecture - -GLM-Image uses a 2-stage multistage pipeline: - -``` -Stage 0 (AR Model) Stage 1 (Diffusion) -┌───────────────────┐ ┌─────────────────────┐ -│ vLLM-optimized │ prior │ GlmImagePipeline │ -│ GlmImageFor │──tokens──►│ ┌───────────────┐ │ -│ Conditional │ │ │ DiT Denoiser │ │ -│ Generation │ │ └───────┬───────┘ │ -│ (9B AR model) │ │ ▼ │ -└───────────────────┘ │ ┌───────────────┐ │ - ▲ │ │ VAE Decode │──┼──► Image - │ │ └───────────────┘ │ - Text / Image └─────────────────────┘ - Input -``` - -## VRAM Requirements - -| Stage | VRAM | -| :---------------- | :--------------------- | -| Stage-0 (AR) | **~18 GiB + KV Cache** | -| Stage-1 (DiT+VAE) | **~20 GiB** | -| Total | **~38 GiB + KV Cache** | - -## File Description - -| File | Description | -| --------------------------- | ------------------------------------- | -| `run_server.sh` | Server startup script | -| `run_curl_text_to_image.sh` | Text-to-image curl example | -| `run_curl_image_edit.sh` | Image-to-image (editing) curl example | -| `openai_chat_client.py` | Python client (t2i + i2i) | - -## FAQ - -- If you encounter OOM errors, adjust `gpu_memory_utilization` in the stage config: - -```yaml -# In glm_image.yaml, reduce from default 0.6: -gpu_memory_utilization: 0.5 -``` - -- The first request may be slow due to model warmup. Subsequent requests will be faster. - -- If you encounter `Transformers does not recognize this architecture` error, your have to upgrade `transformers` package to `5.3.0` or above: - -``` -pip install --upgrade transformers -``` diff --git a/examples/online_serving/glm_image/openai_chat_client.py b/examples/online_serving/glm_image/openai_chat_client.py deleted file mode 100644 index e142b0719..000000000 --- a/examples/online_serving/glm_image/openai_chat_client.py +++ /dev/null @@ -1,172 +0,0 @@ -#!/usr/bin/env python3 -""" -GLM-Image OpenAI-compatible chat client for text-to-image and image-to-image. - -Usage: - # Text-to-image - python openai_chat_client.py --prompt "A cute cat" --output output.png - - # Image-to-image (image editing) - python openai_chat_client.py --prompt "Convert to watercolor style" --image input.png --output output.png -""" - -import argparse -import base64 -from pathlib import Path - -import requests - - -def generate_image( - prompt: str, - server_url: str = "http://localhost:8091", - image_path: str | None = None, - height: int = 1024, - width: int = 1024, - steps: int = 50, - guidance_scale: float = 1.5, - seed: int | None = None, - negative_prompt: str | None = None, -) -> bytes | None: - """Generate or edit an image using the chat completions API. - - Args: - prompt: Text description or editing instruction - server_url: Server URL - image_path: Path to input image (for image-to-image editing) - height: Image height in pixels - width: Image width in pixels - steps: Number of inference steps - guidance_scale: Classifier-free guidance scale - seed: Random seed for reproducibility - negative_prompt: Negative prompt - - Returns: - Image bytes or None if failed - """ - # Build message content - content: list[dict] = [{"type": "text", "text": prompt}] - - if image_path: - img_path = Path(image_path) - if not img_path.exists(): - print(f"Error: Image file not found: {image_path}") - return None - b64_data = base64.b64encode(img_path.read_bytes()).decode("utf-8") - suffix = img_path.suffix.lstrip(".").lower() - mime = {"jpg": "jpeg", "jpeg": "jpeg", "png": "png", "webp": "webp"}.get(suffix, "png") - content.append( - { - "type": "image_url", - "image_url": {"url": f"data:image/{mime};base64,{b64_data}"}, - } - ) - - messages = [{"role": "user", "content": content}] - - # Build request payload - extra_body: dict = { - "height": height, - "width": width, - "num_inference_steps": steps, - "guidance_scale": guidance_scale, - } - if seed is not None: - extra_body["seed"] = seed - if negative_prompt: - extra_body["negative_prompt"] = negative_prompt - - payload = {"messages": messages, "extra_body": extra_body} - - # Send request - try: - mode = "image-to-image" if image_path else "text-to-image" - print(f"Sending {mode} request to {server_url}...") - response = requests.post( - f"{server_url}/v1/chat/completions", - headers={"Content-Type": "application/json"}, - json=payload, - timeout=600, - ) - response.raise_for_status() - data = response.json() - - # Extract image from response - choices = data.get("choices", []) - for choice in choices: - choice_content = choice.get("message", {}).get("content") - if isinstance(choice_content, list): - for item in choice_content: - if isinstance(item, dict) and "image_url" in item: - img_url = item["image_url"].get("url", "") - if img_url.startswith("data:image"): - _, b64 = img_url.split(",", 1) - return base64.b64decode(b64) - - print(f"Unexpected response format: {data}") - return None - - except Exception as e: - print(f"Error: {e}") - return None - - -def main(): - parser = argparse.ArgumentParser(description="GLM-Image chat client") - parser.add_argument( - "--prompt", - "-p", - default="A beautiful sunset over the ocean with sailing boats", - help="Text prompt", - ) - parser.add_argument("--output", "-o", default="glm_image_output.png", help="Output file") - parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL") - - # Image-to-image - parser.add_argument( - "--image", - "-i", - type=str, - help="Input image path (for image-to-image editing)", - ) - - # Generation parameters - parser.add_argument("--height", type=int, default=1024, help="Image height") - parser.add_argument("--width", type=int, default=1024, help="Image width") - parser.add_argument("--steps", type=int, default=50, help="Inference steps") - parser.add_argument("--guidance-scale", type=float, default=1.5, help="CFG guidance scale") - parser.add_argument("--seed", type=int, default=42, help="Random seed") - parser.add_argument("--negative", help="Negative prompt") - - args = parser.parse_args() - - mode = "image-to-image" if args.image else "text-to-image" - print(f"Mode: {mode}") - print(f"Prompt: {args.prompt}") - if args.image: - print(f"Input image: {args.image}") - - image_bytes = generate_image( - prompt=args.prompt, - server_url=args.server, - image_path=args.image, - height=args.height, - width=args.width, - steps=args.steps, - guidance_scale=args.guidance_scale, - seed=args.seed, - negative_prompt=args.negative, - ) - - if image_bytes: - output_path = Path(args.output) - output_path.write_bytes(image_bytes) - print(f"Image saved to: {output_path}") - print(f"Size: {len(image_bytes) / 1024:.1f} KB") - else: - print("Failed to generate image") - exit(1) - - -if __name__ == "__main__": - main() diff --git a/examples/online_serving/glm_image/run_curl_image_edit.sh b/examples/online_serving/glm_image/run_curl_image_edit.sh deleted file mode 100755 index bb1e851ba..000000000 --- a/examples/online_serving/glm_image/run_curl_image_edit.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/bin/bash -# GLM-Image image-edit (image-to-image) curl example - -set -euo pipefail - -if [[ $# -lt 2 ]]; then - echo "Usage: $0 \"\" [output_file]" >&2 - exit 1 -fi - -INPUT_IMG=$1 -PROMPT=$2 -SERVER="${SERVER:-http://localhost:8091}" -CURRENT_TIME=$(date +%Y%m%d%H%M%S) -OUTPUT="${3:-glm_image_i2i_${CURRENT_TIME}.png}" - -if [[ ! -f "$INPUT_IMG" ]]; then - echo "Input image not found: $INPUT_IMG" >&2 - exit 1 -fi - -# base64 encode (macOS uses -i, Linux uses -w0) -if [[ "$(uname)" == "Darwin" ]]; then - IMG_B64=$(base64 < "$INPUT_IMG" | tr -d '\n') -else - IMG_B64=$(base64 -w0 "$INPUT_IMG") -fi - -REQUEST_JSON=$( - jq -n --arg prompt "$PROMPT" --arg img "$IMG_B64" '{ - messages: [{ - role: "user", - content: [ - {"type": "text", "text": $prompt}, - {"type": "image_url", "image_url": {"url": ("data:image/png;base64," + $img)}} - ] - }], - extra_body: { - height: 1024, - width: 1024, - num_inference_steps: 50, - guidance_scale: 1.5, - seed: 42 - } - }' -) - -echo "Generating edited image..." -echo "Server: $SERVER" -echo "Prompt: $PROMPT" -echo "Input : $INPUT_IMG" -echo "Output: $OUTPUT" - -curl -s "$SERVER/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d "$REQUEST_JSON" \ - | jq -r '.choices[0].message.content[0].image_url.url' \ - | cut -d',' -f2- \ - | base64 -d > "$OUTPUT" - -echo "Image saved to: $OUTPUT" diff --git a/examples/online_serving/glm_image/run_curl_text_to_image.sh b/examples/online_serving/glm_image/run_curl_text_to_image.sh deleted file mode 100755 index aecb6953c..000000000 --- a/examples/online_serving/glm_image/run_curl_text_to_image.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -# GLM-Image text-to-image curl example - -set -euo pipefail - -PROMPT="${1:-A beautiful sunset over the ocean with sailing boats}" -SERVER="${SERVER:-http://localhost:8091}" -OUTPUT="${OUTPUT:-glm_image_t2i_output.png}" - -echo "Generating image..." -echo "Server: $SERVER" -echo "Prompt: $PROMPT" -echo "Output: $OUTPUT" - -curl -s "$SERVER/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d "{ - \"messages\": [ - {\"role\": \"user\", \"content\": \"$PROMPT\"} - ], - \"extra_body\": { - \"height\": 1024, - \"width\": 1024, - \"num_inference_steps\": 50, - \"guidance_scale\": 1.5, - \"seed\": 42 - } - }" | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > "$OUTPUT" - -echo "Image saved to: $OUTPUT" diff --git a/examples/online_serving/glm_image/run_server.sh b/examples/online_serving/glm_image/run_server.sh deleted file mode 100755 index b47d9f885..000000000 --- a/examples/online_serving/glm_image/run_server.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash -# GLM-Image online serving startup script - -MODEL="${MODEL:-zai-org/GLM-Image}" -PORT="${PORT:-8091}" - -echo "Starting GLM-Image server..." -echo "Model: $MODEL" -echo "Port: $PORT" - -vllm serve "$MODEL" --omni \ - --port "$PORT" diff --git a/vllm_omni/config/pipeline_registry.py b/vllm_omni/config/pipeline_registry.py index 69a004015..98aa13286 100644 --- a/vllm_omni/config/pipeline_registry.py +++ b/vllm_omni/config/pipeline_registry.py @@ -43,6 +43,10 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = { "vllm_omni.model_executor.models.qwen3_tts.pipeline", "QWEN3_TTS_PIPELINE", ), + "glm_image": ( + "vllm_omni.model_executor.models.glm_image.pipeline", + "GLM_IMAGE_PIPELINE", + ), "voxcpm2": ( "vllm_omni.model_executor.models.voxcpm2.pipeline", "VOXCPM2_PIPELINE", diff --git a/vllm_omni/config/stage_config.py b/vllm_omni/config/stage_config.py index 9839a5389..6bd2faf7e 100644 --- a/vllm_omni/config/stage_config.py +++ b/vllm_omni/config/stage_config.py @@ -169,6 +169,12 @@ class StagePipelineConfig: prompt_expand_func: str | None = None cfg_kv_collect_func: str | None = None omni_kv_config: dict[str, Any] | None = None + # Model subdirectory indirections: for multi-component HF repos where the + # stage's config/tokenizer lives in a subdirectory (e.g. GLM-Image's AR + # config is in ``vision_language_encoder/``). Consumed at stage-init time + # by ``stage_init_utils._resolve_model_tokenizer_paths``. + model_subdir: str | None = None + tokenizer_subdir: str | None = None extras: dict[str, Any] = field(default_factory=dict) @@ -185,6 +191,11 @@ class PipelineConfig: # matches ``hf_config.architectures[*]`` against this tuple to route # to the correct pipeline. Leave empty for models with unique model_type. hf_architectures: tuple[str, ...] = () + # Diffusers pipeline class name: for models that ship a ``model_index.json`` + # (no root ``config.json``), the ``_class_name`` field is matched against + # this value to auto-detect the pipeline. Only needed for diffusers-style + # multi-component repos (e.g. GLM-Image). ``None`` = not a diffusers model. + diffusers_class_name: str | None = None def get_stage(self, stage_id: int) -> StagePipelineConfig | None: """Look up a stage by its ID.""" @@ -702,6 +713,13 @@ def _build_engine_args( engine_args["engine_output_type"] = ps.engine_output_type if next_stage_proc: engine_args["custom_process_next_stage_input_func"] = next_stage_proc + # Subdirectory indirections from StagePipelineConfig (structural, not + # deployment knobs). Deploy YAML ``engine_extras`` can still override + # these per-stage if needed. + if ps.model_subdir: + engine_args["model_subdir"] = ps.model_subdir + if ps.tokenizer_subdir: + engine_args["tokenizer_subdir"] = ps.tokenizer_subdir # Pipeline-wide top-level DeployConfig settings, applied to every stage. for name in _PIPELINE_WIDE_ENGINE_FIELDS: @@ -1378,6 +1396,27 @@ class StageConfigFactory: except Exception as e: logger.debug(f"Failed to auto-detect model type for {model}: {e}") + # Fallback for diffusers-style models: check model_index.json. + # Some models (e.g. GLM-Image) have no root config.json but ship a + # model_index.json with _class_name that maps to a pipeline key via + # PipelineConfig.diffusers_class_name. + try: + from vllm.transformers_utils.config import get_hf_file_to_dict + + model_index = get_hf_file_to_dict("model_index.json", model, revision=None) + if model_index and "_class_name" in model_index: + class_name = model_index["_class_name"] + for pipeline_cfg in _PIPELINE_REGISTRY.values(): + if pipeline_cfg.diffusers_class_name == class_name: + logger.info( + "Detected pipeline %r from model_index.json (_class_name=%r)", + pipeline_cfg.model_type, + class_name, + ) + return pipeline_cfg.model_type, None + except Exception: + pass + # Final fallback: some models (e.g. CosyVoice3) ship an empty # config.json and rely on naming conventions. Match the model path # basename against registered pipeline keys — longest match wins diff --git a/vllm_omni/deploy/glm_image.yaml b/vllm_omni/deploy/glm_image.yaml new file mode 100644 index 000000000..47cd02aa8 --- /dev/null +++ b/vllm_omni/deploy/glm_image.yaml @@ -0,0 +1,38 @@ +# GLM-Image deploy: AR (stage 0) + Diffusion (stage 1). +# Topology declared in vllm_omni/model_executor/models/glm_image/pipeline.py. +# +# Fields omitted from a stage fall back to StageDeployConfig dataclass +# defaults (see vllm_omni/config/stage_config.py). Pipeline-wide settings +# (trust_remote_code, distributed_executor_backend, etc.) use DeployConfig +# defaults unless overridden here. +async_chunk: false + +stages: + # Stage 0: AR Model (GlmImageForConditionalGeneration) + # Generates prior_token_ids for conditioning the diffusion process. + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.6 + enforce_eager: false + max_num_batched_tokens: 32768 + devices: "0" + default_sampling_params: + temperature: 0.9 + top_p: 0.75 + top_k: 16512 + stop_token_ids: [16385] + seed: 42 + detokenize: false + + # Stage 1: Diffusion (DiT + VAE) + # Receives prior_token_ids from AR, performs denoising + VAE decode. + - stage_id: 1 + max_num_seqs: 1 + enforce_eager: true + devices: "1" + default_sampling_params: + seed: 42 + num_inference_steps: 50 + guidance_scale: 1.5 + height: 1024 + width: 1024 diff --git a/vllm_omni/model_executor/models/glm_image/__init__.py b/vllm_omni/model_executor/models/glm_image/__init__.py index d37044c09..6153f07c0 100644 --- a/vllm_omni/model_executor/models/glm_image/__init__.py +++ b/vllm_omni/model_executor/models/glm_image/__init__.py @@ -1,3 +1,16 @@ -from .glm_image_ar import GlmImageForConditionalGeneration +def __getattr__(name: str): + """Lazy import to avoid importing transformers.models.glm_image at module init. + + The AR model depends on ``transformers.models.glm_image`` which is only + available when the model weights are present. Importing it eagerly in + ``__init__.py`` breaks the pipeline registry lookup for environments that + don't have this custom transformers extension installed. + """ + if name == "GlmImageForConditionalGeneration": + from .glm_image_ar import GlmImageForConditionalGeneration + + return GlmImageForConditionalGeneration + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + __all__ = ["GlmImageForConditionalGeneration"] diff --git a/vllm_omni/model_executor/models/glm_image/pipeline.py b/vllm_omni/model_executor/models/glm_image/pipeline.py new file mode 100644 index 000000000..bd041292c --- /dev/null +++ b/vllm_omni/model_executor/models/glm_image/pipeline.py @@ -0,0 +1,47 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""GLM-Image pipeline topologies (frozen). +Two-stage (default): + Stage 0: AR — multimodal understanding + token_ids generation + Stage 1: DiT — diffusion image generation +""" + +from vllm_omni.config.stage_config import ( + PipelineConfig, + StageExecutionType, + StagePipelineConfig, +) + +GLM_IMAGE_PIPELINE = PipelineConfig( + model_type="glm_image", + model_arch="GlmImageForConditionalGeneration", + hf_architectures=("GlmImageForConditionalGeneration",), + diffusers_class_name="GlmImagePipeline", + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="ar", + execution_type=StageExecutionType.LLM_AR, + requires_multimodal_data=True, + input_sources=(), + final_output=False, + owns_tokenizer=True, + model_arch="GlmImageForConditionalGeneration", + engine_output_type="token_ids", + model_subdir="vision_language_encoder", + tokenizer_subdir="processor", + ), + StagePipelineConfig( + stage_id=1, + model_stage="dit", + execution_type=StageExecutionType.DIFFUSION, + input_sources=(0,), + requires_multimodal_data=True, + final_output=True, + final_output_type="image", + model_arch="GlmImagePipeline", + custom_process_input_func="vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion", + omni_kv_config={"need_recv_cache": False}, + ), + ), +) diff --git a/vllm_omni/model_executor/stage_configs/glm_image.yaml b/vllm_omni/model_executor/stage_configs/glm_image.yaml deleted file mode 100644 index f3ed6c721..000000000 --- a/vllm_omni/model_executor/stage_configs/glm_image.yaml +++ /dev/null @@ -1,74 +0,0 @@ -# Stage config for running GLM-Image with 2-stage architecture -# Stage 0: AR Model (vLLM implementation) - generates prior_token_ids -# Stage 1: Diffusion (DiT + VAE) - denoising and image decoding - -stage_args: - # Stage 0: AR Model (GlmImageForConditionalGeneration) - # This stage uses the vLLM-optimized AR model to generate prior tokens - # for conditioning the diffusion process. - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0" - requires_multimodal_data: true # Required for i2i mode to receive source images - engine_args: - model_stage: ar - max_num_seqs: 1 - model_arch: GlmImageForConditionalGeneration - model_subdir: vision_language_encoder # AR model config.json is in this subdirectory - tokenizer_subdir: processor # Use processor's tokenizer (not ByT5 from tokenizer/) - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.6 - enforce_eager: false - trust_remote_code: true - engine_output_type: token_ids # Output prior_token_ids for diffusion stage - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - final_output: false # AR is not the final output - is_comprehension: true - default_sampling_params: - temperature: 0.9 # From model's generation_config.json - top_p: 0.75 # From model's generation_config.json - top_k: 16512 # vision_vocab_size from generation_config.json - stop_token_ids: [16385] # eos_token_id from generation_config.json - seed: 42 - detokenize: false - - # Stage 1: Diffusion (DiT + VAE) - # This stage receives prior_token_ids from AR and performs denoising + VAE decode - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "1" # Can use different GPU, or same GPU if memory allows - requires_multimodal_data: true # Required for i2i mode to pass condition images - engine_args: - model_stage: dit - max_num_seqs: 1 - model_arch: GlmImagePipeline # Required for diffusion model class resolution - # Diffusion-specific parameters - num_gpus: 1 - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - engine_input_source: [0] # Input from AR stage - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion - final_output: true - final_output_type: image - default_sampling_params: - # Diffusion-specific parameters only (no LLM params like temperature/top_p/top_k) - seed: 42 - num_inference_steps: 50 - guidance_scale: 1.5 - height: 1024 - width: 1024 - -# Top-level runtime config -runtime: - enabled: true - edges: - - from: 0 # AR → Diffusion: trigger after AR completes - to: 1 diff --git a/vllm_omni/model_executor/stage_configs/glm_image_muilticonnector.yaml b/vllm_omni/model_executor/stage_configs/glm_image_muilticonnector.yaml deleted file mode 100644 index 2a85a6dad..000000000 --- a/vllm_omni/model_executor/stage_configs/glm_image_muilticonnector.yaml +++ /dev/null @@ -1,87 +0,0 @@ -# Stage config for running GLM-Image with 2-stage architecture (MultiConnector version) -# Stage 0: AR Model (vLLM implementation) - generates prior_token_ids -# Stage 1: Diffusion (DiT + VAE) - denoising and image decoding -# -# This config uses OmniConnectors for inter-stage communication, -# enabling efficient tensor transfer between stages on different processes/nodes. - -stage_args: - # Stage 0: AR Model (GlmImageForConditionalGeneration) - # This stage uses the vLLM-optimized AR model to generate prior tokens - # for conditioning the diffusion process. - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0" - requires_multimodal_data: true # Required for i2i mode to receive source images - engine_args: - model_stage: ar - max_num_seqs: 1 - model_arch: GlmImageForConditionalGeneration - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.6 - enforce_eager: false - trust_remote_code: true - engine_output_type: token_ids # Output prior_token_ids for diffusion stage - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - hf_config_name: vision_language_encoder # Subfolder in model path - final_output: false # AR is not the final output - is_comprehension: true - default_sampling_params: - temperature: 0.9 # From model's generation_config.json - top_p: 0.75 # From model's generation_config.json - top_k: 16512 # vision_vocab_size from generation_config.json - stop_token_ids: [16385] # eos_token_id from generation_config.json - seed: 42 - detokenize: false - - # Stage 1: Diffusion (DiT + VAE) - # This stage receives prior_token_ids from AR and performs denoising + VAE decode - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "1" # Use separate GPU for diffusion - requires_multimodal_data: true # Required for i2i mode to pass condition images - engine_args: - model_stage: dit - max_num_seqs: 1 - # Diffusion-specific parameters - num_gpus: 1 - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - engine_input_source: [0] # Input from AR stage - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.glm_image.ar2diffusion - final_output: true - final_output_type: image - default_sampling_params: - seed: 42 - num_inference_steps: 50 - guidance_scale: 1.5 - height: 1024 - width: 1024 - -# Top-level runtime config with MultiConnector support -runtime: - enabled: true - edges: - - from: 0 # AR → Diffusion - to: 1 - -# OmniConnector configuration for efficient inter-stage tensor transfer -connectors: - - type: tensor_transfer - source_stage: 0 - target_stage: 1 - # Transfer prior_token_ids efficiently between stages - fields: - - name: prior_token_ids - dtype: int64 - - name: prior_token_image_ids - dtype: int64 - optional: true