[Refactor]Base class for dit pipelines with unified parameter declaration[1/N] (#4225)

Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
authored and GitHub committed 2026-06-18 17:14:18 +08:00
1 parent dff8cda844
commit f75c437a60
26 files changed
+1026 -1586

No files matched your search

-391
View File
@@ -1,391 +0,0 @@
# BAGEL-7B-MoT
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
## Architecture
BAGEL-7B-MoT is a Mixture-of-Transformers (MoT) model supporting both image generation and understanding. It offers two deployment topologies:
| Topology | Stages | Description |
| :------- | :----- | :---------- |
| **Two-stage** (default) | Stage 0 (Thinker, AR) + Stage 1 (DiT, Diffusion) | Thinker handles text/understanding via vLLM AR engine; DiT handles image generation. KV cache is transferred between stages. |
| **Single-stage** | Stage 0 (DiT, Diffusion) only | The DiT stage contains a full LLM, ViT, VAE, and tokenizer internally. All modalities are handled within a single diffusion process. |
Both topologies support all four modalities: `text2img`, `img2img`, `img2text`, `text2text`.
## Quick Start
```bash
cd examples/offline_inference/bagel
# Default two-stage mode (auto-detected)
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat"
# Single-stage mode
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat" \
--deploy-config vllm_omni/deploy/bagel_single_stage.yaml
```
> **Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. For dual-GPU setups, modify the deploy YAML to distribute stages across devices.
## Modality Control
Control the mode using the `--modality` argument:
| Modality | Input | Output | Description |
| :------- | :---- | :----- | :---------- |
| `text2img` | Text | Image | Generate images from text prompts |
| `img2img` | Image + Text | Image | Transform images using text guidance |
| `img2text` | Image + Text | Text | Generate text descriptions from images |
| `text2text` | Text | Text | Pure text generation (language model mode) |
### Text to Image (text2img)
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat" \
--steps 50
```
### Image to Image (img2img)
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality img2img \
--image-path /path/to/image.jpg \
--prompts "Let the woman wear a blue dress"
```
### Image to Text (img2text)
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality img2text \
--image-path /path/to/image.jpg \
--prompts "Describe this image in detail"
```
### Text to Text (text2text)
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2text \
--prompts "What is the capital of France?"
# Load prompts from a text file (one prompt per line):
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2text \
--txt-prompts /path/to/prompts.txt
```
## Think Mode
Think mode enables the model to generate `<think>...</think>` planning/reasoning tokens before producing the final output. This improves generation quality for complex prompts.
- **Two-stage**: The Thinker (AR) stage decodes think tokens, then transfers the augmented KV cache to the DiT stage for image generation.
- **Single-stage**: The DiT's internal LLM generates think tokens in-place before proceeding to denoise.
```bash
# Think + text2img: plan before generating
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A futuristic city with flying cars" \
--think \
--max-think-tokens 1000
# Think + img2img: reason about the edit
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality img2img \
--image-path /path/to/image.jpg \
--prompts "Make it look like a watercolor painting" \
--think
# Think + img2text: reason before describing
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality img2text \
--image-path /path/to/image.jpg \
--prompts "What is happening in this image?" \
--think
# Think + text2text: chain-of-thought reasoning
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2text \
--prompts "Solve: 23 * 47" \
--think
```
Think mode parameters:
| Argument | Default | Description |
| :------- | :------ | :---------- |
| `--think` | `False` | Enable thinking mode |
| `--max-think-tokens` | `1000` | Maximum tokens for think generation |
| `--do-sample` | `False` | Enable sampling (vs. greedy) for text generation |
| `--text-temperature` | `0.3` | Temperature for text generation sampling |
## Classifier-Free Guidance (CFG)
CFG controls the trade-off between prompt fidelity and diversity. These parameters apply to image generation modalities (`text2img`, `img2img`).
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A photorealistic portrait" \
--cfg-text-scale 6.0 \
--cfg-img-scale 2.0 \
--negative-prompt "blurry, low quality, distorted" \
--cfg-interval 0.4 1.0 \
--cfg-renorm-type global \
--cfg-renorm-min 0.0
```
| Argument | Default | Description |
| :------- | :------ | :---------- |
| `--cfg-text-scale` | `4.0` | Text CFG scale (higher = more prompt-adherent) |
| `--cfg-img-scale` | `1.5` | Image CFG scale (for img2img) |
| `--negative-prompt` | `None` | Negative prompt for CFG conditioning |
| `--cfg-interval` | pipeline default | CFG active interval `[start, end]` as fractions of total timesteps |
| `--cfg-renorm-type` | `None` | Renormalization type: `global`, `text_channel`, `channel` |
| `--cfg-renorm-min` | `None` | Minimum renormalization value |
| `--cfg-parallel-size` | `1` | CFG parallel size: `1` = batched (single GPU), `2` = 2-branch parallel, `3` = full 3-GPU parallel |
## Deployment Topologies
### Two-Stage (Default)
The default topology auto-detected from the model. No extra flags needed.
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat"
```
The pipeline is defined in [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml). Stage 0 (Thinker) and Stage 1 (DiT) share GPU 0 by default. For dual-GPU setups, customize the deploy YAML and set `devices: "1"` for stage 1.
### Single-Stage
Pass the single-stage deploy config via `--deploy-config`:
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat" \
--deploy-config vllm_omni/deploy/bagel_single_stage.yaml
```
See [`bagel_single_stage.yaml`](../../../vllm_omni/deploy/bagel_single_stage.yaml) for configuration details. The `pipeline: bagel_single_stage` field selects the single-stage topology from the pipeline registry.
### Tensor Parallelism (TP)
For larger models or multi-GPU environments, customize the deploy YAML (see [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml)) and set per-stage `tensor_parallel_size` and `devices`:
```yaml
# Example: TP=2 on GPUs 0,1 for the Thinker stage
stages:
- stage_id: 0
tensor_parallel_size: 2
devices: "0,1"
```
### VAE Patch Parallelism
[VAE Patch Parallelism](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism/vae_patch_parallel.html) splits Bagel VAE **decode/encode** tiles across multiple GPUs on the **DiT stage**, reducing **per-GPU peak memory during VAE decode**. Use it when high-resolution `text2img` or `img2img` hits VAE OOM or large decode spikes.
**Bagel-specific notes:**
- Implemented in `BagelPipeline` via `DistributedAutoEncoder` (DiT stage only).
- **Single-stage** is the simplest path: one DiT process with TP + VAE patch parallel.
- **Two-stage**: enable on **stage 1 (DiT)** only; stage 0 (Thinker) keeps encoder-only `VAEEncoder` and does not use VAE patch parallel.
- You need a DiT `world_size` ≥ `vae_patch_parallel_size` (typically `tensor_parallel_size=2` on that stage). VAE PP reuses the DiT process group; it is not a standalone second-GPU VAE worker.
**Single-stage via deploy YAML** (recommended for `end2end.py`):
```yaml
pipeline: bagel_single_stage
async_chunk: false
stages:
- stage_id: 0
max_num_batched_tokens: 32768
max_num_seqs: 1
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
devices: "0,1"
vae_use_tiling: true
parallel_config:
tensor_parallel_size: 2
vae_patch_parallel_size: 2
default_sampling_params:
seed: 52
```
```bash
cd examples/offline_inference/bagel
CUDA_VISIBLE_DEVICES=0,1 python end2end.py \
--model /path/to/BAGEL-7B-MoT \
--deploy-config /path/to/bagel_single_stage_vae_pp.yaml \
--modality text2img \
--prompts "A cute cat" \
--steps 10 \
--output ./out_vae_pp
```
**Single-stage via `Omni` kwargs** (same flags as online serving):
```python
from vllm_omni.entrypoints.omni import Omni
omni = Omni(
model="ByteDance-Seed/BAGEL-7B-MoT",
deploy_config="vllm_omni/deploy/bagel_single_stage.yaml",
tensor_parallel_size=2,
vae_patch_parallel_size=2,
vae_use_tiling=True,
)
# Then call omni.generate(...) as in end2end.py
```
**Two-stage (VAE PP on DiT only):**
```yaml
stages:
- stage_id: 0
devices: "0"
# AR Thinker — no vae_patch_parallel here
- stage_id: 1
devices: "0,1"
vae_use_tiling: true
parallel_config:
tensor_parallel_size: 2
vae_patch_parallel_size: 2
```
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--deploy-config /path/to/bagel_vae_pp.yaml \
--modality text2img \
--prompts "A cute cat"
```
**Startup log checks:**
```text
INFO ... vae_patch_parallel_size=2 requires vae_use_tiling; automatically enabling it.
```
| Setting | Role |
| :------ | :--- |
| `parallel_config.tensor_parallel_size` | DiT world size / TP (must be ≥ `vae_patch_parallel_size`) |
| `parallel_config.vae_patch_parallel_size` | Number of ranks for distributed VAE tiles (`1` = off) |
| `vae_use_tiling` | Enable spatial tiling (auto-enabled when `vae_patch_parallel_size > 1`) |
#### Hybrid Sharded Data Parallel (HSDP)
For larger Bagel deployments on multiple GPUs, you can enable HSDP (Hybrid Sharded Data Parallel) by modifying the stage configuration (for example, [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml)). HSDP shards transformer weights across GPUs to reduce per-GPU memory usage.
1. **Enable HSDP**: Set `use_hsdp: true`.
2. **Set shard size**: Set `hsdp_shard_size` to the number of GPUs used for sharding (for example, `4`).
3. **Set replicate size**: Usually keep `hsdp_replicate_size: 1` unless you want replicated HSDP groups.
4. **Set devices**: Specify the comma-separated GPU IDs used by the diffusion stage (for example, `"0,1,2,3"`).
Example configuration for HSDP across 4 GPUs:
```yaml
- stage_id: 1
devices: "0,1,2,3"
parallel_config:
use_hsdp: true
hsdp_shard_size: 4
hsdp_replicate_size: 1
```
Then pass the custom deploy YAML:
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat" \
--deploy-config /path/to/custom_bagel.yaml
```
### FP8 Quantization
```bash
python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
--modality text2img \
--prompts "A cute cat" \
--quantization fp8
```
## Command Line Reference
### Core Arguments
| Argument | Type | Default | Description |
| :------- | :--- | :------ | :---------- |
| `--model` | string | `ByteDance-Seed/BAGEL-7B-MoT` | Model path or HuggingFace name |
| `--modality` | choice | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
| `--prompts` | list | `None` | Input text prompts |
| `--txt-prompts` | string | `None` | Path to text file with one prompt per line |
| `--image-path` | string | `None` | Input image path (required for `img2img`/`img2text`) |
| `--output` | string | `.` | Output directory for saved images |
| `--steps` | int | `50` | Number of diffusion inference steps |
| `--seed` | int | `None` | Random seed for reproducibility |
### Think Mode Arguments
| Argument | Type | Default | Description |
| :------- | :--- | :------ | :---------- |
| `--think` | flag | `False` | Enable `<think>...</think>` planning/reasoning |
| `--max-think-tokens` | int | `1000` | Maximum tokens for think generation |
| `--do-sample` | flag | `False` | Use sampling instead of greedy decoding |
| `--text-temperature` | float | `0.3` | Sampling temperature for text generation |
### CFG Arguments
| Argument | Type | Default | Description |
| :------- | :--- | :------ | :---------- |
| `--cfg-text-scale` | float | `4.0` | Text CFG guidance scale |
| `--cfg-img-scale` | float | `1.5` | Image CFG guidance scale |
| `--negative-prompt` | string | `None` | Negative prompt for CFG |
| `--cfg-parallel-size` | int | `1` | CFG parallel GPU count (1, 2, or 3) |
| `--cfg-interval` | float[2] | pipeline default | CFG active window `[start, end]` |
| `--cfg-renorm-type` | string | `None` | `global`, `text_channel`, or `channel` |
| `--cfg-renorm-min` | float | `None` | Minimum renormalization value |
### Engine Arguments
| Argument | Type | Default | Description |
| :------- | :--- | :------ | :---------- |
| `--deploy-config` | string | `None` | Path to deploy YAML (auto-detected if omitted) |
| `--worker-backend` | choice | `process` | `process` or `ray` |
| `--ray-address` | string | `None` | Ray cluster address |
| `--quantization` | string | `None` | Quantization method (e.g. `fp8`) |
| `--log-stats` | flag | `False` | Enable statistics logging |
| `--init-timeout` | int | `300` | Initialization timeout (seconds) |
| `--batch-timeout` | int | `5` | Batch timeout (seconds) |
| `--enable-diffusion-pipeline-profiler` | flag | `False` | Profile diffusion stage durations |
## FAQ
- If you encounter OOM errors, try decreasing `max_model_len` or `gpu_memory_utilization` in the deploy YAML.
**Two-stage VRAM usage:**
| Stage | VRAM |
| :---- | :--- |
| Stage 0 (Thinker) | **15.04 GiB + KV Cache** |
| Stage 1 (DiT) | **26.50 GiB** |
| Total | **~42 GiB + KV Cache** |
**Single-stage VRAM usage:** The DiT loads the full model (~42 GiB) in one process.
-278
View File
@@ -1,278 +0,0 @@
import argparse
import os
from vllm_omni.inputs.data import OmniPromptType
from vllm_omni.model_executor.stage_input_processors.bagel import (
GEN_THINK_SYSTEM_PROMPT,
VLM_THINK_SYSTEM_PROMPT,
)
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"--model",
default="ByteDance-Seed/BAGEL-7B-MoT",
help="Path to merged model directory.",
)
parser.add_argument("--prompts", nargs="+", default=None, help="Input text prompts.")
parser.add_argument(
"--txt-prompts",
type=str,
default=None,
help="Path to a .txt file with one prompt per line (preferred).",
)
parser.add_argument("--prompt-type", default="text", choices=["text"])
parser.add_argument(
"--modality",
default="text2img",
choices=["text2img", "img2img", "img2text", "text2text"],
help="Modality mode to control stage execution.",
)
parser.add_argument(
"--image-path",
type=str,
default=None,
help="Path to input image for img2img.",
)
parser.add_argument(
"--output",
type=str,
default=".",
help="Output directory to save images.",
)
# OmniLLM init args
parser.add_argument("--log-stats", action="store_true", default=False)
parser.add_argument("--init-sleep-seconds", type=int, default=20)
parser.add_argument("--batch-timeout", type=int, default=5)
parser.add_argument("--init-timeout", type=int, default=300)
parser.add_argument("--shm-threshold-bytes", type=int, default=65536)
parser.add_argument("--worker-backend", type=str, default="process", choices=["process", "ray"])
parser.add_argument("--ray-address", type=str, default=None)
parser.add_argument(
"--deploy-config",
type=str,
default=None,
help="Path to deploy YAML. If unset, auto-loads vllm_omni/deploy/bagel.yaml based on the HF model_type.",
)
parser.add_argument("--steps", type=int, default=50, help="Number of inference steps.")
parser.add_argument("--cfg-text-scale", type=float, default=4.0, help="Text CFG scale (default: 4.0)")
parser.add_argument("--cfg-img-scale", type=float, default=1.5, help="Image CFG scale (default: 1.5)")
parser.add_argument(
"--negative-prompt", type=str, default=None, help="Negative prompt for CFG (default: empty prompt)"
)
parser.add_argument(
"--cfg-parallel-size",
type=int,
default=1,
choices=[1, 2, 3],
help="CFG parallel size: 1=batched (single GPU), 2=parallel with 2 branches (text CFG only), 3=parallel (3 GPUs).",
)
parser.add_argument("--seed", type=int, default=None, help="Random seed for generation.")
parser.add_argument(
"--cfg-interval",
type=float,
nargs=2,
default=None,
help="CFG interval [start, end] (default: pipeline default)",
)
parser.add_argument(
"--cfg-renorm-type", type=str, default=None, help="CFG renorm type: global, text_channel, channel"
)
parser.add_argument("--cfg-renorm-min", type=float, default=None, help="CFG renorm min")
parser.add_argument(
"--enable-diffusion-pipeline-profiler",
action="store_true",
help="Enable diffusion pipeline profiler to display stage durations.",
)
parser.add_argument(
"--quantization",
type=str,
default=None,
help="Quantization method (e.g. 'fp8').",
)
parser.add_argument(
"--think",
action="store_true",
default=False,
help="Enable thinking mode: AR stage decodes <think>...</think> planning tokens before image generation.",
)
parser.add_argument(
"--max-think-tokens",
type=int,
default=1000,
help="Maximum number of tokens for thinking text generation (default: 1000).",
)
parser.add_argument(
"--do-sample",
action="store_true",
default=False,
help="Enable sampling for text generation (default: greedy).",
)
parser.add_argument(
"--text-temperature",
type=float,
default=0.3,
help="Temperature for text generation sampling (default: 0.3).",
)
args = parser.parse_args()
return args
def main():
args = parse_args()
os.makedirs(args.output, exist_ok=True)
model_name = args.model
prompts: list[OmniPromptType] = []
try:
if getattr(args, "txt_prompts", None) and args.prompt_type == "text":
with open(args.txt_prompts, encoding="utf-8") as f:
lines = [ln.strip() for ln in f.readlines()]
prompts = [ln for ln in lines if ln != ""]
print(f"[Info] Loaded {len(prompts)} prompts from {args.txt_prompts}")
else:
prompts = args.prompts
except Exception as e:
print(f"[Error] Failed to load prompts: {e}")
raise
if not prompts:
prompts = ["A cute cat"]
print(f"[Info] No prompts provided, using default: {prompts}")
from PIL import Image
from vllm_omni.entrypoints.omni import Omni
omni_kwargs = vars(args).copy()
omni_kwargs.pop("model", None)
deploy_config = args.deploy_config
if args.think and deploy_config is None:
deploy_config = "vllm_omni/deploy/bagel_think.yaml"
print(f"[Info] Think mode enabled, using deploy config: {deploy_config}")
if deploy_config:
omni_kwargs["deploy_config"] = deploy_config
if args.quantization:
omni_kwargs["quantization_config"] = args.quantization
omni = Omni(model=model_name, **omni_kwargs)
formatted_prompts = []
for p in prompts:
if args.modality == "img2img":
if not args.image_path or not os.path.exists(args.image_path):
raise ValueError(f"img2img requires --image-path pointing to an existing file, got: {args.image_path}")
loaded_image = Image.open(args.image_path).convert("RGB")
think_prefix = f"<|im_start|>{GEN_THINK_SYSTEM_PROMPT}<|im_end|>" if args.think else ""
final_prompt_text = f"{think_prefix}<|fim_middle|><|im_start|>{p}<|im_end|>"
prompt_dict = {
"prompt": final_prompt_text,
"multi_modal_data": {"img2img": loaded_image},
"modalities": ["img2img"],
}
if args.negative_prompt is not None:
prompt_dict["negative_prompt"] = args.negative_prompt
formatted_prompts.append(prompt_dict)
elif args.modality == "img2text":
if args.image_path:
loaded_image = Image.open(args.image_path).convert("RGB")
think_prefix = f"<|im_start|>system\n{VLM_THINK_SYSTEM_PROMPT}<|im_end|>\n" if args.think else ""
final_prompt_text = (
f"{think_prefix}<|im_start|>user\n<|image_pad|>\n{p}<|im_end|>\n<|im_start|>assistant\n"
)
prompt_dict = {
"prompt": final_prompt_text,
"multi_modal_data": {"image": loaded_image},
"modalities": ["text"],
}
formatted_prompts.append(prompt_dict)
elif args.modality == "text2text":
think_prefix = f"<|im_start|>{VLM_THINK_SYSTEM_PROMPT}<|im_end|>" if args.think else ""
final_prompt_text = f"{think_prefix}<|im_start|>{p}<|im_end|><|im_start|>"
prompt_dict = {"prompt": final_prompt_text, "modalities": ["text"]}
formatted_prompts.append(prompt_dict)
else:
think_prefix = f"<|im_start|>{GEN_THINK_SYSTEM_PROMPT}<|im_end|>" if args.think else ""
final_prompt_text = f"{think_prefix}<|im_start|>{p}<|im_end|>"
prompt_dict = {"prompt": final_prompt_text, "modalities": ["image"]}
if args.negative_prompt is not None:
prompt_dict["negative_prompt"] = args.negative_prompt
formatted_prompts.append(prompt_dict)
params_list = omni.default_sampling_params_list
# Bagel exposes 1 sampling param set for single-stage (DiT-only) and
# 2 for two-stage (Thinker + DiT). This heuristic may need updating
# if future pipelines break that 1:1 mapping.
is_single_stage = len(params_list) == 1
diffusion_params_idx = 0 if is_single_stage else (1 if len(params_list) > 1 else 0)
diffusion_params = params_list[diffusion_params_idx]
if args.modality in ("text2img", "img2img"):
diffusion_params.num_inference_steps = args.steps # type: ignore
diffusion_params.cfg_parallel_size = args.cfg_parallel_size # type: ignore
if args.seed is not None:
diffusion_params.seed = args.seed # type: ignore
extra = getattr(diffusion_params, "extra_args", {}) or {}
extra["cfg_text_scale"] = args.cfg_text_scale
extra["cfg_img_scale"] = args.cfg_img_scale
if args.cfg_interval is not None:
extra["cfg_interval"] = tuple(args.cfg_interval)
if args.cfg_renorm_type is not None:
extra["cfg_renorm_type"] = args.cfg_renorm_type
if args.cfg_renorm_min is not None:
extra["cfg_renorm_min"] = args.cfg_renorm_min
if args.negative_prompt is not None:
extra["negative_prompt"] = args.negative_prompt
needs_text_gen = is_single_stage and (args.think or args.modality in ("text2text", "img2text"))
if needs_text_gen:
if args.think:
extra["think"] = True
extra["max_think_tokens"] = args.max_think_tokens
extra["do_sample"] = args.do_sample
extra["text_temperature"] = args.text_temperature
diffusion_params.extra_args = extra # type: ignore
omni_outputs = list(omni.generate(prompts=formatted_prompts, sampling_params_list=params_list))
img_idx = 0
for req_output in omni_outputs:
# 2-stage think mode: text output from thinker stage
ro = getattr(req_output, "request_output", None)
if ro and getattr(ro, "outputs", None):
txt = "".join(getattr(o, "text", "") or "" for o in ro.outputs)
if txt:
if args.think:
print(f"[Think]\n{txt}")
else:
print(f"[Output] Text:\n{txt}")
# Single-stage DiT: text from custom_output
custom = getattr(req_output, "_custom_output", {}) or {}
if custom.get("think_text"):
print(f"[Think]\n{custom['think_text']}")
if custom.get("text_output"):
print(f"[Output] Text:\n{custom['text_output']}")
images = getattr(req_output, "images", None)
if not images:
continue
for j, img in enumerate(images):
save_path = os.path.join(args.output, f"output_{img_idx}_{j}.png")
img.save(save_path)
print(f"[Output] Saved image to {save_path}")
img_idx += 1
if __name__ == "__main__":
main()
@@ -97,9 +97,16 @@ import torch
from PIL import Image
from vllm_omni.diffusion.data import DiffusionParallelConfig
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.model_extras import (
build_image_to_image_prompt,
get_extra_body_params,
get_model_class_name,
should_init_extra_args_for_non_diffusion_stages,
)
from vllm_omni.platforms import current_omni_platform
@@ -187,6 +194,12 @@ def parse_args() -> argparse.Namespace:
parser.add_argument(
"--guidance-scale-2", type=float, default=None, help="image guidance scale for image-to-image generation."
)
parser.add_argument(
"--extra-args",
type=parse_profiler_config,
default=None,
help="JSON object copied to OmniDiffusionSamplingParams.extra_args, e.g. '{\"cfg_text_scale\": 4.0}'.",
)
parser.add_argument(
"--output",
type=str,
@@ -468,6 +481,8 @@ def main():
enable_diffusion_pipeline_profiler=args.enable_diffusion_pipeline_profiler,
profiler_config=args.profiler_config,
)
model_class_name = get_model_class_name(omni)
declared_extra_body_params = get_extra_body_params(model_class_name)
print("Pipeline loaded")
profiler_enabled = args.profiler_config is not None
@@ -478,6 +493,8 @@ def main():
print(f" Model: {args.model}")
print(f" Inference steps: {args.num_inference_steps}")
print(f" Cache backend: {args.cache_backend if args.cache_backend else 'None (no acceleration)'}")
if args.height is not None or args.width is not None:
print(f" Output size: {args.width or 'auto'}x{args.height or 'auto'}")
if isinstance(input_image, list):
print(f" Number of input images: {len(input_image)}")
for idx, img in enumerate(input_image):
@@ -487,7 +504,6 @@ def main():
print(
f" Parallel configuration: ulysses_degree={args.ulysses_degree}, ring_degree={args.ring_degree}, cfg_parallel_size={args.cfg_parallel_size}, tensor_parallel_size={args.tensor_parallel_size}, enable_expert_parallel: {args.enable_expert_parallel}"
)
print(f" Output image size: {args.width or 'default'}x{args.height or 'default'}")
print(f"{'=' * 60}\n")
generation_start = time.perf_counter()
@@ -496,10 +512,20 @@ def main():
print("[Profiler] Starting profiling...")
omni.start_profile()
# Generate edited image
sampling_params = OmniDiffusionSamplingParams(
width=args.width,
prompt_dict = build_image_to_image_prompt(
model_class_name=model_class_name,
prompt=args.prompt,
negative_prompt=args.negative_prompt,
input_image=input_image,
height=args.height,
width=args.width,
)
extra_args_from_cli = dict(args.extra_args or {})
if args.negative_prompt is not None:
extra_args_from_cli.setdefault("negative_prompt", args.negative_prompt)
diffusion_params = OmniDiffusionSamplingParams(
generator=generator,
true_cfg_scale=args.cfg_scale,
guidance_scale=args.guidance_scale,
@@ -507,17 +533,44 @@ def main():
num_inference_steps=args.num_inference_steps,
num_outputs_per_prompt=args.num_outputs_per_prompt,
layers=args.layers,
resolution=args.resolution,
height=args.height,
width=args.width,
)
if args.resolution:
sampling_params.resolution = args.resolution
outputs = omni.generate(
{
"prompt": args.prompt,
"negative_prompt": args.negative_prompt,
"multi_modal_data": {"image": input_image},
},
sampling_params,
if declared_extra_body_params:
apply_declared_extra_args(
diffusion_params,
declared_extra_body_params,
extra_args_from_cli,
)
else:
diffusion_params.extra_args.update({k: v for k, v in extra_args_from_cli.items() if v is not None})
# Build per-stage sampling params for multi-stage models
init_non_diffusion = should_init_extra_args_for_non_diffusion_stages(
model_class_name,
)
defaults = list(omni.default_sampling_params_list or [])
sampling_params_list = [clone_sampling_params(p) for p in defaults]
if not sampling_params_list:
sampling_params_list = [diffusion_params]
diffusion_replaced = False
for idx, params in enumerate(sampling_params_list):
if isinstance(params, OmniDiffusionSamplingParams):
merged_extra = dict(getattr(params, "extra_args", {}) or {})
merged_extra.update(diffusion_params.extra_args)
diffusion_params.extra_args = merged_extra
sampling_params_list[idx] = diffusion_params
diffusion_replaced = True
elif init_non_diffusion and hasattr(params, "extra_args"):
if params.extra_args is None:
params.extra_args = {}
if not diffusion_replaced and len(sampling_params_list) == 1:
sampling_params_list = [diffusion_params]
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
generation_end = time.perf_counter()
generation_time = generation_end - generation_start
@@ -544,17 +597,16 @@ def main():
if not outputs:
raise ValueError("No output generated from omni.generate()")
# Extract images from OmniRequestOutput
# omni.generate() returns list[OmniRequestOutput], extract images from request_output.images
first_output = outputs[0]
if not hasattr(first_output, "request_output") or not first_output.request_output:
raise ValueError("No request_output found in OmniRequestOutput")
images = None
for output in outputs:
images = getattr(output, "images", None)
if images:
break
req_out = getattr(output, "request_output", None)
images = getattr(req_out, "images", None) if req_out is not None else None
if images:
break
req_out = first_output.request_output
if not isinstance(req_out, OmniRequestOutput) or not hasattr(req_out, "images"):
raise ValueError("Invalid request_output structure or missing 'images' key")
images = req_out.images
if not images:
raise ValueError("No images found in request_output")
@@ -10,10 +10,18 @@ from typing import Any
import torch
from vllm_omni.diffusion.data import logger
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.lora.request import LoRARequest
from vllm_omni.lora.utils import stable_lora_int_id
from vllm_omni.model_extras import (
build_text_to_image_prompt,
get_extra_body_params,
get_model_class_name,
should_init_extra_args_for_non_diffusion_stages,
)
from vllm_omni.platforms import current_omni_platform
@@ -259,6 +267,19 @@ def parse_args() -> argparse.Namespace:
action="store_true",
help="[NextStep-1.1 only] Apply layer normalization to sampled tokens.",
)
parser.add_argument(
"--extra-body",
type=parse_profiler_config,
default=None,
help=(
"Model-specific generation params as a JSON object, e.g. "
'\'{"timestep_shift": 3.0, "cfg_text_scale": 4.0, "cfg_interval": [0.4, 1.0]}\'. '
"Each key is filtered against the model's declared extra_body_params "
"(see vllm_omni/model_extras), so unknown keys for the chosen model are "
"silently dropped. Values here take precedence over the equivalent "
"model-specific flags above."
),
)
parser.add_argument(
"--enable-diffusion-pipeline-profiler",
action="store_true",
@@ -405,6 +426,8 @@ def main():
# NextStep-1.1 requires explicit pipeline class
omni_kwargs["model_class_name"] = "NextStep11Pipeline"
omni = Omni(**omni_kwargs)
model_class_name = get_model_class_name(omni)
declared_extra_body_params = get_extra_body_params(model_class_name)
if profiler_enabled:
print("[Profiler] Starting profiling...")
@@ -445,34 +468,79 @@ def main():
)
generation_start = time.perf_counter()
extra_args = {
prompt_dict = build_text_to_image_prompt(
model_class_name=model_class_name,
prompt=args.prompt,
negative_prompt=args.negative_prompt,
height=args.height,
width=args.width,
)
diffusion_params = OmniDiffusionSamplingParams(
height=args.height,
width=args.width,
seed=args.seed,
generator=generator,
true_cfg_scale=args.cfg_scale,
guidance_scale=args.guidance_scale,
guidance_scale_2=args.guidance_scale_2,
num_inference_steps=args.num_inference_steps,
num_outputs_per_prompt=args.num_images_per_prompt,
)
# Base layer: backward-compatible model-specific flags. New model params should
# instead be declared in vllm_omni/model_extras and passed via --extra-body, so
# this dict does not need to grow per-model.
user_extra = {
"cfg_scale": args.cfg_scale,
"cfg_text_scale": args.cfg_scale,
"negative_prompt": args.negative_prompt,
"timestep_shift": args.timesteps_shift,
"timesteps_shift": args.timesteps_shift,
"cfg_schedule": args.cfg_schedule,
"use_norm": args.use_norm,
"use_system_prompt": args.use_system_prompt,
"system_prompt": args.system_prompt,
}
if lora_request:
extra_args["lora_request"] = lora_request
extra_args["lora_scale"] = args.lora_scale
# Override layer: generic JSON passthrough wins over the flags above. Keys are
# still filtered against the model's declared extra_body_params downstream.
if args.extra_body:
user_extra.update(args.extra_body)
if declared_extra_body_params:
apply_declared_extra_args(diffusion_params, declared_extra_body_params, user_extra)
else:
diffusion_params.extra_args.update({k: v for k, v in user_extra.items() if v is not None})
outputs = omni.generate(
{
"prompt": args.prompt,
"negative_prompt": args.negative_prompt,
},
OmniDiffusionSamplingParams(
height=args.height,
width=args.width,
generator=generator,
true_cfg_scale=args.cfg_scale,
guidance_scale=args.guidance_scale,
guidance_scale_2=args.guidance_scale_2,
num_inference_steps=args.num_inference_steps,
num_outputs_per_prompt=args.num_images_per_prompt,
extra_args=extra_args,
),
if lora_request:
diffusion_params.extra_args["lora_request"] = lora_request
diffusion_params.extra_args["lora_scale"] = args.lora_scale
# Build per-stage sampling params for multi-stage models (e.g. BAGEL),
# or wrap single diffusion params for single-stage models.
init_non_diffusion = should_init_extra_args_for_non_diffusion_stages(
model_class_name,
)
defaults = list(omni.default_sampling_params_list or [])
sampling_params_list = [clone_sampling_params(p) for p in defaults]
if not sampling_params_list:
sampling_params_list = [diffusion_params]
diffusion_replaced = False
for idx, params in enumerate(sampling_params_list):
if isinstance(params, OmniDiffusionSamplingParams):
sampling_params_list[idx] = diffusion_params
diffusion_replaced = True
elif init_non_diffusion and hasattr(params, "extra_args"):
if params.extra_args is None:
params.extra_args = {}
if args.seed is not None and hasattr(params, "seed"):
params.seed = args.seed
if not diffusion_replaced and len(sampling_params_list) == 1:
sampling_params_list = [diffusion_params]
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
generation_end = time.perf_counter()
generation_time = generation_end - generation_start
@@ -502,15 +570,16 @@ def main():
raise ValueError("No output generated from omni.generate()")
logger.info(f"Outputs: {outputs}")
first_output = outputs[0]
if not hasattr(first_output, "request_output") or not first_output.request_output:
raise ValueError("No request_output found in OmniRequestOutput")
images = None
for output in outputs:
images = getattr(output, "images", None)
if images:
break
req_out = getattr(output, "request_output", None)
images = getattr(req_out, "images", None) if req_out is not None else None
if images:
break
req_out = first_output.request_output
if not hasattr(req_out, "images"):
raise ValueError("Invalid request_output structure or missing 'images'.")
images = req_out.images
if not images:
raise ValueError("No images found in request_output")
-404
View File
@@ -1,404 +0,0 @@
# BAGEL-7B-MoT
## Installation
Please refer to [README.md](../../../README.md)
## Architecture
BAGEL-7B-MoT is a Mixture-of-Transformers (MoT) model supporting both image generation and understanding. It offers two deployment topologies:
| Topology | Stages | Description |
| :------- | :----- | :---------- |
| **Two-stage** (default) | Stage 0 (Thinker, AR) + Stage 1 (DiT, Diffusion) | Thinker handles text/understanding via vLLM AR engine; DiT handles image generation. KV cache is transferred between stages. |
| **Single-stage** | Stage 0 (DiT, Diffusion) only | The DiT stage contains a full LLM, ViT, VAE, and tokenizer internally. All modalities are handled within a single diffusion process. |
Both topologies support all four modalities: `text2img`, `img2img`, `img2text`, `text2text`.
> **Note**: These examples work with the default configuration on an **NVIDIA A100 (80GB)**. We also tested on dual **NVIDIA RTX 5000 Ada (32GB each)**. For dual-GPU setups, modify the deploy YAML to distribute stages across devices.
## Launch the Server
### Two-Stage (Default)
The default pipeline is auto-detected from the model. No extra flags needed:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091
```
Or use the convenience script:
```bash
cd examples/online_serving/bagel
bash run_server.sh
# Initialize each stage in a discrete isolated process terminal
bash run_server_stage_cli.sh --stage 0
bash run_server_stage_cli.sh --stage 1
```
To use a custom deploy YAML, pass it via `--deploy-config`:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \
--deploy-config /path/to/deploy_config.yaml
```
See [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml) for the default two-stage deploy configuration.
### Single-Stage
The DiT stage contains a full LLM, ViT, VAE, and tokenizer, so it can handle all modalities (text2img, img2img, img2text, text2text, think) without a separate Thinker stage:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \
--deploy-config vllm_omni/deploy/bagel_single_stage.yaml
```
See [`bagel_single_stage.yaml`](../../../vllm_omni/deploy/bagel_single_stage.yaml) for configuration. The `pipeline: bagel_single_stage` field selects the single-stage topology from the pipeline registry.
### Tensor Parallelism (TP)
For larger models or multi-GPU environments, enable TP via CLI:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --tensor-parallel-size 2
```
Or set `tensor_parallel_size` per stage in a custom deploy YAML.
### VAE Patch Parallelism
[VAE Patch Parallelism](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism/vae_patch_parallel.html) distributes Bagel VAE **decode/encode** across multiple GPUs by splitting latent tiles. It lowers **per-GPU peak memory during VAE decode**, which helps high-resolution `text2img` / `img2img` when VAE becomes a bottleneck.
**Scope for Bagel:**
| Topology | VAE patch parallel |
| :------- | :----------------- |
| **Single-stage** (DiT only) | Supported on stage 0 (`BagelPipeline` + `DistributedAutoEncoder`) |
| **Two-stage** | Supported on **stage 1 (DiT)** only; stage 0 (Thinker) uses encoder-only VAE and is unrelated |
**Requirements:**
- `vae_patch_parallel_size > 1` and a distributed VAE (`DistributedAutoEncoder` on the DiT pipeline).
- The DiT process group must have at least `vae_patch_parallel_size` ranks. In practice this means the diffusion stage `world_size` must be ≥ 2 (commonly `tensor_parallel_size=2` on that stage).
- `vae_use_tiling` must be enabled. If you set `vae_patch_parallel_size > 1` and omit tiling, the registry auto-enables `vae_use_tiling` at startup.
VAE patch parallel **reuses the DiT process group** (`dit_group`); it does not create a separate VAE-only worker pool. It is not a substitute for single-GPU VAE tiling (`vae_pp=1`).
**Online serving (single-stage, 2 GPUs):**
```bash
CUDA_VISIBLE_DEVICES=0,1 vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \
--deploy-config vllm_omni/deploy/bagel_single_stage.yaml \
--tensor-parallel-size 2 \
--vae-patch-parallel-size 2 \
--vae-use-tiling
```
**Online serving (two-stage, VAE PP on DiT stage 1):** use a custom deploy YAML, for example:
```yaml
stages:
- stage_id: 0
devices: "0"
# Thinker (AR) — no VAE patch parallel here
- stage_id: 1
devices: "0,1"
vae_use_tiling: true
parallel_config:
tensor_parallel_size: 2
vae_patch_parallel_size: 2
```
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \
--deploy-config /path/to/bagel_vae_pp.yaml
```
**Verify it is active** (check server logs at startup):
```text
INFO ... vae_patch_parallel_size=2 requires vae_use_tiling; automatically enabling it.
```
| CLI flag | Default | Description |
| :------- | :------ | :---------- |
| `--vae-patch-parallel-size` | `1` | Number of DiT ranks used for VAE tile parallelism. Set to `2` or higher to enable. Should be ≤ DiT process group size (typically match `--tensor-parallel-size` on the diffusion stage). |
| `--vae-use-tiling` | off | Enable VAE spatial tiling. Required for VAE patch parallel (auto-enabled when `vae_patch_parallel_size > 1`). |
#### Hybrid Sharded Data Parallel (HSDP)
For larger Bagel deployments on multiple GPUs, you can enable HSDP (Hybrid Sharded Data Parallel) by modifying the stage configuration (for example, [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml)). HSDP shards transformer weights across GPUs to reduce per-GPU memory usage.
1. **Enable HSDP**: Set `use_hsdp: true`.
2. **Set shard size**: Set `hsdp_shard_size` to the number of GPUs used for sharding (for example, `4`).
3. **Set replicate size**: Usually keep `hsdp_replicate_size: 1` unless you want replicated HSDP groups.
4. **Set devices**: Specify the comma-separated GPU IDs used by the diffusion stage (for example, `"0,1,2,3"`).
Example configuration for HSDP across 4 GPUs:
```yaml
- stage_id: 1
devices: "0,1,2,3"
parallel_config:
use_hsdp: true
hsdp_shard_size: 4
hsdp_replicate_size: 1
```
### Multi-Node Deployment
Deploy each stage on a **separate node** for better resource utilization. Replace `<ORCHESTRATOR_IP>` with the actual IP address of your orchestrator node.
**1. Launch Stage 0 (Thinker / Orchestrator)** on the orchestrator node:
```bash
# API server port for client requests: 8000
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
--port 8000 \
--stage-id 0 \
--omni-master-address <ORCHESTRATOR_IP> \
--omni-master-port 8091
```
**2. Launch Stage 1 (DiT)** on the remote node in headless mode:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \
--stage-id 1 \
--headless \
--omni-master-address <ORCHESTRATOR_IP> \
--omni-master-port 8091
```
Or use the convenience script:
```bash
# Terminal 1: Stage 0
bash run_server_stage_cli.sh --stage 0
# Terminal 2: Stage 1
bash run_server_stage_cli.sh --stage 1
# With extra args
bash run_server_stage_cli.sh --stage 0 -- --tensor-parallel-size 2
bash run_server_stage_cli.sh --stage 1 -- --gpu-memory-utilization 0.9
```
**vllm serve arguments:**
| Argument | Description |
| :------- | :---------- |
| `--stage-id` | Which stage this process runs (0 = Thinker, 1 = DiT) |
| `--headless` | Run without the API server (worker-only mode) |
| `-oma` / `--omni-master-address` | Orchestrator master address |
| `-omp` / `--omni-master-port` | Orchestrator master port |
> [!IMPORTANT]
> **Startup Order**: Stage 0 (orchestrator) must be launched **before** Stage 1 (headless).
> Stage 0 will appear to hang on startup until Stage 1 (worker) connects — this is expected behavior.
### Inter-Stage Connectors
When deploying stages across nodes, configure the connector type in the deploy YAML:
- **SharedMemoryConnector** (default): Used for single-node deployments. No explicit configuration needed.
- **MooncakeTransferEngineConnector**: For multi-node setups with RDMA hardware. Defined in [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml) under `connectors.rdma_connector`.
To use Mooncake, create a custom deploy YAML that binds `output_connectors` / `input_connectors` on each stage to the `rdma_connector` defined in the `connectors` section.
## Send Requests
```bash
cd examples/online_serving/bagel
```
### Text to Image (text2img)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "A beautiful sunset over mountains" \
--modality text2img \
--output sunset.png \
--steps 50
```
**curl:**
```bash
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>A beautiful sunset over mountains<|im_end|>"}]}],
"modalities": ["image"],
"height": 512,
"width": 512,
"num_inference_steps": 50,
"seed": 42
}'
```
### Image to Image (img2img)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "Make the cat stand up" \
--modality img2img \
--image-url /path/to/input.jpg \
--output transformed.png
```
**curl:**
```bash
IMAGE_BASE64=$(base64 -w 0 cat.jpg)
cat <<EOF > payload.json
{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "<|im_start|>Make the cat stand up<|im_end|>"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
]
}],
"modalities": ["image"],
"height": 512,
"width": 512,
"num_inference_steps": 50,
"seed": 42
}
EOF
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d @payload.json
```
### Image to Text (img2text)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "Describe this image in detail" \
--modality img2text \
--image-url /path/to/image.jpg
```
**curl:**
```bash
IMAGE_BASE64=$(base64 -w 0 cat.jpg)
cat <<EOF > payload.json
{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "<|im_start|>user\n<|image_pad|>\nDescribe this image in detail<|im_end|>\n<|im_start|>assistant\n"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMAGE_BASE64}"}}
]
}],
"modalities": ["text"]
}
EOF
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d @payload.json
```
### Text to Text (text2text)
**Python client:**
```bash
python openai_chat_client.py \
--prompt "What is the capital of France?" \
--modality text2text
```
**curl:**
```bash
curl http://localhost:8091/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": [{"type": "text", "text": "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"}]}],
"modalities": ["text"]
}'
```
### Python Client Arguments
| Argument | Default | Description |
| :------- | :------ | :---------- |
| `--prompt` / `-p` | `A cute cat` | Text prompt |
| `--output` / `-o` | `bagel_output.png` | Output file path |
| `--server` / `-s` | `http://localhost:8091` | Server URL |
| `--image-url` / `-i` | `None` | Input image URL or local path (img2img/img2text) |
| `--modality` / `-m` | `text2img` | `text2img`, `img2img`, `img2text`, `text2text` |
| `--height` | `512` | Image height in pixels |
| `--width` | `512` | Image width in pixels |
| `--steps` | `25` | Number of inference steps |
| `--seed` | `42` | Random seed |
| `--negative` | `None` | Negative prompt for CFG |
Example with custom parameters:
```bash
python openai_chat_client.py \
--prompt "A futuristic city" \
--modality text2img \
--height 768 \
--width 768 \
--steps 50 \
--seed 42 \
--negative "blurry, low quality"
```
## Configuration Reference
### Deploy YAML Files
| File | Description |
| :--- | :---------- |
| [`bagel.yaml`](../../../vllm_omni/deploy/bagel.yaml) | Two-stage default (Thinker + DiT on GPU 0) |
| [`bagel_single_stage.yaml`](../../../vllm_omni/deploy/bagel_single_stage.yaml) | Single-stage (DiT only) |
### Key Deploy YAML Fields
| Field | Scope | Description |
| :---- | :---- | :---------- |
| `pipeline` | top-level | Override auto-detected pipeline (e.g. `bagel_single_stage`) |
| `stages[].stage_id` | per-stage | Stage identifier (0, 1, ...) |
| `stages[].devices` | per-stage | GPU device IDs (e.g. `"0"`, `"0,1"`) |
| `stages[].max_num_seqs` | per-stage | Maximum concurrent sequences |
| `stages[].gpu_memory_utilization` | per-stage | Fraction of GPU memory to use |
| `stages[].enforce_eager` | per-stage | Disable CUDA graphs |
| `stages[].tensor_parallel_size` | per-stage | TP degree for this stage |
| `stages[].parallel_config.vae_patch_parallel_size` | per-stage (DiT) | VAE tile parallelism degree (DiT stage only) |
| `stages[].vae_use_tiling` | per-stage (DiT) | Enable VAE tiling (required for VAE patch parallel) |
| `connectors` | top-level | Define available connector instances (SHM, Mooncake) |
| `platforms` | top-level | Platform-specific overrides (e.g. `xpu`) |
## FAQ
- If you encounter OOM errors, try decreasing `max_model_len` or `gpu_memory_utilization` in the deploy YAML.
**Two-stage VRAM usage:**
| Stage | VRAM |
| :---- | :--- |
| Stage 0 (Thinker) | **15.04 GiB + KV Cache** |
| Stage 1 (DiT) | **26.50 GiB** |
| Total | **~42 GiB + KV Cache** |
**Single-stage VRAM usage:** The DiT loads the full model (~42 GiB) in one process.
@@ -1,184 +0,0 @@
#!/usr/bin/env python3
"""
Bagel OpenAI-compatible chat client for image generation and multimodal tasks.
Usage:
python openai_chat_client.py --prompt "A cute cat" --output output.png
python openai_chat_client.py --prompt "Describe this image" --image-url https://example.com/image.png
"""
import argparse
import base64
from pathlib import Path
import requests
def generate_image(
prompt: str,
server_url: str = "http://localhost:8091",
image_url: str | None = None,
height: int | None = None,
width: int | None = None,
steps: int | None = None,
seed: int | None = None,
negative_prompt: str | None = None,
modality: str = "text2img", # "text2img" (default), "img2img", "img2text", "text2text"
) -> bytes | str | None:
"""Generate an image or text using the chat completions API.
Args:
prompt: Text description or prompt
server_url: Server URL
image_url: URL or path to input image (for img2img/img2text)
height: Image height in pixels
width: Image width in pixels
steps: Number of inference steps
seed: Random seed
negative_prompt: Negative prompt
modality: Task modality hint
Returns:
Image bytes (for image outputs) or Text string (for text outputs) or None if failed
"""
# Construct Message Content
content = [{"type": "text", "text": f"<|im_start|>{prompt}<|im_end|>"}]
if image_url:
# Check if local file
if Path(image_url).exists():
with open(image_url, "rb") as f:
b64_data = base64.b64encode(f.read()).decode("utf-8")
final_image_url = f"data:image/jpeg;base64,{b64_data}"
else:
final_image_url = image_url
content.append({"type": "image_url", "image_url": {"url": final_image_url}})
messages = [{"role": "user", "content": content}]
# Build request payload with all parameters at top level
# Note: vLLM ignores "extra_body", so we put parameters directly in the payload
payload = {"messages": messages}
# Set output modalities at top level
if modality == "text2img" or modality == "img2img":
payload["modalities"] = ["image"]
elif modality == "img2text" or modality == "text2text":
payload["modalities"] = ["text"]
# Add generation parameters directly to payload
if height is not None:
payload["height"] = height
if width is not None:
payload["width"] = width
if steps is not None:
payload["num_inference_steps"] = steps
if seed is not None:
payload["seed"] = seed
if negative_prompt:
payload["negative_prompt"] = negative_prompt
# Send request
try:
print(f"Sending request to {server_url} with modality {modality}...")
response = requests.post(
f"{server_url}/v1/chat/completions",
headers={"Content-Type": "application/json"},
json=payload,
timeout=300,
)
response.raise_for_status()
data = response.json()
# Extract content - check ALL choices since server may return multiple
# (e.g., text in choices[0], image in choices[1])
choices = data.get("choices", [])
# First pass: look for image output in any choice
for choice in choices:
choice_content = choice.get("message", {}).get("content")
# Handle Image Output
if isinstance(choice_content, list) and len(choice_content) > 0:
first_item = choice_content[0]
if isinstance(first_item, dict) and "image_url" in first_item:
img_url_str = first_item["image_url"].get("url", "")
if img_url_str.startswith("data:image"):
_, b64_data = img_url_str.split(",", 1)
return base64.b64decode(b64_data)
# Second pass: look for text output if no image found
for choice in choices:
choice_content = choice.get("message", {}).get("content")
if isinstance(choice_content, str) and choice_content:
return choice_content
print(f"Unexpected response format: {choices}")
return None
except Exception as e:
print(f"Error: {e}")
return None
def main():
parser = argparse.ArgumentParser(description="Bagel multimodal chat client")
parser.add_argument("--prompt", "-p", default="A cute cat", help="Text prompt")
parser.add_argument("--output", "-o", default="bagel_output.png", help="Output file (for image results)")
parser.add_argument("--server", "-s", default="http://localhost:8091", help="Server URL")
# Modality Control
parser.add_argument("--image-url", "-i", type=str, help="Input image URL or local path")
parser.add_argument(
"--modality",
"-m",
default="text2img",
choices=["text2img", "img2img", "img2text", "text2text"],
help="Task modality",
)
# Generation Params
parser.add_argument("--height", type=int, default=512, help="Image height")
parser.add_argument("--width", type=int, default=512, help="Image width")
parser.add_argument("--steps", type=int, default=25, help="Inference steps")
parser.add_argument("--seed", type=int, default=42, help="Random seed")
parser.add_argument("--negative", help="Negative prompt")
args = parser.parse_args()
print(f"Mode: {args.modality}")
if args.image_url:
print(f"Input Image: {args.image_url}")
result = generate_image(
prompt=args.prompt,
server_url=args.server,
image_url=args.image_url,
height=args.height,
width=args.width,
steps=args.steps,
seed=args.seed,
negative_prompt=args.negative,
modality=args.modality,
)
if result:
if isinstance(result, bytes):
# It's an image
output_path = Path(args.output)
output_path.write_bytes(result)
print(f"Image saved to: {output_path}")
print(f"Size: {len(result) / 1024:.1f} KB")
elif isinstance(result, str):
# It's text
print("Response:")
print(result)
else:
print("Failed to generate response")
exit(1)
if __name__ == "__main__":
main()
@@ -1,12 +0,0 @@
#!/bin/bash
# Bagel online serving startup script
MODEL="${MODEL:-ByteDance-Seed/BAGEL-7B-MoT}"
PORT="${PORT:-8091}"
echo "Starting Bagel server..."
echo "Model: $MODEL"
echo "Port: $PORT"
vllm serve "$MODEL" --omni \
--port "$PORT"
@@ -1,164 +0,0 @@
#!/bin/bash
# Bagel multi-stage online serving startup script.
#
# Usage:
# ./run_server_stage_cli.sh --stage 0
# ./run_server_stage_cli.sh --stage 1
# ./run_server_stage_cli.sh --stage 0 -- --tensor-parallel-size 2
# ./run_server_stage_cli.sh --stage 1 -- --gpu-memory-utilization 0.9
#
# By default, `--stage all` keeps the old behavior and launches both stages in
# one session. Use `--stage 0` / `--stage 1` to launch each stage separately in
# different terminal sessions, with stage-specific extra CLI arguments passed
# after `--`.
set -euo pipefail
MODEL="${MODEL:-ByteDance-Seed/BAGEL-7B-MoT}"
PORT="${PORT:-8091}"
MASTER_ADDRESS="${MASTER_ADDRESS:-127.0.0.1}"
MASTER_PORT="${MASTER_PORT:-8092}"
STAGE="all"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_CONFIG="${DEPLOY_CONFIG:-$SCRIPT_DIR/../../../vllm_omni/deploy/bagel.yaml}"
EXTRA_ARGS=()
usage() {
cat <<EOF
Usage: $0 [OPTIONS] [-- EXTRA_VLLM_ARGS...]
Options:
--stage {0|1|all} Stage to launch (default: all)
--model MODEL Model name/path (default: $MODEL)
--port PORT API port for stage 0 (default: $PORT)
--master-address ADDRESS Master/orchestrator address (default: $MASTER_ADDRESS)
--master-port PORT Master/orchestrator port (default: $MASTER_PORT)
--deploy-config PATH Deploy config YAML path (default: $DEPLOY_CONFIG)
--help Show this help message
Examples:
$0 --stage 0
$0 --stage 1
$0 --stage 0 -- --tensor-parallel-size 2
$0 --stage 1 -- --gpu-memory-utilization 0.9
Notes:
- Use different terminal sessions to launch stage 0 and stage 1 separately.
- Extra args after '--' are forwarded only to the selected stage.
- When using '--stage all', the extra args are forwarded to both stages.
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--stage)
STAGE="$2"
shift 2
;;
--model)
MODEL="$2"
shift 2
;;
--port)
PORT="$2"
shift 2
;;
--master-address)
MASTER_ADDRESS="$2"
shift 2
;;
--master-port)
MASTER_PORT="$2"
shift 2
;;
--deploy-config)
DEPLOY_CONFIG="$2"
shift 2
;;
--help|-h)
usage
exit 0
;;
--)
shift
EXTRA_ARGS=("$@")
break
;;
*)
echo "Unknown option: $1" >&2
usage
exit 1
;;
esac
done
if [[ "$STAGE" != "0" && "$STAGE" != "1" && "$STAGE" != "all" ]]; then
echo "Invalid --stage value: $STAGE" >&2
usage
exit 1
fi
print_config() {
echo "Model: $MODEL"
echo "API Port: $PORT"
echo "Master Address: $MASTER_ADDRESS"
echo "Master Port: $MASTER_PORT"
echo "Deploy Config: $DEPLOY_CONFIG"
echo "Selected Stage: $STAGE"
if [[ ${#EXTRA_ARGS[@]} -gt 0 ]]; then
echo "Extra Args: ${EXTRA_ARGS[*]}"
fi
}
run_stage_0() {
echo "Starting Stage 0 (Thinker) as master..."
vllm serve "$MODEL" --omni \
--port "$PORT" \
--deploy-config "$DEPLOY_CONFIG" \
--stage-id 0 \
--omni-master-address "$MASTER_ADDRESS" \
--omni-master-port "$MASTER_PORT" \
"${EXTRA_ARGS[@]}"
}
run_stage_1() {
echo "Starting Stage 1 (DiT) in headless mode..."
vllm serve "$MODEL" --omni \
--deploy-config "$DEPLOY_CONFIG" \
--stage-id 1 \
--headless \
--omni-master-address "$MASTER_ADDRESS" \
--omni-master-port "$MASTER_PORT" \
"${EXTRA_ARGS[@]}"
}
echo "Starting Bagel multi-stage server..."
print_config
case "$STAGE" in
0)
run_stage_0
;;
1)
run_stage_1
;;
all)
echo "Launching both stages in one session (legacy mode)..."
echo "Starting Stage 0 (Thinker) in background first..."
run_stage_0 &
STAGE_0_PID=$!
cleanup() {
if [[ -n "${STAGE_0_PID:-}" ]]; then
kill "$STAGE_0_PID" 2>/dev/null || true
wait "$STAGE_0_PID" 2>/dev/null || true
fi
}
trap cleanup EXIT INT TERM
echo "Waiting briefly for Stage 0 to initialize..."
sleep 2
run_stage_1
;;
esac
@@ -1,6 +1,8 @@
# Image-To-Image
This example demonstrates how to deploy Qwen-Image-Edit model for online image editing service using vLLM-Omni.
This example demonstrates how to deploy image-to-image models for online image editing service using vLLM-Omni.
Supported models include Qwen-Image-Edit, BAGEL, and other image-to-image pipelines.
For **multi-image** input editing, use **Qwen-Image-Edit-2509** (QwenImageEditPlusPipeline) and send multiple images in the user message content.
@@ -21,6 +23,12 @@ vllm serve Qwen/Qwen-Image-Edit --omni --port 8092
vllm serve Qwen/Qwen-Image-Edit-2509 --omni --port 8092
```
### BAGEL
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091
```
### Start with Parameters
@@ -119,6 +127,16 @@ python openai_chat_client.py --input input.png --prompt "Convert to oil painting
python openai_chat_client.py --input input1.png input2.png --prompt "Combine these images into a single scene" --output output.png
```
Pass model-specific parameters through `--extra-body` (e.g. for BAGEL):
```bash
python openai_chat_client.py \
--input input.png \
--prompt "Make the scene look like a watercolor painting" \
--server http://localhost:8091 \
--extra-body '{"cfg_text_scale": 4.0, "cfg_img_scale": 1.5}'
```
### Method 4: Using Gradio Demo
```bash
@@ -318,6 +336,9 @@ count, use `size` and `n` rather than `height`, `width`, or
| `layers` | int | 4 | Number of layers (Qwen-Image-Layered) |
| `resolution` | int | 640 | Resolution, 640 or 1024 (Qwen-Image-Layered) |
Models like BAGEL accept additional parameters via `extra_body` (e.g.
`cfg_text_scale`, `cfg_img_scale`). See the [BAGEL recipe](../../../recipes/Bagel/BAGEL-7B-MoT.md) for the full list.
## Response Format
```json
@@ -38,6 +38,7 @@ def edit_image(
guidance_scale: float | None = None,
seed: int | None = None,
negative_prompt: str | None = None,
extra_body: dict | None = None,
) -> bytes | None:
"""Edit an image using the chat completions API.
@@ -51,6 +52,7 @@ def edit_image(
guidance_scale: CFG guidance scale
seed: Random seed
negative_prompt: Negative prompt
extra_body: Additional model-specific params (e.g. cfg_text_scale for BAGEL)
Returns:
Edited image bytes or None if failed
@@ -75,24 +77,26 @@ def edit_image(
]
# Build extra_body with generation parameters
extra_body = {}
merged_extra_body: dict[str, object] = {}
if height is not None:
extra_body["height"] = height
merged_extra_body["height"] = height
if width is not None:
extra_body["width"] = width
merged_extra_body["width"] = width
if steps is not None:
extra_body["num_inference_steps"] = steps
merged_extra_body["num_inference_steps"] = steps
if guidance_scale is not None:
extra_body["guidance_scale"] = guidance_scale
merged_extra_body["guidance_scale"] = guidance_scale
if seed is not None:
extra_body["seed"] = seed
merged_extra_body["seed"] = seed
if negative_prompt:
extra_body["negative_prompt"] = negative_prompt
merged_extra_body["negative_prompt"] = negative_prompt
if extra_body:
merged_extra_body.update(extra_body)
# Build request payload
payload = {"messages": messages}
if extra_body:
payload["extra_body"] = extra_body
payload: dict[str, object] = {"messages": messages}
if merged_extra_body:
payload["extra_body"] = merged_extra_body
# Send request
try:
@@ -121,6 +125,19 @@ def edit_image(
return None
def parse_extra_body(value: str) -> dict:
"""Parse a JSON string into a dict for --extra-body."""
import json
try:
obj = json.loads(value)
except json.JSONDecodeError as e:
raise argparse.ArgumentTypeError(f"--extra-body must be valid JSON: {e}") from e
if not isinstance(obj, dict):
raise argparse.ArgumentTypeError("--extra-body must be a JSON object")
return obj
def main():
parser = argparse.ArgumentParser(description="Qwen-Image-Edit chat client")
parser.add_argument("--input", "-i", required=True, nargs="+", help="Input image path(s)")
@@ -133,6 +150,12 @@ def main():
parser.add_argument("--guidance", type=float, default=7.5, help="Guidance scale")
parser.add_argument("--seed", type=int, default=0, help="Random seed")
parser.add_argument("--negative", help="Negative prompt")
parser.add_argument(
"--extra-body",
type=parse_extra_body,
default=None,
help="JSON object merged into request extra_body, e.g. '{\"cfg_text_scale\": 4.0}'.",
)
args = parser.parse_args()
@@ -152,6 +175,7 @@ def main():
guidance_scale=args.guidance,
seed=args.seed,
negative_prompt=args.negative,
extra_body=args.extra_body,
)
if image_bytes:
@@ -48,7 +48,7 @@ def generate(
forwarded as top-level fields in the request payload. The serving layer
maps standard keys (height, width, seed, num_inference_steps) to sampling
params and forwards model-specific keys (think, cfg_norm, img_cfg_scale,
…) to ``extra_args`` based on the pipeline's ``EXTRA_BODY_PARAMS``.
…) to ``extra_args`` based on the model extra registry.
Returns:
Image bytes (for image outputs) or text string (for text outputs).
@@ -131,7 +131,7 @@ def main():
parser.add_argument("--width", type=int, default=2048, help="Image width")
parser.add_argument("--num-steps", type=int, default=50, help="Inference steps")
parser.add_argument("--seed", type=int, default=42, help="Random seed")
# Model-specific parameters (forwarded via EXTRA_BODY_PARAMS)
# Model-specific parameters forwarded via the model extra registry.
parser.add_argument("--cfg-scale", type=float, default=4.0, help="CFG scale")
parser.add_argument("--img-cfg-scale", type=float, help="Image CFG scale (img2img)")
parser.add_argument("--cfg-norm", type=str, help="CFG normalization mode")