anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Config] Add HunyuanImage3 deploy configs (#3172)

Browse Source 
Signed-off-by: KexiongYu <yukexiong1@huawei.com>
Signed-off-by: Y. Fisher <yukexiong1@huawei.com>
This commit is contained in:
Y. Fisher authored and GitHub committed 2026-05-11 22:49:19 +08:00
1 parent c9a8556c24
commit a33e2eb588
26 files changed
+672 -742
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/hunyuan_image3/README.md
+109 -125
View File
@@ -1,172 +1,156 @@
# HunyuanImage-3.0-Instruct
## Set up
This example runs HunyuanImage-3.0-Instruct offline with the unified deploy
YAMLs under `vllm_omni/deploy/`.
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
## Deploy Configs
## Run examples
| File | Topology | Default use |
| :--- | :--- | :--- |
| `vllm_omni/deploy/hunyuan_image3.yaml` | AR + DiT | Default for `text2img` and `img2img`. |
| `vllm_omni/deploy/hunyuan_image3_ar.yaml` | AR only | Default for `img2text` and `text2text`. |
| `vllm_omni/deploy/hunyuan_image3_dit.yaml` | DiT only | Standalone diffusion stage. Pass it explicitly with `--deploy-config`. |
**Note**: These examples work with the default configuration on **8x NVIDIA L40S (48GB)**. For different GPU setups, modify the stage configuration to adjust device allocation and memory utilization.
The example chooses a deploy config automatically when `--deploy-config` and
`--stage-configs-path` are both omitted:
Get into the hunyuan_image3 folder:
| `--modality` | `mode` passed to Omni | Default deploy |
| :--- | :--- | :--- |
| `text2img` | `text-to-image` | `hunyuan_image3.yaml` |
| `img2img` | `image-editing` | `hunyuan_image3.yaml` |
| `img2text` | `image-to-text` | `hunyuan_image3_ar.yaml` |
| `text2text` | `text-to-text` | `hunyuan_image3_ar.yaml` |
`--modality` is an offline example convenience flag. It maps to the internal
`mode` argument passed to `Omni(...)` by this script. HunyuanImage3 uses
separate deploy YAMLs for AR + DiT, AR-only, and DiT-only topologies, so the
stage topology is selected by the deploy file rather than by YAML mode
overrides.
Online serving does not expose a `--modality` flag or accept `mode` as an API
request field. Choose the deploy topology when starting the server with
`--deploy-config`, then use the OpenAI-compatible endpoint and request shape for
the scenario. The `modalities` request field is used by the chat completions
path; the image endpoints infer the image task from the endpoint and payload.
| Online scenario | Server deploy | Request |
| :--- | :--- | :--- |
| Text to image | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/generations`, or `POST /v1/chat/completions` with `"modalities": ["image"]`. |
| Image editing | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/edits`. |
| Image/text to text | `--deploy-config vllm_omni/deploy/hunyuan_image3_ar.yaml` | `POST /v1/chat/completions` for text output, for example with `"modalities": ["text"]`. |
| DiT-only image generation | `--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml` | `POST /v1/images/generations`. |
## Run Examples
Text to image, using the default AR + DiT deploy:
```bash
cd examples/offline_inference/hunyuan_image3
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--prompts "A cute cat sitting on a windowsill watching the sunset"
```
### Modality Control
HunyuanImage-3.0-Instruct supports multiple modality modes. You can control the mode using the `--modality` argument:
#### Text to Image (text2img)
- **Pipeline**: Text → AR (CoT + latent tokens) → DiT (denoise) → VAE Decode → Image
- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT)
- **KV Transfer**: AR sends KV cache to DiT for conditioned generation
- **Default Config**: `hunyuan_image3_t2i.yaml`
Image editing, using the default AR + DiT deploy:
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--prompts "A cute cat sitting on a windowsill watching the sunset"
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality img2img \
--image-path /path/to/image.png \
--prompts "Make the petals neon pink"
```
**With VAE tiling (required on A100 GPUs):**
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--prompts "A cute cat sitting on a windowsill watching the sunset" \
--vae-use-tiling
```
#### Image to Image (img2img)
- **Pipeline**: Image + Text → AR (CoT + recaption + latent) → DiT → Edited Image
- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT)
- **KV Transfer**: AR sends KV cache to DiT
- **Default Config**: `hunyuan_image3_it2i.yaml`
Image to text, using the AR-only deploy:
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality img2img \
--image-path /path/to/image.png \
--prompts "Make the petals neon pink"
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality img2text \
--image-path /path/to/image.jpg \
--prompts "Describe the content of the picture."
```
#### Image to Text (img2text)
- **Pipeline**: Image + Question → AR → Text description
- **Stages Used**: Stage 0 (AR) only
- **Default Config**: `hunyuan_image3_i2t.yaml`
Text to text, using the AR-only deploy:
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality img2text \
--image-path /path/to/image.jpg \
--prompts "Describe the content of the picture."
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2text \
--prompts "What is the capital of France?"
```
#### Text to Text (text2text)
- **Pipeline**: Text → AR → Text
- **Stages Used**: Stage 0 (AR) only
- **Default Config**: `hunyuan_image3_t2t.yaml`
Standalone DiT, using the DiT-only deploy explicitly:
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality text2text \
--prompts "What is the capital of France?"
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \
--prompts "A cinematic portrait of an astronaut in a greenhouse"
```
### Inference Steps & Guidance
Control generation quality for image modalities:
Override the default full AR + DiT deploy explicitly:
```bash
python end2end.py --modality text2img \
--steps 50 \
--guidance-scale 5.0 \
--height 1024 --width 1024 \
--prompts "A photo-realistic sunset over the ocean"
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--deploy-config vllm_omni/deploy/hunyuan_image3.yaml \
--prompts "A cute cat"
```
### Key Arguments
## Key Arguments
#### 📌 Command Line Arguments (end2end.py)
| Argument | Description |
| :--- | :--- |
| `--deploy-config` | Preferred config path for unified deploy YAMLs. |
| `--stage-configs-path` | Legacy stage config path, kept only for compatibility. Prefer `--deploy-config`. |
| `--modality` | Offline-only convenience flag. One of `text2img`, `img2img`, `img2text`, `text2text`. It selects prompt formatting, internal `mode`, and default deploy config for this script. Online serving uses `--deploy-config` plus the endpoint and, for chat completions, request `modalities` instead. |
| `--steps` | Number of diffusion inference steps for image generation. |
| `--guidance-scale` | Classifier-free guidance scale for image generation. |
| `--height`, `--width` | Output image size for `text2img`. |
| `--bot-task` | Prompt behavior. `auto` selects the default from `--modality`; `think` adds `<think>`; `recaption` adds `<recaption>`; `vanilla` uses the text-to-image pretrain template. |
| `--sys-type` | Override the system prompt type, for example `en_unified` or `en_vanilla`. |
| `--vae-use-tiling` | Enable VAE tiling for memory reduction. |
| Argument | Type | Default | Description |
| :--------------------- | :----- | :----------------------------------- | :----------------------------------------------------------- |
| `--model` | string | `tencent/HunyuanImage-3.0-Instruct` | Model path or name |
| `--modality` | choice | `text2img` | Modality: `text2img`, `img2img`, `img2text`, `text2text` |
| `--prompts` | list | `None` | Input text prompts |
| `--image-path` | string | `None` | Input image path (for `img2img`/`img2text`) |
| `--output` | string | `.` | Output directory for saved images |
| `--steps` | int | `50` | Number of inference steps |
| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale |
| `--seed` | int | `42` | Random seed |
| `--height` | int | `1024` | Output image height |
| `--width` | int | `1024` | Output image width |
| `--bot-task` | string | auto | Override prompt task (e.g. `it2i_think`, `t2i_recaption`) |
| `--sys-type` | string | auto | Override system prompt type (e.g. `en_unified`, `en_vanilla`) |
| `--stage-configs-path` | string | auto | Custom stage config YAML path |
| `--enforce-eager` | flag | `False` | Disable torch.compile |
| `--init-timeout` | int | `300` | Initialization timeout (seconds) |
| `--vae-use-tiling` | flag | `False` | Enable VAE tiling for memory optimization (required to avoid OOM on A100) |
## Notes
------
#### ⚙️ Stage Configurations
| Config YAML | Modality | Stages | GPUs | Description |
| :---------------------------------- | :-------- | :----- | :----- | :------------------------------------ |
| `hunyuan_image3_t2i.yaml` | text2img | 2 | 8 | T2I with AR→DiT, 4 GPU each |
| `hunyuan_image3_it2i.yaml` | img2img | 2 | 8 | IT2I with AR→DiT, 4 GPU each |
| `hunyuan_image3_i2t.yaml` | img2text | 1 | 4 | I2T (AR only) |
| `hunyuan_image3_t2t.yaml` | text2text | 1 | 4 | T2T (AR only) |
| `hunyuan_image3_t2i_2gpu.yaml` | text2img | 2 | 2 | T2I for 2-GPU setups |
| `hunyuan_image3_moe.yaml` | text2img | 2 | 8 | T2I with MoE AR→DiT KV reuse |
| `hunyuan_image3_moe_dit_2gpu_fp8.yaml` | text2img | 2 | 2 | T2I with FP8 quantization |
------
## Using MoE Config
The `hunyuan_image3_moe.yaml` config enables AR→DiT KV cache reuse with 8 GPUs (4 for AR + 4 for DiT).
```bash
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--stage-configs-path hunyuan_image3_moe.yaml \
--prompts "A cute cat"
```
------
- `hunyuan_image3_ar.yaml` is a 4-card AR-only text/comprehension deploy. It sets `engine_output_type: text`, `final_output_type: text`, and text sampling defaults.
- `hunyuan_image3_dit.yaml` is a single-stage DiT deploy with `stage_id: 0`; it does not require stage 1 or a running AR stage.
- The old HunyuanImage3 YAMLs under `model_executor/stage_configs/` and `platforms/*/stage_configs/` have been folded into the deploy YAMLs.
- This PR does not keep the HunyuanImage3 AR-to-DiT KV reuse wiring. The deploy YAMLs describe the topology and platform settings only.
## Prompt Format
HunyuanImage-3.0-Instruct uses an instruct chat template:
```
<|startoftext|>{system_prompt}\n\nUser: {<img>?}{user_prompt}\n\nAssistant: {trigger_tag?}
```text
<|startoftext|>{system_prompt}
User: {<img>?}{user_prompt}
Assistant: {trigger_tag?}
```
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline)
- Trigger tags: `<think>` (CoT), `<recaption>` (recaptioning) — placed AFTER `Assistant: `
- System prompt: Auto-selected based on task
- `t2i_vanilla` is the only task that uses the bare pretrain template (no chat structure)
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline).
- Trigger tags: `<think>` for CoT and `<recaption>` for recaptioning, placed after `Assistant: `.
- System prompt: Auto-selected based on task.
- `t2i_vanilla` is the only task that uses the bare pretrain template without chat structure.
- The example composes the internal prompt task from `--modality` and `--bot-task`
before calling `prompt_utils`; for example, `img2text + think` becomes
`i2t_think` for prompt and stop-token lookup.
The shared `vllm_omni.diffusion.models.hunyuan_image3.prompt_utils.build_prompt_tokens()`
helper handles segment-by-segment tokenization (matches HF `apply_chat_template` byte-for-byte).
------
helper handles segment-by-segment tokenization and matches HF `apply_chat_template`.
## FAQ
- **OOM errors**: Decrease `gpu_memory_utilization` in the YAML stage config, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling` (required on A100 GPUs).
- **OOM errors**: Decrease `gpu_memory_utilization` in the deploy YAML, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling`.
- **Custom image sizes**: Use `--height` and `--width` flags (multiples of 16 recommended).
| Stage | VRAM (approx) |
| :---------------- | :------------------- |
| Stage 0 (AR) | ~15 GiB + KV Cache |
| Stage 1 (DiT) | ~30 GiB |
| Total (8-GPU) | ~45 GiB + KV Cache |
| Stage | VRAM (approx) |
| :--- | :--- |
| Stage 0 (AR) | ~15 GiB + KV Cache |
| Stage 1 (DiT) | ~30 GiB |
| Total (8-GPU) | ~45 GiB + KV Cache |
examples/offline_inference/hunyuan_image3/end2end.py
+79 -35
View File
@@ -15,42 +15,43 @@ Usage:
import argparse
import os
from pathlib import Path
from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import (
_TASK_PRESETS,
build_prompt_tokens,
resolve_stop_token_ids,
)
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniPromptType
# task -> (sys_type, bot_task, trigger_tag)
_TASK_PRESETS: dict[str, tuple[str, str | None, str | None]] = {
"t2t": ("en_unified", None, None),
"i2t": ("en_unified", None, None),
"it2i_think": ("en_unified", "think", "<think>"),
"it2i_recaption": ("en_unified", "recaption", "<recaption>"),
"t2i_think": ("en_unified", "think", "<think>"),
"t2i_recaption": ("en_unified", "recaption", "<recaption>"),
"t2i_vanilla": ("en_vanilla", "image", None),
# Default deploy configs are absolute so this example works from any cwd.
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3.yaml")
_DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml")
_MODALITY_DEFAULT_DEPLOY_CONFIG = {
"text2img": _DEFAULT_DEPLOY_CONFIG,
"img2img": _DEFAULT_DEPLOY_CONFIG,
"img2text": _DEFAULT_AR_DEPLOY_CONFIG,
"text2text": _DEFAULT_AR_DEPLOY_CONFIG,
}
_MODALITY_MODE = {
"text2img": "text-to-image",
"img2img": "image-editing",
"img2text": "image-to-text",
"text2text": "text-to-text",
}
# Modality → prompt_utils task mapping
_MODALITY_TASK_MAP = {
"text2img": "t2i_think",
"img2img": "it2i_think",
"text2img": "t2i",
"img2img": "it2i",
"img2text": "i2t",
"text2text": "t2t",
}
# Modality → default stage config
_MODALITY_DEFAULT_CONFIG = {
"text2img": "hunyuan_image3_t2i.yaml",
"img2img": "hunyuan_image3_it2i.yaml",
"img2text": "hunyuan_image3_i2t.yaml",
"text2text": "hunyuan_image3_t2t.yaml",
}
def parse_args():
parser = argparse.ArgumentParser(description="HunyuanImage-3.0-Instruct end-to-end inference.")
parser.add_argument(
@@ -94,8 +95,13 @@ def parse_args():
parser.add_argument(
"--bot-task",
type=str,
default=None,
help="Override prompt task (e.g. it2i_think, t2i_recaption). Default: auto from modality.",
default="auto",
choices=["auto", "think", "recaption", "think_recaption", "vanilla"],
help=(
"Prompt behavior. 'auto' selects the default for the modality; "
"'think' adds <think>; 'recaption' adds <recaption>; "
"'vanilla' uses the t2i pretrain template."
),
)
parser.add_argument(
"--sys-type",
@@ -105,7 +111,8 @@ def parse_args():
)
# Omni init args
parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom stage config YAML path.")
parser.add_argument("--deploy-config", type=str, default=None, help="Custom deploy YAML path.")
parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom legacy stage config YAML path.")
parser.add_argument("--log-stats", action="store_true", default=False)
parser.add_argument("--init-timeout", type=int, default=300, help="Initialization timeout in seconds.")
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
@@ -120,23 +127,44 @@ def main():
args = parse_args()
os.makedirs(args.output, exist_ok=True)
# Determine task for prompt formatting
task = args.bot_task or _MODALITY_TASK_MAP[args.modality]
# Determine task for prompt formatting from modality + bot behavior.
task = _MODALITY_TASK_MAP[args.modality]
assert task is not None
bot_task = args.bot_task
if bot_task != "auto":
task = task + "_" + bot_task
if task not in _TASK_PRESETS:
valid_bot_tasks = {
"text2img": ["think", "recaption", "vanilla"],
"img2img": ["think", "recaption"],
"img2text": ["auto"],
"text2text": ["auto"],
}[args.modality]
raise ValueError(
f"--bot-task {bot_task!r} is not supported for {args.modality}. Choose from: {valid_bot_tasks}"
)
# Determine stage config
stage_configs_path = args.stage_configs_path or _MODALITY_DEFAULT_CONFIG[args.modality]
if args.deploy_config is not None and args.stage_configs_path is not None:
raise ValueError("--deploy-config and --stage-configs-path are mutually exclusive.")
deploy_config = args.deploy_config
stage_configs_path = args.stage_configs_path
if deploy_config is None and stage_configs_path is None:
deploy_config = _MODALITY_DEFAULT_DEPLOY_CONFIG[args.modality]
# Build Omni
omni_kwargs = {
"model": args.model,
"vae_use_tiling": args.vae_use_tiling,
"stage_configs_path": stage_configs_path,
"log_stats": args.log_stats,
"init_timeout": args.init_timeout,
"enforce_eager": args.enforce_eager,
}
if args.modality in ("text2img", "img2img"):
omni_kwargs["mode"] = "text-to-image"
if deploy_config is not None:
omni_kwargs["deploy_config"] = deploy_config
else:
omni_kwargs["stage_configs_path"] = stage_configs_path
omni_kwargs["mode"] = _MODALITY_MODE[args.modality]
omni = Omni(**omni_kwargs)
@@ -199,7 +227,9 @@ def main():
# Override diffusion params if applicable
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
for i, sp in enumerate(params_list):
ar_stop_token_ids = resolve_stop_token_ids(task=task, bot_task=bot_task, tokenizer=tokenizer)
assert ar_stop_token_ids is not None
for sp in params_list:
if isinstance(sp, OmniDiffusionSamplingParams):
sp.num_inference_steps = args.steps
sp.guidance_scale = args.guidance_scale
@@ -209,13 +239,20 @@ def main():
if args.modality in ("text2img",):
sp.height = args.height
sp.width = args.width
elif hasattr(sp, "stop_token_ids"):
sp.stop_token_ids = ar_stop_token_ids
# Print configuration
print(f"\n{'=' * 60}")
print("HunyuanImage-3.0 Generation Configuration:")
print(f" Model: {args.model}")
print(f" Modality: {args.modality}")
print(f" Stage config: {stage_configs_path}")
print(f" Prompt task: {task}")
print(f" Bot task: {bot_task}")
if deploy_config is not None:
print(f" Deploy config: {deploy_config}")
else:
print(f" Stage config: {stage_configs_path}")
print(f" Num stages: {omni.num_stages}")
if args.modality in ("text2img", "img2img"):
print(f" Inference steps: {args.steps}")
@@ -236,10 +273,17 @@ def main():
for req_output in omni_outputs:
# Text output (AR stage or text-only)
ro = getattr(req_output, "request_output", None)
txt = ""
if ro and getattr(ro, "outputs", None):
txt = "".join(getattr(o, "text", "") or "" for o in ro.outputs)
if txt:
print(f"[Output] Text:\n{txt}")
if not txt:
ar_text = getattr(req_output, "custom_output", {}).get("ar_generated_text")
if isinstance(ar_text, list):
txt = "\n".join(text for text in ar_text if text)
else:
txt = ar_text or ""
if txt:
print(f"[Output] Text:\n{txt}")
# Image output (DiT stage)
images = getattr(req_output, "images", None)