[Config] Add HunyuanImage3 deploy configs (#3172)
Signed-off-by: KexiongYu <yukexiong1@huawei.com> Signed-off-by: Y. Fisher <yukexiong1@huawei.com>
This commit is contained in:
Y. Fisher authored and GitHub committed
1 parent
c9a8556c24
commit
a33e2eb588
26 files changed
+672
-742
No files matched your search
@@ -1,172 +1,156 @@
|
||||
# HunyuanImage-3.0-Instruct
|
||||
|
||||
## Set up
|
||||
This example runs HunyuanImage-3.0-Instruct offline with the unified deploy
|
||||
YAMLs under `vllm_omni/deploy/`.
|
||||
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
## Deploy Configs
|
||||
|
||||
## Run examples
|
||||
| File | Topology | Default use |
|
||||
| :--- | :--- | :--- |
|
||||
| `vllm_omni/deploy/hunyuan_image3.yaml` | AR + DiT | Default for `text2img` and `img2img`. |
|
||||
| `vllm_omni/deploy/hunyuan_image3_ar.yaml` | AR only | Default for `img2text` and `text2text`. |
|
||||
| `vllm_omni/deploy/hunyuan_image3_dit.yaml` | DiT only | Standalone diffusion stage. Pass it explicitly with `--deploy-config`. |
|
||||
|
||||
**Note**: These examples work with the default configuration on **8x NVIDIA L40S (48GB)**. For different GPU setups, modify the stage configuration to adjust device allocation and memory utilization.
|
||||
The example chooses a deploy config automatically when `--deploy-config` and
|
||||
`--stage-configs-path` are both omitted:
|
||||
|
||||
Get into the hunyuan_image3 folder:
|
||||
| `--modality` | `mode` passed to Omni | Default deploy |
|
||||
| :--- | :--- | :--- |
|
||||
| `text2img` | `text-to-image` | `hunyuan_image3.yaml` |
|
||||
| `img2img` | `image-editing` | `hunyuan_image3.yaml` |
|
||||
| `img2text` | `image-to-text` | `hunyuan_image3_ar.yaml` |
|
||||
| `text2text` | `text-to-text` | `hunyuan_image3_ar.yaml` |
|
||||
|
||||
`--modality` is an offline example convenience flag. It maps to the internal
|
||||
`mode` argument passed to `Omni(...)` by this script. HunyuanImage3 uses
|
||||
separate deploy YAMLs for AR + DiT, AR-only, and DiT-only topologies, so the
|
||||
stage topology is selected by the deploy file rather than by YAML mode
|
||||
overrides.
|
||||
|
||||
Online serving does not expose a `--modality` flag or accept `mode` as an API
|
||||
request field. Choose the deploy topology when starting the server with
|
||||
`--deploy-config`, then use the OpenAI-compatible endpoint and request shape for
|
||||
the scenario. The `modalities` request field is used by the chat completions
|
||||
path; the image endpoints infer the image task from the endpoint and payload.
|
||||
|
||||
| Online scenario | Server deploy | Request |
|
||||
| :--- | :--- | :--- |
|
||||
| Text to image | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/generations`, or `POST /v1/chat/completions` with `"modalities": ["image"]`. |
|
||||
| Image editing | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/edits`. |
|
||||
| Image/text to text | `--deploy-config vllm_omni/deploy/hunyuan_image3_ar.yaml` | `POST /v1/chat/completions` for text output, for example with `"modalities": ["text"]`. |
|
||||
| DiT-only image generation | `--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml` | `POST /v1/images/generations`. |
|
||||
|
||||
## Run Examples
|
||||
|
||||
Text to image, using the default AR + DiT deploy:
|
||||
|
||||
```bash
|
||||
cd examples/offline_inference/hunyuan_image3
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--prompts "A cute cat sitting on a windowsill watching the sunset"
|
||||
```
|
||||
|
||||
### Modality Control
|
||||
|
||||
HunyuanImage-3.0-Instruct supports multiple modality modes. You can control the mode using the `--modality` argument:
|
||||
|
||||
#### Text to Image (text2img)
|
||||
|
||||
- **Pipeline**: Text → AR (CoT + latent tokens) → DiT (denoise) → VAE Decode → Image
|
||||
- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT)
|
||||
- **KV Transfer**: AR sends KV cache to DiT for conditioned generation
|
||||
- **Default Config**: `hunyuan_image3_t2i.yaml`
|
||||
Image editing, using the default AR + DiT deploy:
|
||||
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--prompts "A cute cat sitting on a windowsill watching the sunset"
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2img \
|
||||
--image-path /path/to/image.png \
|
||||
--prompts "Make the petals neon pink"
|
||||
```
|
||||
|
||||
**With VAE tiling (required on A100 GPUs):**
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--prompts "A cute cat sitting on a windowsill watching the sunset" \
|
||||
--vae-use-tiling
|
||||
```
|
||||
|
||||
#### Image to Image (img2img)
|
||||
|
||||
- **Pipeline**: Image + Text → AR (CoT + recaption + latent) → DiT → Edited Image
|
||||
- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT)
|
||||
- **KV Transfer**: AR sends KV cache to DiT
|
||||
- **Default Config**: `hunyuan_image3_it2i.yaml`
|
||||
Image to text, using the AR-only deploy:
|
||||
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2img \
|
||||
--image-path /path/to/image.png \
|
||||
--prompts "Make the petals neon pink"
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2text \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompts "Describe the content of the picture."
|
||||
```
|
||||
|
||||
#### Image to Text (img2text)
|
||||
|
||||
- **Pipeline**: Image + Question → AR → Text description
|
||||
- **Stages Used**: Stage 0 (AR) only
|
||||
- **Default Config**: `hunyuan_image3_i2t.yaml`
|
||||
Text to text, using the AR-only deploy:
|
||||
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2text \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompts "Describe the content of the picture."
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2text \
|
||||
--prompts "What is the capital of France?"
|
||||
```
|
||||
|
||||
#### Text to Text (text2text)
|
||||
|
||||
- **Pipeline**: Text → AR → Text
|
||||
- **Stages Used**: Stage 0 (AR) only
|
||||
- **Default Config**: `hunyuan_image3_t2t.yaml`
|
||||
Standalone DiT, using the DiT-only deploy explicitly:
|
||||
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2text \
|
||||
--prompts "What is the capital of France?"
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \
|
||||
--prompts "A cinematic portrait of an astronaut in a greenhouse"
|
||||
```
|
||||
|
||||
### Inference Steps & Guidance
|
||||
|
||||
Control generation quality for image modalities:
|
||||
Override the default full AR + DiT deploy explicitly:
|
||||
|
||||
```bash
|
||||
python end2end.py --modality text2img \
|
||||
--steps 50 \
|
||||
--guidance-scale 5.0 \
|
||||
--height 1024 --width 1024 \
|
||||
--prompts "A photo-realistic sunset over the ocean"
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--deploy-config vllm_omni/deploy/hunyuan_image3.yaml \
|
||||
--prompts "A cute cat"
|
||||
```
|
||||
|
||||
### Key Arguments
|
||||
## Key Arguments
|
||||
|
||||
#### 📌 Command Line Arguments (end2end.py)
|
||||
| Argument | Description |
|
||||
| :--- | :--- |
|
||||
| `--deploy-config` | Preferred config path for unified deploy YAMLs. |
|
||||
| `--stage-configs-path` | Legacy stage config path, kept only for compatibility. Prefer `--deploy-config`. |
|
||||
| `--modality` | Offline-only convenience flag. One of `text2img`, `img2img`, `img2text`, `text2text`. It selects prompt formatting, internal `mode`, and default deploy config for this script. Online serving uses `--deploy-config` plus the endpoint and, for chat completions, request `modalities` instead. |
|
||||
| `--steps` | Number of diffusion inference steps for image generation. |
|
||||
| `--guidance-scale` | Classifier-free guidance scale for image generation. |
|
||||
| `--height`, `--width` | Output image size for `text2img`. |
|
||||
| `--bot-task` | Prompt behavior. `auto` selects the default from `--modality`; `think` adds `<think>`; `recaption` adds `<recaption>`; `vanilla` uses the text-to-image pretrain template. |
|
||||
| `--sys-type` | Override the system prompt type, for example `en_unified` or `en_vanilla`. |
|
||||
| `--vae-use-tiling` | Enable VAE tiling for memory reduction. |
|
||||
|
||||
| Argument | Type | Default | Description |
|
||||
| :--------------------- | :----- | :----------------------------------- | :----------------------------------------------------------- |
|
||||
| `--model` | string | `tencent/HunyuanImage-3.0-Instruct` | Model path or name |
|
||||
| `--modality` | choice | `text2img` | Modality: `text2img`, `img2img`, `img2text`, `text2text` |
|
||||
| `--prompts` | list | `None` | Input text prompts |
|
||||
| `--image-path` | string | `None` | Input image path (for `img2img`/`img2text`) |
|
||||
| `--output` | string | `.` | Output directory for saved images |
|
||||
| `--steps` | int | `50` | Number of inference steps |
|
||||
| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale |
|
||||
| `--seed` | int | `42` | Random seed |
|
||||
| `--height` | int | `1024` | Output image height |
|
||||
| `--width` | int | `1024` | Output image width |
|
||||
| `--bot-task` | string | auto | Override prompt task (e.g. `it2i_think`, `t2i_recaption`) |
|
||||
| `--sys-type` | string | auto | Override system prompt type (e.g. `en_unified`, `en_vanilla`) |
|
||||
| `--stage-configs-path` | string | auto | Custom stage config YAML path |
|
||||
| `--enforce-eager` | flag | `False` | Disable torch.compile |
|
||||
| `--init-timeout` | int | `300` | Initialization timeout (seconds) |
|
||||
| `--vae-use-tiling` | flag | `False` | Enable VAE tiling for memory optimization (required to avoid OOM on A100) |
|
||||
## Notes
|
||||
|
||||
------
|
||||
|
||||
#### ⚙️ Stage Configurations
|
||||
|
||||
| Config YAML | Modality | Stages | GPUs | Description |
|
||||
| :---------------------------------- | :-------- | :----- | :----- | :------------------------------------ |
|
||||
| `hunyuan_image3_t2i.yaml` | text2img | 2 | 8 | T2I with AR→DiT, 4 GPU each |
|
||||
| `hunyuan_image3_it2i.yaml` | img2img | 2 | 8 | IT2I with AR→DiT, 4 GPU each |
|
||||
| `hunyuan_image3_i2t.yaml` | img2text | 1 | 4 | I2T (AR only) |
|
||||
| `hunyuan_image3_t2t.yaml` | text2text | 1 | 4 | T2T (AR only) |
|
||||
| `hunyuan_image3_t2i_2gpu.yaml` | text2img | 2 | 2 | T2I for 2-GPU setups |
|
||||
| `hunyuan_image3_moe.yaml` | text2img | 2 | 8 | T2I with MoE AR→DiT KV reuse |
|
||||
| `hunyuan_image3_moe_dit_2gpu_fp8.yaml` | text2img | 2 | 2 | T2I with FP8 quantization |
|
||||
|
||||
------
|
||||
|
||||
## Using MoE Config
|
||||
|
||||
The `hunyuan_image3_moe.yaml` config enables AR→DiT KV cache reuse with 8 GPUs (4 for AR + 4 for DiT).
|
||||
|
||||
```bash
|
||||
python end2end.py --model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--stage-configs-path hunyuan_image3_moe.yaml \
|
||||
--prompts "A cute cat"
|
||||
```
|
||||
|
||||
------
|
||||
- `hunyuan_image3_ar.yaml` is a 4-card AR-only text/comprehension deploy. It sets `engine_output_type: text`, `final_output_type: text`, and text sampling defaults.
|
||||
- `hunyuan_image3_dit.yaml` is a single-stage DiT deploy with `stage_id: 0`; it does not require stage 1 or a running AR stage.
|
||||
- The old HunyuanImage3 YAMLs under `model_executor/stage_configs/` and `platforms/*/stage_configs/` have been folded into the deploy YAMLs.
|
||||
- This PR does not keep the HunyuanImage3 AR-to-DiT KV reuse wiring. The deploy YAMLs describe the topology and platform settings only.
|
||||
|
||||
## Prompt Format
|
||||
|
||||
HunyuanImage-3.0-Instruct uses an instruct chat template:
|
||||
|
||||
```
|
||||
<|startoftext|>{system_prompt}\n\nUser: {<img>?}{user_prompt}\n\nAssistant: {trigger_tag?}
|
||||
```text
|
||||
<|startoftext|>{system_prompt}
|
||||
|
||||
User: {<img>?}{user_prompt}
|
||||
|
||||
Assistant: {trigger_tag?}
|
||||
```
|
||||
|
||||
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline)
|
||||
- Trigger tags: `<think>` (CoT), `<recaption>` (recaptioning) — placed AFTER `Assistant: `
|
||||
- System prompt: Auto-selected based on task
|
||||
- `t2i_vanilla` is the only task that uses the bare pretrain template (no chat structure)
|
||||
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline).
|
||||
- Trigger tags: `<think>` for CoT and `<recaption>` for recaptioning, placed after `Assistant: `.
|
||||
- System prompt: Auto-selected based on task.
|
||||
- `t2i_vanilla` is the only task that uses the bare pretrain template without chat structure.
|
||||
- The example composes the internal prompt task from `--modality` and `--bot-task`
|
||||
before calling `prompt_utils`; for example, `img2text + think` becomes
|
||||
`i2t_think` for prompt and stop-token lookup.
|
||||
|
||||
The shared `vllm_omni.diffusion.models.hunyuan_image3.prompt_utils.build_prompt_tokens()`
|
||||
helper handles segment-by-segment tokenization (matches HF `apply_chat_template` byte-for-byte).
|
||||
|
||||
------
|
||||
helper handles segment-by-segment tokenization and matches HF `apply_chat_template`.
|
||||
|
||||
## FAQ
|
||||
|
||||
- **OOM errors**: Decrease `gpu_memory_utilization` in the YAML stage config, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling` (required on A100 GPUs).
|
||||
- **OOM errors**: Decrease `gpu_memory_utilization` in the deploy YAML, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling`.
|
||||
- **Custom image sizes**: Use `--height` and `--width` flags (multiples of 16 recommended).
|
||||
|
||||
| Stage | VRAM (approx) |
|
||||
| :---------------- | :------------------- |
|
||||
| Stage 0 (AR) | ~15 GiB + KV Cache |
|
||||
| Stage 1 (DiT) | ~30 GiB |
|
||||
| Total (8-GPU) | ~45 GiB + KV Cache |
|
||||
| Stage | VRAM (approx) |
|
||||
| :--- | :--- |
|
||||
| Stage 0 (AR) | ~15 GiB + KV Cache |
|
||||
| Stage 1 (DiT) | ~30 GiB |
|
||||
| Total (8-GPU) | ~45 GiB + KV Cache |
|
||||
@@ -15,42 +15,43 @@ Usage:
|
||||
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import (
|
||||
_TASK_PRESETS,
|
||||
build_prompt_tokens,
|
||||
resolve_stop_token_ids,
|
||||
)
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniPromptType
|
||||
|
||||
# task -> (sys_type, bot_task, trigger_tag)
|
||||
_TASK_PRESETS: dict[str, tuple[str, str | None, str | None]] = {
|
||||
"t2t": ("en_unified", None, None),
|
||||
"i2t": ("en_unified", None, None),
|
||||
"it2i_think": ("en_unified", "think", "<think>"),
|
||||
"it2i_recaption": ("en_unified", "recaption", "<recaption>"),
|
||||
"t2i_think": ("en_unified", "think", "<think>"),
|
||||
"t2i_recaption": ("en_unified", "recaption", "<recaption>"),
|
||||
"t2i_vanilla": ("en_vanilla", "image", None),
|
||||
# Default deploy configs are absolute so this example works from any cwd.
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3.yaml")
|
||||
_DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml")
|
||||
|
||||
_MODALITY_DEFAULT_DEPLOY_CONFIG = {
|
||||
"text2img": _DEFAULT_DEPLOY_CONFIG,
|
||||
"img2img": _DEFAULT_DEPLOY_CONFIG,
|
||||
"img2text": _DEFAULT_AR_DEPLOY_CONFIG,
|
||||
"text2text": _DEFAULT_AR_DEPLOY_CONFIG,
|
||||
}
|
||||
|
||||
_MODALITY_MODE = {
|
||||
"text2img": "text-to-image",
|
||||
"img2img": "image-editing",
|
||||
"img2text": "image-to-text",
|
||||
"text2text": "text-to-text",
|
||||
}
|
||||
|
||||
# Modality → prompt_utils task mapping
|
||||
_MODALITY_TASK_MAP = {
|
||||
"text2img": "t2i_think",
|
||||
"img2img": "it2i_think",
|
||||
"text2img": "t2i",
|
||||
"img2img": "it2i",
|
||||
"img2text": "i2t",
|
||||
"text2text": "t2t",
|
||||
}
|
||||
|
||||
|
||||
# Modality → default stage config
|
||||
_MODALITY_DEFAULT_CONFIG = {
|
||||
"text2img": "hunyuan_image3_t2i.yaml",
|
||||
"img2img": "hunyuan_image3_it2i.yaml",
|
||||
"img2text": "hunyuan_image3_i2t.yaml",
|
||||
"text2text": "hunyuan_image3_t2t.yaml",
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="HunyuanImage-3.0-Instruct end-to-end inference.")
|
||||
parser.add_argument(
|
||||
@@ -94,8 +95,13 @@ def parse_args():
|
||||
parser.add_argument(
|
||||
"--bot-task",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Override prompt task (e.g. it2i_think, t2i_recaption). Default: auto from modality.",
|
||||
default="auto",
|
||||
choices=["auto", "think", "recaption", "think_recaption", "vanilla"],
|
||||
help=(
|
||||
"Prompt behavior. 'auto' selects the default for the modality; "
|
||||
"'think' adds <think>; 'recaption' adds <recaption>; "
|
||||
"'vanilla' uses the t2i pretrain template."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sys-type",
|
||||
@@ -105,7 +111,8 @@ def parse_args():
|
||||
)
|
||||
|
||||
# Omni init args
|
||||
parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom stage config YAML path.")
|
||||
parser.add_argument("--deploy-config", type=str, default=None, help="Custom deploy YAML path.")
|
||||
parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom legacy stage config YAML path.")
|
||||
parser.add_argument("--log-stats", action="store_true", default=False)
|
||||
parser.add_argument("--init-timeout", type=int, default=300, help="Initialization timeout in seconds.")
|
||||
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
|
||||
@@ -120,23 +127,44 @@ def main():
|
||||
args = parse_args()
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
# Determine task for prompt formatting
|
||||
task = args.bot_task or _MODALITY_TASK_MAP[args.modality]
|
||||
# Determine task for prompt formatting from modality + bot behavior.
|
||||
task = _MODALITY_TASK_MAP[args.modality]
|
||||
assert task is not None
|
||||
bot_task = args.bot_task
|
||||
if bot_task != "auto":
|
||||
task = task + "_" + bot_task
|
||||
if task not in _TASK_PRESETS:
|
||||
valid_bot_tasks = {
|
||||
"text2img": ["think", "recaption", "vanilla"],
|
||||
"img2img": ["think", "recaption"],
|
||||
"img2text": ["auto"],
|
||||
"text2text": ["auto"],
|
||||
}[args.modality]
|
||||
raise ValueError(
|
||||
f"--bot-task {bot_task!r} is not supported for {args.modality}. Choose from: {valid_bot_tasks}"
|
||||
)
|
||||
|
||||
# Determine stage config
|
||||
stage_configs_path = args.stage_configs_path or _MODALITY_DEFAULT_CONFIG[args.modality]
|
||||
if args.deploy_config is not None and args.stage_configs_path is not None:
|
||||
raise ValueError("--deploy-config and --stage-configs-path are mutually exclusive.")
|
||||
|
||||
deploy_config = args.deploy_config
|
||||
stage_configs_path = args.stage_configs_path
|
||||
if deploy_config is None and stage_configs_path is None:
|
||||
deploy_config = _MODALITY_DEFAULT_DEPLOY_CONFIG[args.modality]
|
||||
|
||||
# Build Omni
|
||||
omni_kwargs = {
|
||||
"model": args.model,
|
||||
"vae_use_tiling": args.vae_use_tiling,
|
||||
"stage_configs_path": stage_configs_path,
|
||||
"log_stats": args.log_stats,
|
||||
"init_timeout": args.init_timeout,
|
||||
"enforce_eager": args.enforce_eager,
|
||||
}
|
||||
if args.modality in ("text2img", "img2img"):
|
||||
omni_kwargs["mode"] = "text-to-image"
|
||||
if deploy_config is not None:
|
||||
omni_kwargs["deploy_config"] = deploy_config
|
||||
else:
|
||||
omni_kwargs["stage_configs_path"] = stage_configs_path
|
||||
omni_kwargs["mode"] = _MODALITY_MODE[args.modality]
|
||||
|
||||
omni = Omni(**omni_kwargs)
|
||||
|
||||
@@ -199,7 +227,9 @@ def main():
|
||||
# Override diffusion params if applicable
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
for i, sp in enumerate(params_list):
|
||||
ar_stop_token_ids = resolve_stop_token_ids(task=task, bot_task=bot_task, tokenizer=tokenizer)
|
||||
assert ar_stop_token_ids is not None
|
||||
for sp in params_list:
|
||||
if isinstance(sp, OmniDiffusionSamplingParams):
|
||||
sp.num_inference_steps = args.steps
|
||||
sp.guidance_scale = args.guidance_scale
|
||||
@@ -209,13 +239,20 @@ def main():
|
||||
if args.modality in ("text2img",):
|
||||
sp.height = args.height
|
||||
sp.width = args.width
|
||||
elif hasattr(sp, "stop_token_ids"):
|
||||
sp.stop_token_ids = ar_stop_token_ids
|
||||
|
||||
# Print configuration
|
||||
print(f"\n{'=' * 60}")
|
||||
print("HunyuanImage-3.0 Generation Configuration:")
|
||||
print(f" Model: {args.model}")
|
||||
print(f" Modality: {args.modality}")
|
||||
print(f" Stage config: {stage_configs_path}")
|
||||
print(f" Prompt task: {task}")
|
||||
print(f" Bot task: {bot_task}")
|
||||
if deploy_config is not None:
|
||||
print(f" Deploy config: {deploy_config}")
|
||||
else:
|
||||
print(f" Stage config: {stage_configs_path}")
|
||||
print(f" Num stages: {omni.num_stages}")
|
||||
if args.modality in ("text2img", "img2img"):
|
||||
print(f" Inference steps: {args.steps}")
|
||||
@@ -236,10 +273,17 @@ def main():
|
||||
for req_output in omni_outputs:
|
||||
# Text output (AR stage or text-only)
|
||||
ro = getattr(req_output, "request_output", None)
|
||||
txt = ""
|
||||
if ro and getattr(ro, "outputs", None):
|
||||
txt = "".join(getattr(o, "text", "") or "" for o in ro.outputs)
|
||||
if txt:
|
||||
print(f"[Output] Text:\n{txt}")
|
||||
if not txt:
|
||||
ar_text = getattr(req_output, "custom_output", {}).get("ar_generated_text")
|
||||
if isinstance(ar_text, list):
|
||||
txt = "\n".join(text for text in ar_text if text)
|
||||
else:
|
||||
txt = ar_text or ""
|
||||
if txt:
|
||||
print(f"[Output] Text:\n{txt}")
|
||||
|
||||
# Image output (DiT stage)
|
||||
images = getattr(req_output, "images", None)
|
||||
|
||||
@@ -24,9 +24,11 @@ import pathlib
|
||||
import pytest
|
||||
|
||||
from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import (
|
||||
HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS,
|
||||
available_tasks,
|
||||
build_prompt,
|
||||
build_prompt_tokens,
|
||||
resolve_stop_token_ids,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
@@ -38,7 +40,7 @@ pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
class FakeTokenizer:
|
||||
"""Minimal tokenizer stub that records every encode() call.
|
||||
|
||||
Returns deterministic ids: special tokens map to small ints (1-4),
|
||||
Returns deterministic ids from convert_tokens_to_ids while
|
||||
encode() returns one id per character starting at 100. This lets
|
||||
tests both verify segmentation (by inspecting `encode_calls`) and
|
||||
locate substrings inside the returned id list.
|
||||
@@ -49,10 +51,17 @@ class FakeTokenizer:
|
||||
"<img>": 2,
|
||||
"<think>": 3,
|
||||
"<recaption>": 4,
|
||||
"<|endoftext|>": 5,
|
||||
"</recaption>": 6,
|
||||
"</answer>": 7,
|
||||
"<boi>": 8,
|
||||
"</think>": 9,
|
||||
**{f"<img_ratio_{i}>": 1000 + i for i in range(33)},
|
||||
}
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.encode_calls: list[str] = []
|
||||
self.eos_token_id = self.SPECIAL["<|endoftext|>"]
|
||||
|
||||
def convert_tokens_to_ids(self, tok: str) -> int:
|
||||
return self.SPECIAL.get(tok, 0)
|
||||
@@ -75,6 +84,14 @@ def test_available_tasks_covers_all_modalities():
|
||||
}
|
||||
|
||||
|
||||
def test_resolve_stop_token_ids_uses_answer_for_generation_tasks():
|
||||
tok = FakeTokenizer()
|
||||
|
||||
answer_id = HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS["<answer>"]
|
||||
assert resolve_stop_token_ids(task="t2i_think", tokenizer=tok) == [answer_id]
|
||||
assert resolve_stop_token_ids(task="t2i_recaption", tokenizer=tok) == [answer_id]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"task",
|
||||
[
|
||||
@@ -164,19 +181,24 @@ def test_build_prompt_tokens_segments_each_boundary():
|
||||
def test_build_prompt_tokens_image_placeholder_present_for_image_tasks():
|
||||
tok = FakeTokenizer()
|
||||
ids = build_prompt_tokens("hi", tok, task="i2t")
|
||||
assert ids[0] == 1, "BOS (<|startoftext|>) must be the first token"
|
||||
assert 2 in ids, "<img> placeholder must be present for i2t/it2i tasks"
|
||||
assert ids[0] == FakeTokenizer.SPECIAL["<|startoftext|>"], "BOS (<|startoftext|>) must be the first token"
|
||||
assert FakeTokenizer.SPECIAL["<img>"] in ids, "<img> placeholder must be present for i2t/it2i tasks"
|
||||
|
||||
|
||||
def test_build_prompt_tokens_no_image_for_text_only_tasks():
|
||||
tok = FakeTokenizer()
|
||||
ids = build_prompt_tokens("hi", tok, task="t2t")
|
||||
assert 2 not in ids, "<img> must NOT appear for text-only tasks"
|
||||
assert FakeTokenizer.SPECIAL["<img>"] not in ids, "<img> must NOT appear for text-only tasks"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"task,trigger_id",
|
||||
[("it2i_think", 3), ("t2i_think", 3), ("it2i_recaption", 4), ("t2i_recaption", 4)],
|
||||
[
|
||||
("it2i_think", FakeTokenizer.SPECIAL["<think>"]),
|
||||
("t2i_think", FakeTokenizer.SPECIAL["<think>"]),
|
||||
("it2i_recaption", FakeTokenizer.SPECIAL["<recaption>"]),
|
||||
("t2i_recaption", FakeTokenizer.SPECIAL["<recaption>"]),
|
||||
],
|
||||
)
|
||||
def test_build_prompt_tokens_trigger_is_last_token(task: str, trigger_id: int):
|
||||
"""Trigger tag id must be the LAST token (after `Assistant: ` segment)."""
|
||||
@@ -189,7 +211,10 @@ def test_build_prompt_tokens_no_trigger_for_plain_tasks():
|
||||
"""Tasks without trigger_tag (t2t / i2t) must NOT append a trigger id."""
|
||||
tok = FakeTokenizer()
|
||||
ids = build_prompt_tokens("hi", tok, task="t2t")
|
||||
assert ids[-1] not in {3, 4} # neither <think> nor <recaption>
|
||||
assert ids[-1] not in {
|
||||
FakeTokenizer.SPECIAL["<think>"],
|
||||
FakeTokenizer.SPECIAL["<recaption>"],
|
||||
}
|
||||
|
||||
|
||||
# -------------------- end2end.py wiring guard --------------------
|
||||
@@ -234,10 +259,16 @@ def test_end2end_routes_through_shared_prompt_utils():
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module and node.module.endswith("hunyuan_image3.prompt_utils"):
|
||||
imported_from_prompt_utils.update(alias.name for alias in node.names)
|
||||
assert "build_prompt_tokens" in imported_from_prompt_utils, (
|
||||
"end2end.py must import build_prompt_tokens from "
|
||||
expected_imports = {
|
||||
"_TASK_PRESETS",
|
||||
"build_prompt_tokens",
|
||||
"resolve_stop_token_ids",
|
||||
}
|
||||
assert expected_imports <= imported_from_prompt_utils, (
|
||||
"end2end.py must import the HunyuanImage3 prompt and stop-token helpers from "
|
||||
"vllm_omni.diffusion.models.hunyuan_image3.prompt_utils -- the shared "
|
||||
"helper is the single source of truth for the AR-prefill template."
|
||||
"module is the single source of truth for the AR-prefill template and "
|
||||
"bot_task-derived AR stop token ids."
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
# ruff: noqa: E501
|
||||
from collections.abc import Generator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
@@ -9,6 +8,7 @@ from PIL import Image
|
||||
from transformers import CLIPModel, CLIPProcessor
|
||||
|
||||
from tests.helpers.runtime import OmniRunner
|
||||
from tests.helpers.stage_config import get_deploy_config_path
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
@@ -16,8 +16,7 @@ from vllm_omni.platforms import current_omni_platform
|
||||
PROMPT = "A brown and white dog is running on the grass"
|
||||
MODEL_NAME = "tencent/HunyuanImage-3.0"
|
||||
LOCAL_CLIP_PATH = "openai/clip-vit-base-patch32"
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
STAGE_CONFIG_PATH = REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "hunyuan_image3_t2i.yaml"
|
||||
DEPLOY_CONFIG_PATH = get_deploy_config_path("hunyuan_image3.yaml")
|
||||
|
||||
pytestmark = [pytest.mark.advanced_model, pytest.mark.diffusion]
|
||||
|
||||
@@ -274,7 +273,8 @@ def clip_bundle() -> tuple[CLIPModel, CLIPProcessor]:
|
||||
def omni() -> Generator[Omni, None, None]:
|
||||
with OmniRunner(
|
||||
MODEL_NAME,
|
||||
stage_configs_path=str(STAGE_CONFIG_PATH),
|
||||
deploy_config=str(DEPLOY_CONFIG_PATH),
|
||||
mode="text-to-image",
|
||||
) as runner:
|
||||
yield runner.omni
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ from vllm_omni.entrypoints.utils import (
|
||||
_filter_dict_like_object,
|
||||
coerce_param_message_types,
|
||||
filter_dataclass_kwargs,
|
||||
filter_stages,
|
||||
load_and_resolve_stage_configs,
|
||||
load_stage_configs_from_yaml,
|
||||
resolve_model_config_path,
|
||||
@@ -401,6 +402,44 @@ class TestLoadAndResolveStageConfigs:
|
||||
assert stage_configs[1].runtime.num_replicas == 3
|
||||
assert stage_configs[1].runtime.devices == "1,2,3"
|
||||
|
||||
def test_filter_stages_selects_mode_stages_without_mutating_stage_config(self, tmp_path):
|
||||
config_path = tmp_path / "deploy.yaml"
|
||||
config_path.write_text(
|
||||
"""modes:
|
||||
- mode: text-to-text
|
||||
stages: [0]
|
||||
- mode: text-to-image
|
||||
stages: [0, 1]
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
stages = [
|
||||
create_config(
|
||||
{
|
||||
"stage_id": 0,
|
||||
"runtime": {"requires_multimodal_data": True},
|
||||
"final_output": False,
|
||||
"final_output_type": None,
|
||||
}
|
||||
),
|
||||
create_config(
|
||||
{
|
||||
"stage_id": 1,
|
||||
"runtime": {"requires_multimodal_data": True},
|
||||
"final_output": True,
|
||||
"final_output_type": "image",
|
||||
}
|
||||
),
|
||||
]
|
||||
|
||||
filtered = filter_stages(str(config_path), stages, {"mode": "text-to-text"})
|
||||
|
||||
assert len(filtered) == 1
|
||||
assert filtered[0].stage_id == 0
|
||||
assert filtered[0].runtime.requires_multimodal_data is True
|
||||
assert filtered[0].final_output is False
|
||||
assert filtered[0].final_output_type is None
|
||||
|
||||
|
||||
class TestLoadStageConfigsFromYaml:
|
||||
"""Regression tests for stage-config loading and merging."""
|
||||
|
||||
@@ -941,6 +941,34 @@ class TestDeployConfigLoading:
|
||||
assert stages[1].yaml_runtime["devices"] == "1,2"
|
||||
assert stages[1].yaml_runtime["num_replicas"] == 2
|
||||
|
||||
def test_merge_pipeline_deploy_preserves_requires_multimodal_data(self):
|
||||
from vllm_omni.config.stage_config import (
|
||||
DeployConfig,
|
||||
PipelineConfig,
|
||||
StageDeployConfig,
|
||||
StageExecutionType,
|
||||
StagePipelineConfig,
|
||||
merge_pipeline_deploy,
|
||||
)
|
||||
|
||||
pipeline = PipelineConfig(
|
||||
model_type="test_mm",
|
||||
model_arch="TestModel",
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="ar",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
requires_multimodal_data=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
deploy = DeployConfig(async_chunk=False, stages=[StageDeployConfig(stage_id=0)])
|
||||
|
||||
stages = merge_pipeline_deploy(pipeline, deploy)
|
||||
|
||||
assert stages[0].yaml_runtime["requires_multimodal_data"] is True
|
||||
|
||||
|
||||
class TestQwen3OmniPipeline:
|
||||
def test_registered(self):
|
||||
|
||||
@@ -65,6 +65,18 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = {
|
||||
"vllm_omni.model_executor.models.glm_image.pipeline",
|
||||
"GLM_IMAGE_PIPELINE",
|
||||
),
|
||||
"hunyuan_image3": (
|
||||
"vllm_omni.model_executor.models.hunyuan_image3.pipeline",
|
||||
"HUNYUAN_IMAGE3_PIPELINE",
|
||||
),
|
||||
"hunyuan_image3_ar": (
|
||||
"vllm_omni.model_executor.models.hunyuan_image3.pipeline",
|
||||
"HUNYUAN_IMAGE3_AR_PIPELINE",
|
||||
),
|
||||
"hunyuan_image3_dit": (
|
||||
"vllm_omni.model_executor.models.hunyuan_image3.pipeline",
|
||||
"HUNYUAN_IMAGE3_DIT_PIPELINE",
|
||||
),
|
||||
"voxcpm2": (
|
||||
"vllm_omni.model_executor.models.voxcpm2.pipeline",
|
||||
"VOXCPM2_PIPELINE",
|
||||
|
||||
@@ -863,6 +863,7 @@ def merge_pipeline_deploy(
|
||||
if ds.devices is not None:
|
||||
runtime["devices"] = ds.devices
|
||||
runtime["num_replicas"] = ds.num_replicas
|
||||
runtime["requires_multimodal_data"] = ps.requires_multimodal_data
|
||||
|
||||
result.append(
|
||||
StageConfig(
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# HunyuanImage-3.0-Instruct deploy: AR (stage 0) + DiT (stage 1)
|
||||
# with AR-to-DiT KV reuse. The base CUDA layout uses 2 GPUs for AR
|
||||
# and 2 GPUs for DiT.
|
||||
pipeline: hunyuan_image3
|
||||
async_chunk: false
|
||||
trust_remote_code: true
|
||||
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
protocol: "rdma"
|
||||
device_name: ""
|
||||
memory_pool_size: 4294967296
|
||||
memory_pool_device: "cpu"
|
||||
shared_memory_connector:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
shm_threshold_bytes: 65536
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
is_comprehension: false
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 32768
|
||||
devices: "0,1"
|
||||
tensor_parallel_size: 2
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
output_connectors:
|
||||
to_stage_1: shared_memory_connector
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1
|
||||
top_k: -1
|
||||
max_tokens: 8192
|
||||
detokenize: true
|
||||
skip_special_tokens: false
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
enforce_eager: true
|
||||
devices: "2,3"
|
||||
distributed_executor_backend: "mp"
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
parallel_config:
|
||||
tensor_parallel_size: 2
|
||||
enable_expert_parallel: true
|
||||
input_connectors:
|
||||
from_stage_0: shared_memory_connector
|
||||
default_sampling_params:
|
||||
num_inference_steps: 50
|
||||
guidance_scale: 0
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
max_inflight: 1
|
||||
|
||||
platforms:
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.8
|
||||
devices: "0,1,2,3"
|
||||
tensor_parallel_size: 4
|
||||
max_num_batched_tokens: 8192
|
||||
- stage_id: 1
|
||||
gpu_memory_utilization: 0.65
|
||||
devices: "4,5,6,7"
|
||||
max_num_batched_tokens: 8192
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: true
|
||||
|
||||
xpu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.95
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
tensor_parallel_size: 8
|
||||
max_num_batched_tokens: 32784
|
||||
quantization: fp8
|
||||
enable_expert_parallel: true
|
||||
worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker
|
||||
default_sampling_params:
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
repetition_penalty: 1.1
|
||||
- stage_id: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
quantization: fp8
|
||||
parallel_config:
|
||||
pipeline_parallel_size: 1
|
||||
data_parallel_size: 1
|
||||
tensor_parallel_size: 8
|
||||
enable_expert_parallel: true
|
||||
sequence_parallel_size: 1
|
||||
ulysses_degree: 1
|
||||
ring_degree: 1
|
||||
cfg_parallel_size: 1
|
||||
vae_patch_parallel_size: 1
|
||||
use_hsdp: false
|
||||
hsdp_shard_size: -1
|
||||
hsdp_replicate_size: 1
|
||||
@@ -0,0 +1,47 @@
|
||||
# HunyuanImage-3.0-Instruct AR-only deploy.
|
||||
#
|
||||
# Use this when AR and DiT are deployed as independent services. This file
|
||||
# resolves to stage 0 only, avoiding the default two-stage topology.
|
||||
pipeline: hunyuan_image3_ar
|
||||
async_chunk: false
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.75
|
||||
trust_remote_code: true
|
||||
enforce_eager: true
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
devices: "0,1,2,3"
|
||||
tensor_parallel_size: 4
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 0.95
|
||||
top_k: 1024
|
||||
max_tokens: 1024
|
||||
detokenize: true
|
||||
|
||||
platforms:
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.75
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
tensor_parallel_size: 8
|
||||
max_num_batched_tokens: 8192
|
||||
|
||||
xpu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.95
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
tensor_parallel_size: 8
|
||||
max_num_batched_tokens: 32784
|
||||
quantization: fp8
|
||||
enable_expert_parallel: true
|
||||
worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker
|
||||
@@ -0,0 +1,64 @@
|
||||
# HunyuanImage-3.0-Instruct DiT-only deploy.
|
||||
#
|
||||
# Use this for standalone DiT/offline execution. This file resolves to one
|
||||
# diffusion stage with stage_id 0 and does not depend on the AR stage.
|
||||
pipeline: hunyuan_image3_dit
|
||||
async_chunk: false
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true
|
||||
devices: "0,1,2,3"
|
||||
vae_use_slicing: false
|
||||
vae_use_tiling: false
|
||||
cache_backend:
|
||||
cache_config:
|
||||
enable_cache_dit_summary: false
|
||||
parallel_config:
|
||||
pipeline_parallel_size: 1
|
||||
data_parallel_size: 1
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: true
|
||||
sequence_parallel_size: 1
|
||||
ulysses_degree: 1
|
||||
ring_degree: 1
|
||||
cfg_parallel_size: 1
|
||||
vae_patch_parallel_size: 1
|
||||
use_hsdp: false
|
||||
hsdp_shard_size: -1
|
||||
hsdp_replicate_size: 1
|
||||
default_sampling_params:
|
||||
seed: 42
|
||||
|
||||
platforms:
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.65
|
||||
devices: "0,1,2,3"
|
||||
max_num_batched_tokens: 32768
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: true
|
||||
|
||||
xpu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
gpu_memory_utilization: 0.9
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
quantization: fp8
|
||||
parallel_config:
|
||||
pipeline_parallel_size: 1
|
||||
data_parallel_size: 1
|
||||
tensor_parallel_size: 8
|
||||
enable_expert_parallel: true
|
||||
sequence_parallel_size: 1
|
||||
ulysses_degree: 1
|
||||
ring_degree: 1
|
||||
cfg_parallel_size: 1
|
||||
vae_patch_parallel_size: 1
|
||||
use_hsdp: false
|
||||
hsdp_shard_size: -1
|
||||
hsdp_replicate_size: 1
|
||||
@@ -1424,7 +1424,11 @@ class HunyuanImage3Pipeline(
|
||||
model_inputs.update(ar_kv_kwargs)
|
||||
|
||||
outputs = self._generate(**model_inputs, **kwargs)
|
||||
custom_output = {}
|
||||
if any(t is not None for t in cot_text_list):
|
||||
custom_output["ar_generated_text"] = cot_text_list[0] if len(cot_text_list) == 1 else cot_text_list
|
||||
return DiffusionOutput(
|
||||
output=outputs[0],
|
||||
custom_output=custom_output,
|
||||
stage_durations=self.stage_durations if hasattr(self, "stage_durations") else None,
|
||||
)
|
||||
@@ -17,17 +17,43 @@ canonical mapping for both flows.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .system_prompt import get_system_prompt
|
||||
|
||||
# HunyuanImage-3.0-Instruct special token ids from tokenizer.json.
|
||||
# Keep offline AR prompt/stop-token behavior independent of runtime
|
||||
# tokenizer lookup for these fixed control tokens.
|
||||
HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS: dict[str, int] = {
|
||||
"<|endoftext|>": 127957,
|
||||
"<|startoftext|>": 127958,
|
||||
"<boi>": 128000,
|
||||
"<eoi>": 128001,
|
||||
"<img>": 128006,
|
||||
"<cfg>": 128010,
|
||||
"<recaption>": 128018,
|
||||
"</recaption>": 128019,
|
||||
"<think>": 128023,
|
||||
"</think>": 128024,
|
||||
"<answer>": 128025,
|
||||
"</answer>": 128026,
|
||||
"<img_size_1024>": 128037,
|
||||
"<img_ratio_0>": 128044,
|
||||
"<img_ratio_32>": 128076,
|
||||
"<img_ratio_33>": 130103,
|
||||
"<img_ratio_36>": 130106,
|
||||
}
|
||||
|
||||
# task -> (sys_type, bot_task, trigger_tag)
|
||||
_TASK_PRESETS: dict[str, tuple[str, str | None, str | None]] = {
|
||||
"t2t": ("en_unified", None, None),
|
||||
"i2t": ("en_unified", None, None),
|
||||
"it2i_think": ("en_unified", "think", "<think>"),
|
||||
"it2i_recaption": ("en_unified", "recaption", "<recaption>"),
|
||||
"t2i": ("en_unified", "image", None),
|
||||
"t2i_vanilla": ("en_vanilla", "image", None),
|
||||
"t2i_think": ("en_unified", "think", "<think>"),
|
||||
"t2i_recaption": ("en_unified", "recaption", "<recaption>"),
|
||||
"t2i_vanilla": ("en_vanilla", "image", None),
|
||||
}
|
||||
|
||||
|
||||
@@ -36,6 +62,14 @@ def available_tasks() -> list[str]:
|
||||
return sorted(_TASK_PRESETS)
|
||||
|
||||
|
||||
def resolve_stop_token_ids(
|
||||
task: str = "it2i_think",
|
||||
bot_task: str = "think",
|
||||
tokenizer: Any | None = None,
|
||||
):
|
||||
return [HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS["<answer>"]]
|
||||
|
||||
|
||||
def build_prompt(
|
||||
user_prompt: str,
|
||||
task: str = "it2i_think",
|
||||
@@ -149,4 +183,4 @@ def build_prompt_tokens(
|
||||
return ids
|
||||
|
||||
|
||||
__all__ = ["build_prompt", "build_prompt_tokens", "available_tasks"]
|
||||
__all__ = ["build_prompt", "build_prompt_tokens", "resolve_stop_token_ids", _TASK_PRESETS]
|
||||
@@ -593,6 +593,7 @@ def run_headless(args: argparse.Namespace) -> None:
|
||||
model,
|
||||
args_dict.get("stage_configs_path"),
|
||||
args_dict,
|
||||
deploy_config_path=args_dict.get("deploy_config"),
|
||||
)
|
||||
|
||||
# Locate the stage config that matches stage_id.
|
||||
|
||||
@@ -1517,14 +1517,9 @@ class HunyuanImage3ForConditionalGeneration(nn.Module, SupportsMultiModal, Suppo
|
||||
|
||||
# For comprehension mode, block image generation tokens but allow
|
||||
# text structure tokens (<think>, <answer>, etc.) so the model can
|
||||
# follow its natural generation pattern. The yaml stop_token_ids
|
||||
# for i2t/t2t now includes </think> (128024) so the AR-only output
|
||||
# terminates after the analysis section, matching HF's
|
||||
# `bot_task="think"` behavior. Without that stop, the model
|
||||
# continues into a recaption section even in comprehension mode
|
||||
# (the stage-transition processor only fires in generation mode,
|
||||
# but the instruct-tuned model writes recaption on its own from
|
||||
# internal habit).
|
||||
# follow its natural generation pattern. Runtime sampling params
|
||||
# decide stop tokens from the active bot_task, matching the official
|
||||
# HunyuanImage3 generation path without hard-coded YAML token ids.
|
||||
self._blocked_token_ids: set[int] = set()
|
||||
if self._is_comprehension:
|
||||
self._blocked_token_ids.update(
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""HunyuanImage3 pipeline topology."""
|
||||
|
||||
from vllm_omni.config.stage_config import (
|
||||
PipelineConfig,
|
||||
StageExecutionType,
|
||||
StagePipelineConfig,
|
||||
)
|
||||
|
||||
_HUNYUAN_IMAGE3_HF_ARCHS = (
|
||||
"HunyuanImage3ForConditionalGeneration",
|
||||
"HunyuanImage3ForCausalMM",
|
||||
)
|
||||
_HUNYUAN_IMAGE3_MODEL_ARCH = "HunyuanImage3ForCausalMM"
|
||||
_HUNYUAN_IMAGE3_INPUT_PROCESSOR = "vllm_omni.model_executor.stage_input_processors.hunyuan_image3"
|
||||
|
||||
|
||||
HUNYUAN_IMAGE3_PIPELINE = PipelineConfig(
|
||||
model_type="hunyuan_image3",
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
hf_architectures=_HUNYUAN_IMAGE3_HF_ARCHS,
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="AR",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
input_sources=(),
|
||||
final_output=False,
|
||||
owns_tokenizer=False,
|
||||
requires_multimodal_data=True,
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
engine_output_type="latent",
|
||||
),
|
||||
StagePipelineConfig(
|
||||
stage_id=1,
|
||||
model_stage="dit",
|
||||
execution_type=StageExecutionType.DIFFUSION,
|
||||
input_sources=(0,),
|
||||
final_output=True,
|
||||
final_output_type="image",
|
||||
requires_multimodal_data=True,
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
custom_process_input_func=f"{_HUNYUAN_IMAGE3_INPUT_PROCESSOR}.ar2diffusion",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
HUNYUAN_IMAGE3_AR_PIPELINE = PipelineConfig(
|
||||
model_type="hunyuan_image3_ar",
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
hf_architectures=(),
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="AR",
|
||||
execution_type=StageExecutionType.LLM_AR,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="text",
|
||||
owns_tokenizer=False,
|
||||
requires_multimodal_data=True,
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
engine_output_type="latent",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
HUNYUAN_IMAGE3_DIT_PIPELINE = PipelineConfig(
|
||||
model_type="hunyuan_image3_dit",
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
hf_architectures=(),
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
model_stage="dit",
|
||||
execution_type=StageExecutionType.DIFFUSION,
|
||||
input_sources=(),
|
||||
final_output=True,
|
||||
final_output_type="image",
|
||||
requires_multimodal_data=True,
|
||||
model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH,
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -1,41 +0,0 @@
|
||||
# Stage config for HunyuanImage-3.0 Image-to-Text (I2T / image understanding).
|
||||
# Single LLM stage: AR model reads image + text prompt, generates text output.
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: true
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
max_num_seqs: 1
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 0.95
|
||||
top_k: 1024
|
||||
max_tokens: 2048
|
||||
stop_token_ids: [127957, 128024, 128026] # <|endoftext|>, </think>, </answer>
|
||||
detokenize: True
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,72 +0,0 @@
|
||||
# Stage config for HunyuanImage-3.0 Image+Text-to-Image (image editing).
|
||||
# Stage 0: AR (HunyuanImage3ForConditionalGeneration) — reads (image, text), emits latent tokens
|
||||
# Stage 1: Diffusion (HunyuanImage3Pipeline / DiT + VAE) — denoise + decode latents → image
|
||||
|
||||
stage_args:
|
||||
# Stage 0: AR Model
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: true # AR needs the original image
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # AR outputs latent for DiT
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
is_comprehension: false # Generation task, not comprehension
|
||||
final_output: false # AR is not the final output
|
||||
default_sampling_params:
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 1024
|
||||
max_tokens: 4096
|
||||
stop_token_ids: [127957] # <|endoftext|>
|
||||
detokenize: true # DiT bridge consumes ar_generated_text; let the AR engine produce it
|
||||
|
||||
# Stage 1: Diffusion (DiT + VAE)
|
||||
# Receives latents from AR stage, performs denoising + VAE decode
|
||||
- stage_id: 1
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
process: true
|
||||
devices: "4,5,6,7"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: true # May need condition images
|
||||
engine_args:
|
||||
model_stage: dit
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: "mp"
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: true
|
||||
engine_input_source: [0] # Input from AR stage
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.hunyuan_image3.ar2diffusion
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
default_sampling_params:
|
||||
num_inference_steps: 50
|
||||
guidance_scale: 2.5
|
||||
|
||||
# Top-level runtime config
|
||||
runtime:
|
||||
enabled: true
|
||||
edges:
|
||||
- from: 0 # AR → Diffusion
|
||||
to: 1
|
||||
@@ -1,89 +0,0 @@
|
||||
stage_args:
|
||||
# Stage 0: AR Model
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: true # AR needs the original image
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # AR outputs latent for DiT
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 2
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
is_comprehension: false # Generation task, not comprehension
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1
|
||||
top_k: -1
|
||||
max_tokens: 8192
|
||||
stop_token_ids: [128025] # <answer>
|
||||
detokenize: true # DiT bridge consumes ar_generated_text; let the AR engine produce it
|
||||
skip_special_tokens: False
|
||||
output_connectors:
|
||||
to_stage_1: rdma_connector
|
||||
|
||||
# Stage 1: Diffusion (DiT + VAE)
|
||||
# Receives latents from AR stage, performs denoising + VAE decode
|
||||
- stage_id: 1
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
process: true
|
||||
devices: "2,3"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: true # May need condition images
|
||||
engine_args:
|
||||
model_stage: dit
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: "mp"
|
||||
omni_kv_config:
|
||||
need_recv_cache: true # Receive AR KV cache from stage 0
|
||||
parallel_config:
|
||||
tensor_parallel_size: 2
|
||||
enable_expert_parallel: true
|
||||
engine_input_source: [0] # Input from AR stage
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.hunyuan_image3.ar2diffusion
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
default_sampling_params:
|
||||
num_inference_steps: 50
|
||||
guidance_scale: 0
|
||||
input_connectors:
|
||||
from_stage_0: rdma_connector
|
||||
|
||||
|
||||
# Top-level runtime config
|
||||
runtime:
|
||||
enabled: true
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
protocol: "rdma"
|
||||
device_name: ""
|
||||
memory_pool_size: 4294967296
|
||||
memory_pool_device: "cpu"
|
||||
edges:
|
||||
- from: 0 # AR → Diffusion
|
||||
to: 1
|
||||
@@ -1,96 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 with AR→DiT KV reuse.
|
||||
# Stage 0: AR Model (vLLM implementation)
|
||||
# Stage 1: DiT Model (diffusion)
|
||||
#
|
||||
# text-to-image flow: AR (stage 0) → KV transfer → DiT (stage 1)
|
||||
# image-to-text flow: AR (stage 0) only
|
||||
#
|
||||
# Compared to hunyuan_image3_t2i.yaml, this config:
|
||||
# 1. Enables both stages [0, 1] for text-to-image (AR prefill + DiT denoising)
|
||||
# 2. Adds omni_kv_config to send/receive KV cache between stages
|
||||
|
||||
# The following config has been verified on 8x L40S-48G GPU (4 for AR + 4 for DiT).
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm # Use llm stage type for AR stages
|
||||
runtime:
|
||||
process: true # Run this stage in a separate process
|
||||
devices: "0,1,2,3" # AR stage uses GPU 0-3
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
max_num_seqs: 1
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true # Now we only support eager mode
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
kv_transfer_criteria:
|
||||
type: prefill_finished # Send KV cache after AR prefill completes
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
- stage_id: 1
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
process: true
|
||||
devices: "4,5,6,7" # DiT stage uses GPU 4-7
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: diffusion
|
||||
enforce_eager: true
|
||||
distributed_executor_backend: "mp"
|
||||
vae_use_slicing: false
|
||||
vae_use_tiling: false
|
||||
cache_backend: null
|
||||
cache_config: null
|
||||
enable_cache_dit_summary: false
|
||||
omni_kv_config:
|
||||
need_recv_cache: true # Receive AR KV cache from stage 0
|
||||
parallel_config:
|
||||
pipeline_parallel_size: 1
|
||||
data_parallel_size: 1
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: false
|
||||
sequence_parallel_size: 1
|
||||
ulysses_degree: 1
|
||||
ring_degree: 1
|
||||
cfg_parallel_size: 1
|
||||
vae_patch_parallel_size: 1
|
||||
use_hsdp: false
|
||||
hsdp_shard_size: -1
|
||||
hsdp_replicate_size: 1
|
||||
engine_input_source: [0] # Receive input (including KV) from stage 0
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
|
||||
# Top-level runtime config: windows, edges, and connectors
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
window_size: -1 # Trigger downstream only after full upstream completion
|
||||
max_inflight: 1 # Process serially within each stage
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
@@ -1,32 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 DiT with FP8 online quantization.
|
||||
# The following config is for 2x H200 GPU.
|
||||
|
||||
# Stage 0: Diffusion (DiT + VAE)
|
||||
# This stage receives noise and timesteps and performs denoising + VAE decode
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
devices: "0,1"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: dit
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: "mp"
|
||||
quantization: "fp8"
|
||||
parallel_config:
|
||||
tensor_parallel_size: 2
|
||||
enable_expert_parallel: true
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
is_comprehension: false
|
||||
default_sampling_params:
|
||||
seed: 42
|
||||
|
||||
# Runtime edges
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,31 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 DiT.
|
||||
# The following config has been verified on 4x H20 GPU.
|
||||
|
||||
# Stage 0: Diffusion (DiT + VAE)
|
||||
# This stage receives noise and timesteps and performs denoising + VAE decode
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
engine_args:
|
||||
max_num_seqs: 1
|
||||
model_stage: dit
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
distributed_executor_backend: "mp"
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
enable_expert_parallel: true
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
is_comprehension: false
|
||||
default_sampling_params:
|
||||
seed: 42
|
||||
|
||||
# Runtime edges
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,41 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 on 2 GPUs with FP8.
|
||||
# Stage 0: AR Model (vLLM implementation)
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1"
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
max_num_seqs: 1
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 2
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,42 +0,0 @@
|
||||
# Stage config for HunyuanImage-3.0 Text-to-Text (T2T / pure text generation).
|
||||
# Single LLM stage: AR model reads text prompt only, generates text output.
|
||||
# Sampling params aligned with official generation_config.json.
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1,2,3"
|
||||
max_batch_size: 1
|
||||
requires_multimodal_data: false
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
max_num_seqs: 1
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
tensor_parallel_size: 4
|
||||
pipeline_parallel_size: 1
|
||||
hf_overrides:
|
||||
rope_parameters:
|
||||
mrope_section: [0, 32, 32]
|
||||
rope_type: default
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 0.95
|
||||
top_k: 1024
|
||||
max_tokens: 2048
|
||||
stop_token_ids: [127957, 128024, 128026] # <|endoftext|>, </think>, </answer>
|
||||
detokenize: True
|
||||
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,35 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 DiT on NPU.
|
||||
# The following config has been verified on 8x A3-64G NPUs.
|
||||
|
||||
# Stage 0: Diffusion (DiT + VAE)
|
||||
# This stage receives noise and timesteps and performs denoising + VAE decode.
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
engine_args:
|
||||
max_num_seqs: 1
|
||||
model_stage: dit
|
||||
gpu_memory_utilization: 0.65
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: image
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
parallel_config:
|
||||
tensor_parallel_size: 8
|
||||
enable_expert_parallel: true
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
is_comprehension: false
|
||||
default_sampling_params:
|
||||
seed: 42
|
||||
|
||||
# Runtime defaults
|
||||
runtime:
|
||||
enabled: true
|
||||
@@ -1,80 +0,0 @@
|
||||
# Stage config for running Hunyuan-Image3.0 with architecture of OmniLLM.
|
||||
# Stage 0: AR Model (vLLM implementation)
|
||||
|
||||
# The following config has been verified on 8x Max 1550 GPU.
|
||||
modes:
|
||||
- mode: text-to-image
|
||||
stages: [1]
|
||||
- mode: image-to-text
|
||||
stages: [0]
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm # Use llm stage type to launch OmniLLM
|
||||
runtime:
|
||||
process: true # Run this stage in a separate process
|
||||
devices: "0,1,2,3,4,5,6,7" # Visible devices for this stage
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: AR
|
||||
model_arch: HunyuanImage3ForCausalMM
|
||||
worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.95
|
||||
enforce_eager: true # Now we only support eager mode
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32784
|
||||
tensor_parallel_size: 8
|
||||
pipeline_parallel_size: 1
|
||||
enable_expert_parallel: true
|
||||
quantization: "fp8"
|
||||
is_comprehension: true
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
- stage_id: 1
|
||||
stage_type: diffusion
|
||||
runtime:
|
||||
process: true
|
||||
devices: "0,1,2,3,4,5,6,7"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: diffusion
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true
|
||||
engine_output_type: image
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
vae_use_slicing: false
|
||||
vae_use_tiling: false
|
||||
cache_backend: null
|
||||
cache_config: null
|
||||
enable_cache_dit_summary: false
|
||||
quantization: "fp8"
|
||||
parallel_config:
|
||||
pipeline_parallel_size: 1
|
||||
data_parallel_size: 1
|
||||
tensor_parallel_size: 8
|
||||
enable_expert_parallel: true
|
||||
sequence_parallel_size: 1
|
||||
ulysses_degree: 1
|
||||
ring_degree: 1
|
||||
cfg_parallel_size: 1
|
||||
vae_patch_parallel_size: 1
|
||||
use_hsdp: false
|
||||
hsdp_shard_size: -1
|
||||
hsdp_replicate_size: 1
|
||||
final_output: true
|
||||
final_output_type: image
|
||||
|
||||
# Top-level runtime config (concise): default windows and stage edges
|
||||
runtime:
|
||||
enabled: true
|
||||
Reference in new issue
Block a user