diff --git a/examples/offline_inference/hunyuan_image3/README.md b/examples/offline_inference/hunyuan_image3/README.md index c1c97bfe1..98908ace0 100644 --- a/examples/offline_inference/hunyuan_image3/README.md +++ b/examples/offline_inference/hunyuan_image3/README.md @@ -1,172 +1,156 @@ # HunyuanImage-3.0-Instruct -## Set up +This example runs HunyuanImage-3.0-Instruct offline with the unified deploy +YAMLs under `vllm_omni/deploy/`. -Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. +## Deploy Configs -## Run examples +| File | Topology | Default use | +| :--- | :--- | :--- | +| `vllm_omni/deploy/hunyuan_image3.yaml` | AR + DiT | Default for `text2img` and `img2img`. | +| `vllm_omni/deploy/hunyuan_image3_ar.yaml` | AR only | Default for `img2text` and `text2text`. | +| `vllm_omni/deploy/hunyuan_image3_dit.yaml` | DiT only | Standalone diffusion stage. Pass it explicitly with `--deploy-config`. | -**Note**: These examples work with the default configuration on **8x NVIDIA L40S (48GB)**. For different GPU setups, modify the stage configuration to adjust device allocation and memory utilization. +The example chooses a deploy config automatically when `--deploy-config` and +`--stage-configs-path` are both omitted: -Get into the hunyuan_image3 folder: +| `--modality` | `mode` passed to Omni | Default deploy | +| :--- | :--- | :--- | +| `text2img` | `text-to-image` | `hunyuan_image3.yaml` | +| `img2img` | `image-editing` | `hunyuan_image3.yaml` | +| `img2text` | `image-to-text` | `hunyuan_image3_ar.yaml` | +| `text2text` | `text-to-text` | `hunyuan_image3_ar.yaml` | + +`--modality` is an offline example convenience flag. It maps to the internal +`mode` argument passed to `Omni(...)` by this script. HunyuanImage3 uses +separate deploy YAMLs for AR + DiT, AR-only, and DiT-only topologies, so the +stage topology is selected by the deploy file rather than by YAML mode +overrides. + +Online serving does not expose a `--modality` flag or accept `mode` as an API +request field. Choose the deploy topology when starting the server with +`--deploy-config`, then use the OpenAI-compatible endpoint and request shape for +the scenario. The `modalities` request field is used by the chat completions +path; the image endpoints infer the image task from the endpoint and payload. + +| Online scenario | Server deploy | Request | +| :--- | :--- | :--- | +| Text to image | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/generations`, or `POST /v1/chat/completions` with `"modalities": ["image"]`. | +| Image editing | `--deploy-config vllm_omni/deploy/hunyuan_image3.yaml` | `POST /v1/images/edits`. | +| Image/text to text | `--deploy-config vllm_omni/deploy/hunyuan_image3_ar.yaml` | `POST /v1/chat/completions` for text output, for example with `"modalities": ["text"]`. | +| DiT-only image generation | `--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml` | `POST /v1/images/generations`. | + +## Run Examples + +Text to image, using the default AR + DiT deploy: ```bash -cd examples/offline_inference/hunyuan_image3 +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality text2img \ + --prompts "A cute cat sitting on a windowsill watching the sunset" ``` -### Modality Control - -HunyuanImage-3.0-Instruct supports multiple modality modes. You can control the mode using the `--modality` argument: - -#### Text to Image (text2img) - -- **Pipeline**: Text → AR (CoT + latent tokens) → DiT (denoise) → VAE Decode → Image -- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT) -- **KV Transfer**: AR sends KV cache to DiT for conditioned generation -- **Default Config**: `hunyuan_image3_t2i.yaml` +Image editing, using the default AR + DiT deploy: ```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality text2img \ - --prompts "A cute cat sitting on a windowsill watching the sunset" +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality img2img \ + --image-path /path/to/image.png \ + --prompts "Make the petals neon pink" ``` -**With VAE tiling (required on A100 GPUs):** -```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality text2img \ - --prompts "A cute cat sitting on a windowsill watching the sunset" \ - --vae-use-tiling -``` - -#### Image to Image (img2img) - -- **Pipeline**: Image + Text → AR (CoT + recaption + latent) → DiT → Edited Image -- **Stages Used**: Stage 0 (AR) + Stage 1 (DiT) -- **KV Transfer**: AR sends KV cache to DiT -- **Default Config**: `hunyuan_image3_it2i.yaml` +Image to text, using the AR-only deploy: ```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality img2img \ - --image-path /path/to/image.png \ - --prompts "Make the petals neon pink" +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality img2text \ + --image-path /path/to/image.jpg \ + --prompts "Describe the content of the picture." ``` -#### Image to Text (img2text) - -- **Pipeline**: Image + Question → AR → Text description -- **Stages Used**: Stage 0 (AR) only -- **Default Config**: `hunyuan_image3_i2t.yaml` +Text to text, using the AR-only deploy: ```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality img2text \ - --image-path /path/to/image.jpg \ - --prompts "Describe the content of the picture." +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality text2text \ + --prompts "What is the capital of France?" ``` -#### Text to Text (text2text) - -- **Pipeline**: Text → AR → Text -- **Stages Used**: Stage 0 (AR) only -- **Default Config**: `hunyuan_image3_t2t.yaml` +Standalone DiT, using the DiT-only deploy explicitly: ```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality text2text \ - --prompts "What is the capital of France?" +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality text2img \ + --deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \ + --prompts "A cinematic portrait of an astronaut in a greenhouse" ``` -### Inference Steps & Guidance - -Control generation quality for image modalities: +Override the default full AR + DiT deploy explicitly: ```bash -python end2end.py --modality text2img \ - --steps 50 \ - --guidance-scale 5.0 \ - --height 1024 --width 1024 \ - --prompts "A photo-realistic sunset over the ocean" +python examples/offline_inference/hunyuan_image3/end2end.py \ + --model tencent/HunyuanImage-3.0-Instruct \ + --modality text2img \ + --deploy-config vllm_omni/deploy/hunyuan_image3.yaml \ + --prompts "A cute cat" ``` -### Key Arguments +## Key Arguments -#### 📌 Command Line Arguments (end2end.py) +| Argument | Description | +| :--- | :--- | +| `--deploy-config` | Preferred config path for unified deploy YAMLs. | +| `--stage-configs-path` | Legacy stage config path, kept only for compatibility. Prefer `--deploy-config`. | +| `--modality` | Offline-only convenience flag. One of `text2img`, `img2img`, `img2text`, `text2text`. It selects prompt formatting, internal `mode`, and default deploy config for this script. Online serving uses `--deploy-config` plus the endpoint and, for chat completions, request `modalities` instead. | +| `--steps` | Number of diffusion inference steps for image generation. | +| `--guidance-scale` | Classifier-free guidance scale for image generation. | +| `--height`, `--width` | Output image size for `text2img`. | +| `--bot-task` | Prompt behavior. `auto` selects the default from `--modality`; `think` adds ``; `recaption` adds ``; `vanilla` uses the text-to-image pretrain template. | +| `--sys-type` | Override the system prompt type, for example `en_unified` or `en_vanilla`. | +| `--vae-use-tiling` | Enable VAE tiling for memory reduction. | -| Argument | Type | Default | Description | -| :--------------------- | :----- | :----------------------------------- | :----------------------------------------------------------- | -| `--model` | string | `tencent/HunyuanImage-3.0-Instruct` | Model path or name | -| `--modality` | choice | `text2img` | Modality: `text2img`, `img2img`, `img2text`, `text2text` | -| `--prompts` | list | `None` | Input text prompts | -| `--image-path` | string | `None` | Input image path (for `img2img`/`img2text`) | -| `--output` | string | `.` | Output directory for saved images | -| `--steps` | int | `50` | Number of inference steps | -| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale | -| `--seed` | int | `42` | Random seed | -| `--height` | int | `1024` | Output image height | -| `--width` | int | `1024` | Output image width | -| `--bot-task` | string | auto | Override prompt task (e.g. `it2i_think`, `t2i_recaption`) | -| `--sys-type` | string | auto | Override system prompt type (e.g. `en_unified`, `en_vanilla`) | -| `--stage-configs-path` | string | auto | Custom stage config YAML path | -| `--enforce-eager` | flag | `False` | Disable torch.compile | -| `--init-timeout` | int | `300` | Initialization timeout (seconds) | -| `--vae-use-tiling` | flag | `False` | Enable VAE tiling for memory optimization (required to avoid OOM on A100) | +## Notes ------- - -#### ⚙️ Stage Configurations - -| Config YAML | Modality | Stages | GPUs | Description | -| :---------------------------------- | :-------- | :----- | :----- | :------------------------------------ | -| `hunyuan_image3_t2i.yaml` | text2img | 2 | 8 | T2I with AR→DiT, 4 GPU each | -| `hunyuan_image3_it2i.yaml` | img2img | 2 | 8 | IT2I with AR→DiT, 4 GPU each | -| `hunyuan_image3_i2t.yaml` | img2text | 1 | 4 | I2T (AR only) | -| `hunyuan_image3_t2t.yaml` | text2text | 1 | 4 | T2T (AR only) | -| `hunyuan_image3_t2i_2gpu.yaml` | text2img | 2 | 2 | T2I for 2-GPU setups | -| `hunyuan_image3_moe.yaml` | text2img | 2 | 8 | T2I with MoE AR→DiT KV reuse | -| `hunyuan_image3_moe_dit_2gpu_fp8.yaml` | text2img | 2 | 2 | T2I with FP8 quantization | - ------- - -## Using MoE Config - -The `hunyuan_image3_moe.yaml` config enables AR→DiT KV cache reuse with 8 GPUs (4 for AR + 4 for DiT). - -```bash -python end2end.py --model tencent/HunyuanImage-3.0-Instruct \ - --modality text2img \ - --stage-configs-path hunyuan_image3_moe.yaml \ - --prompts "A cute cat" -``` - ------- +- `hunyuan_image3_ar.yaml` is a 4-card AR-only text/comprehension deploy. It sets `engine_output_type: text`, `final_output_type: text`, and text sampling defaults. +- `hunyuan_image3_dit.yaml` is a single-stage DiT deploy with `stage_id: 0`; it does not require stage 1 or a running AR stage. +- The old HunyuanImage3 YAMLs under `model_executor/stage_configs/` and `platforms/*/stage_configs/` have been folded into the deploy YAMLs. +- This PR does not keep the HunyuanImage3 AR-to-DiT KV reuse wiring. The deploy YAMLs describe the topology and platform settings only. ## Prompt Format HunyuanImage-3.0-Instruct uses an instruct chat template: -``` -<|startoftext|>{system_prompt}\n\nUser: {?}{user_prompt}\n\nAssistant: {trigger_tag?} +```text +<|startoftext|>{system_prompt} + +User: {?}{user_prompt} + +Assistant: {trigger_tag?} ``` -- ``: Placeholder for each input image (single token; expanded by the multimodal pipeline) -- Trigger tags: `` (CoT), `` (recaptioning) — placed AFTER `Assistant: ` -- System prompt: Auto-selected based on task -- `t2i_vanilla` is the only task that uses the bare pretrain template (no chat structure) +- ``: Placeholder for each input image (single token; expanded by the multimodal pipeline). +- Trigger tags: `` for CoT and `` for recaptioning, placed after `Assistant: `. +- System prompt: Auto-selected based on task. +- `t2i_vanilla` is the only task that uses the bare pretrain template without chat structure. +- The example composes the internal prompt task from `--modality` and `--bot-task` + before calling `prompt_utils`; for example, `img2text + think` becomes + `i2t_think` for prompt and stop-token lookup. The shared `vllm_omni.diffusion.models.hunyuan_image3.prompt_utils.build_prompt_tokens()` -helper handles segment-by-segment tokenization (matches HF `apply_chat_template` byte-for-byte). - ------- +helper handles segment-by-segment tokenization and matches HF `apply_chat_template`. ## FAQ -- **OOM errors**: Decrease `gpu_memory_utilization` in the YAML stage config, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling` (required on A100 GPUs). +- **OOM errors**: Decrease `gpu_memory_utilization` in the deploy YAML, use a smaller `max_num_batched_tokens`, or enable VAE tiling with `--vae-use-tiling`. - **Custom image sizes**: Use `--height` and `--width` flags (multiples of 16 recommended). -| Stage | VRAM (approx) | -| :---------------- | :------------------- | -| Stage 0 (AR) | ~15 GiB + KV Cache | -| Stage 1 (DiT) | ~30 GiB | -| Total (8-GPU) | ~45 GiB + KV Cache | +| Stage | VRAM (approx) | +| :--- | :--- | +| Stage 0 (AR) | ~15 GiB + KV Cache | +| Stage 1 (DiT) | ~30 GiB | +| Total (8-GPU) | ~45 GiB + KV Cache | diff --git a/examples/offline_inference/hunyuan_image3/end2end.py b/examples/offline_inference/hunyuan_image3/end2end.py index f8f92944f..0f41150b6 100644 --- a/examples/offline_inference/hunyuan_image3/end2end.py +++ b/examples/offline_inference/hunyuan_image3/end2end.py @@ -15,42 +15,43 @@ Usage: import argparse import os +from pathlib import Path from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import ( + _TASK_PRESETS, build_prompt_tokens, + resolve_stop_token_ids, ) from vllm_omni.entrypoints.omni import Omni from vllm_omni.inputs.data import OmniPromptType -# task -> (sys_type, bot_task, trigger_tag) -_TASK_PRESETS: dict[str, tuple[str, str | None, str | None]] = { - "t2t": ("en_unified", None, None), - "i2t": ("en_unified", None, None), - "it2i_think": ("en_unified", "think", ""), - "it2i_recaption": ("en_unified", "recaption", ""), - "t2i_think": ("en_unified", "think", ""), - "t2i_recaption": ("en_unified", "recaption", ""), - "t2i_vanilla": ("en_vanilla", "image", None), +# Default deploy configs are absolute so this example works from any cwd. +_REPO_ROOT = Path(__file__).resolve().parents[3] +_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3.yaml") +_DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml") + +_MODALITY_DEFAULT_DEPLOY_CONFIG = { + "text2img": _DEFAULT_DEPLOY_CONFIG, + "img2img": _DEFAULT_DEPLOY_CONFIG, + "img2text": _DEFAULT_AR_DEPLOY_CONFIG, + "text2text": _DEFAULT_AR_DEPLOY_CONFIG, +} + +_MODALITY_MODE = { + "text2img": "text-to-image", + "img2img": "image-editing", + "img2text": "image-to-text", + "text2text": "text-to-text", } -# Modality → prompt_utils task mapping _MODALITY_TASK_MAP = { - "text2img": "t2i_think", - "img2img": "it2i_think", + "text2img": "t2i", + "img2img": "it2i", "img2text": "i2t", "text2text": "t2t", } -# Modality → default stage config -_MODALITY_DEFAULT_CONFIG = { - "text2img": "hunyuan_image3_t2i.yaml", - "img2img": "hunyuan_image3_it2i.yaml", - "img2text": "hunyuan_image3_i2t.yaml", - "text2text": "hunyuan_image3_t2t.yaml", -} - - def parse_args(): parser = argparse.ArgumentParser(description="HunyuanImage-3.0-Instruct end-to-end inference.") parser.add_argument( @@ -94,8 +95,13 @@ def parse_args(): parser.add_argument( "--bot-task", type=str, - default=None, - help="Override prompt task (e.g. it2i_think, t2i_recaption). Default: auto from modality.", + default="auto", + choices=["auto", "think", "recaption", "think_recaption", "vanilla"], + help=( + "Prompt behavior. 'auto' selects the default for the modality; " + "'think' adds ; 'recaption' adds ; " + "'vanilla' uses the t2i pretrain template." + ), ) parser.add_argument( "--sys-type", @@ -105,7 +111,8 @@ def parse_args(): ) # Omni init args - parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom stage config YAML path.") + parser.add_argument("--deploy-config", type=str, default=None, help="Custom deploy YAML path.") + parser.add_argument("--stage-configs-path", type=str, default=None, help="Custom legacy stage config YAML path.") parser.add_argument("--log-stats", action="store_true", default=False) parser.add_argument("--init-timeout", type=int, default=300, help="Initialization timeout in seconds.") parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.") @@ -120,23 +127,44 @@ def main(): args = parse_args() os.makedirs(args.output, exist_ok=True) - # Determine task for prompt formatting - task = args.bot_task or _MODALITY_TASK_MAP[args.modality] + # Determine task for prompt formatting from modality + bot behavior. + task = _MODALITY_TASK_MAP[args.modality] + assert task is not None + bot_task = args.bot_task + if bot_task != "auto": + task = task + "_" + bot_task + if task not in _TASK_PRESETS: + valid_bot_tasks = { + "text2img": ["think", "recaption", "vanilla"], + "img2img": ["think", "recaption"], + "img2text": ["auto"], + "text2text": ["auto"], + }[args.modality] + raise ValueError( + f"--bot-task {bot_task!r} is not supported for {args.modality}. Choose from: {valid_bot_tasks}" + ) - # Determine stage config - stage_configs_path = args.stage_configs_path or _MODALITY_DEFAULT_CONFIG[args.modality] + if args.deploy_config is not None and args.stage_configs_path is not None: + raise ValueError("--deploy-config and --stage-configs-path are mutually exclusive.") + + deploy_config = args.deploy_config + stage_configs_path = args.stage_configs_path + if deploy_config is None and stage_configs_path is None: + deploy_config = _MODALITY_DEFAULT_DEPLOY_CONFIG[args.modality] # Build Omni omni_kwargs = { "model": args.model, "vae_use_tiling": args.vae_use_tiling, - "stage_configs_path": stage_configs_path, "log_stats": args.log_stats, "init_timeout": args.init_timeout, "enforce_eager": args.enforce_eager, } - if args.modality in ("text2img", "img2img"): - omni_kwargs["mode"] = "text-to-image" + if deploy_config is not None: + omni_kwargs["deploy_config"] = deploy_config + else: + omni_kwargs["stage_configs_path"] = stage_configs_path + omni_kwargs["mode"] = _MODALITY_MODE[args.modality] omni = Omni(**omni_kwargs) @@ -199,7 +227,9 @@ def main(): # Override diffusion params if applicable from vllm_omni.inputs.data import OmniDiffusionSamplingParams - for i, sp in enumerate(params_list): + ar_stop_token_ids = resolve_stop_token_ids(task=task, bot_task=bot_task, tokenizer=tokenizer) + assert ar_stop_token_ids is not None + for sp in params_list: if isinstance(sp, OmniDiffusionSamplingParams): sp.num_inference_steps = args.steps sp.guidance_scale = args.guidance_scale @@ -209,13 +239,20 @@ def main(): if args.modality in ("text2img",): sp.height = args.height sp.width = args.width + elif hasattr(sp, "stop_token_ids"): + sp.stop_token_ids = ar_stop_token_ids # Print configuration print(f"\n{'=' * 60}") print("HunyuanImage-3.0 Generation Configuration:") print(f" Model: {args.model}") print(f" Modality: {args.modality}") - print(f" Stage config: {stage_configs_path}") + print(f" Prompt task: {task}") + print(f" Bot task: {bot_task}") + if deploy_config is not None: + print(f" Deploy config: {deploy_config}") + else: + print(f" Stage config: {stage_configs_path}") print(f" Num stages: {omni.num_stages}") if args.modality in ("text2img", "img2img"): print(f" Inference steps: {args.steps}") @@ -236,10 +273,17 @@ def main(): for req_output in omni_outputs: # Text output (AR stage or text-only) ro = getattr(req_output, "request_output", None) + txt = "" if ro and getattr(ro, "outputs", None): txt = "".join(getattr(o, "text", "") or "" for o in ro.outputs) - if txt: - print(f"[Output] Text:\n{txt}") + if not txt: + ar_text = getattr(req_output, "custom_output", {}).get("ar_generated_text") + if isinstance(ar_text, list): + txt = "\n".join(text for text in ar_text if text) + else: + txt = ar_text or "" + if txt: + print(f"[Output] Text:\n{txt}") # Image output (DiT stage) images = getattr(req_output, "images", None) diff --git a/tests/diffusion/models/hunyuan_image3/test_prompt_utils.py b/tests/diffusion/models/hunyuan_image3/test_prompt_utils.py index 501664fe6..f899e5a65 100644 --- a/tests/diffusion/models/hunyuan_image3/test_prompt_utils.py +++ b/tests/diffusion/models/hunyuan_image3/test_prompt_utils.py @@ -24,9 +24,11 @@ import pathlib import pytest from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import ( + HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS, available_tasks, build_prompt, build_prompt_tokens, + resolve_stop_token_ids, ) pytestmark = [pytest.mark.core_model, pytest.mark.cpu] @@ -38,7 +40,7 @@ pytestmark = [pytest.mark.core_model, pytest.mark.cpu] class FakeTokenizer: """Minimal tokenizer stub that records every encode() call. - Returns deterministic ids: special tokens map to small ints (1-4), + Returns deterministic ids from convert_tokens_to_ids while encode() returns one id per character starting at 100. This lets tests both verify segmentation (by inspecting `encode_calls`) and locate substrings inside the returned id list. @@ -49,10 +51,17 @@ class FakeTokenizer: "": 2, "": 3, "": 4, + "<|endoftext|>": 5, + "": 6, + "": 7, + "": 8, + "": 9, + **{f"": 1000 + i for i in range(33)}, } def __init__(self) -> None: self.encode_calls: list[str] = [] + self.eos_token_id = self.SPECIAL["<|endoftext|>"] def convert_tokens_to_ids(self, tok: str) -> int: return self.SPECIAL.get(tok, 0) @@ -75,6 +84,14 @@ def test_available_tasks_covers_all_modalities(): } +def test_resolve_stop_token_ids_uses_answer_for_generation_tasks(): + tok = FakeTokenizer() + + answer_id = HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS[""] + assert resolve_stop_token_ids(task="t2i_think", tokenizer=tok) == [answer_id] + assert resolve_stop_token_ids(task="t2i_recaption", tokenizer=tok) == [answer_id] + + @pytest.mark.parametrize( "task", [ @@ -164,19 +181,24 @@ def test_build_prompt_tokens_segments_each_boundary(): def test_build_prompt_tokens_image_placeholder_present_for_image_tasks(): tok = FakeTokenizer() ids = build_prompt_tokens("hi", tok, task="i2t") - assert ids[0] == 1, "BOS (<|startoftext|>) must be the first token" - assert 2 in ids, " placeholder must be present for i2t/it2i tasks" + assert ids[0] == FakeTokenizer.SPECIAL["<|startoftext|>"], "BOS (<|startoftext|>) must be the first token" + assert FakeTokenizer.SPECIAL[""] in ids, " placeholder must be present for i2t/it2i tasks" def test_build_prompt_tokens_no_image_for_text_only_tasks(): tok = FakeTokenizer() ids = build_prompt_tokens("hi", tok, task="t2t") - assert 2 not in ids, " must NOT appear for text-only tasks" + assert FakeTokenizer.SPECIAL[""] not in ids, " must NOT appear for text-only tasks" @pytest.mark.parametrize( "task,trigger_id", - [("it2i_think", 3), ("t2i_think", 3), ("it2i_recaption", 4), ("t2i_recaption", 4)], + [ + ("it2i_think", FakeTokenizer.SPECIAL[""]), + ("t2i_think", FakeTokenizer.SPECIAL[""]), + ("it2i_recaption", FakeTokenizer.SPECIAL[""]), + ("t2i_recaption", FakeTokenizer.SPECIAL[""]), + ], ) def test_build_prompt_tokens_trigger_is_last_token(task: str, trigger_id: int): """Trigger tag id must be the LAST token (after `Assistant: ` segment).""" @@ -189,7 +211,10 @@ def test_build_prompt_tokens_no_trigger_for_plain_tasks(): """Tasks without trigger_tag (t2t / i2t) must NOT append a trigger id.""" tok = FakeTokenizer() ids = build_prompt_tokens("hi", tok, task="t2t") - assert ids[-1] not in {3, 4} # neither nor + assert ids[-1] not in { + FakeTokenizer.SPECIAL[""], + FakeTokenizer.SPECIAL[""], + } # -------------------- end2end.py wiring guard -------------------- @@ -234,10 +259,16 @@ def test_end2end_routes_through_shared_prompt_utils(): for node in ast.walk(tree): if isinstance(node, ast.ImportFrom) and node.module and node.module.endswith("hunyuan_image3.prompt_utils"): imported_from_prompt_utils.update(alias.name for alias in node.names) - assert "build_prompt_tokens" in imported_from_prompt_utils, ( - "end2end.py must import build_prompt_tokens from " + expected_imports = { + "_TASK_PRESETS", + "build_prompt_tokens", + "resolve_stop_token_ids", + } + assert expected_imports <= imported_from_prompt_utils, ( + "end2end.py must import the HunyuanImage3 prompt and stop-token helpers from " "vllm_omni.diffusion.models.hunyuan_image3.prompt_utils -- the shared " - "helper is the single source of truth for the AR-prefill template." + "module is the single source of truth for the AR-prefill template and " + "bot_task-derived AR stop token ids." ) diff --git a/tests/e2e/offline_inference/test_hunyuanimage3.py b/tests/e2e/offline_inference/test_hunyuanimage3.py index 5b34faa98..ac1cb13cb 100644 --- a/tests/e2e/offline_inference/test_hunyuanimage3.py +++ b/tests/e2e/offline_inference/test_hunyuanimage3.py @@ -1,6 +1,5 @@ # ruff: noqa: E501 from collections.abc import Generator -from pathlib import Path import pytest import torch @@ -9,6 +8,7 @@ from PIL import Image from transformers import CLIPModel, CLIPProcessor from tests.helpers.runtime import OmniRunner +from tests.helpers.stage_config import get_deploy_config_path from vllm_omni import Omni from vllm_omni.inputs.data import OmniDiffusionSamplingParams from vllm_omni.platforms import current_omni_platform @@ -16,8 +16,7 @@ from vllm_omni.platforms import current_omni_platform PROMPT = "A brown and white dog is running on the grass" MODEL_NAME = "tencent/HunyuanImage-3.0" LOCAL_CLIP_PATH = "openai/clip-vit-base-patch32" -REPO_ROOT = Path(__file__).resolve().parents[3] -STAGE_CONFIG_PATH = REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "hunyuan_image3_t2i.yaml" +DEPLOY_CONFIG_PATH = get_deploy_config_path("hunyuan_image3.yaml") pytestmark = [pytest.mark.advanced_model, pytest.mark.diffusion] @@ -274,7 +273,8 @@ def clip_bundle() -> tuple[CLIPModel, CLIPProcessor]: def omni() -> Generator[Omni, None, None]: with OmniRunner( MODEL_NAME, - stage_configs_path=str(STAGE_CONFIG_PATH), + deploy_config=str(DEPLOY_CONFIG_PATH), + mode="text-to-image", ) as runner: yield runner.omni diff --git a/tests/entrypoints/test_utils.py b/tests/entrypoints/test_utils.py index 98ecc8ae5..b52b49d68 100644 --- a/tests/entrypoints/test_utils.py +++ b/tests/entrypoints/test_utils.py @@ -18,6 +18,7 @@ from vllm_omni.entrypoints.utils import ( _filter_dict_like_object, coerce_param_message_types, filter_dataclass_kwargs, + filter_stages, load_and_resolve_stage_configs, load_stage_configs_from_yaml, resolve_model_config_path, @@ -401,6 +402,44 @@ class TestLoadAndResolveStageConfigs: assert stage_configs[1].runtime.num_replicas == 3 assert stage_configs[1].runtime.devices == "1,2,3" + def test_filter_stages_selects_mode_stages_without_mutating_stage_config(self, tmp_path): + config_path = tmp_path / "deploy.yaml" + config_path.write_text( + """modes: + - mode: text-to-text + stages: [0] + - mode: text-to-image + stages: [0, 1] +""", + encoding="utf-8", + ) + stages = [ + create_config( + { + "stage_id": 0, + "runtime": {"requires_multimodal_data": True}, + "final_output": False, + "final_output_type": None, + } + ), + create_config( + { + "stage_id": 1, + "runtime": {"requires_multimodal_data": True}, + "final_output": True, + "final_output_type": "image", + } + ), + ] + + filtered = filter_stages(str(config_path), stages, {"mode": "text-to-text"}) + + assert len(filtered) == 1 + assert filtered[0].stage_id == 0 + assert filtered[0].runtime.requires_multimodal_data is True + assert filtered[0].final_output is False + assert filtered[0].final_output_type is None + class TestLoadStageConfigsFromYaml: """Regression tests for stage-config loading and merging.""" diff --git a/tests/test_config_factory.py b/tests/test_config_factory.py index 7b620ea6e..57313fe3b 100644 --- a/tests/test_config_factory.py +++ b/tests/test_config_factory.py @@ -941,6 +941,34 @@ class TestDeployConfigLoading: assert stages[1].yaml_runtime["devices"] == "1,2" assert stages[1].yaml_runtime["num_replicas"] == 2 + def test_merge_pipeline_deploy_preserves_requires_multimodal_data(self): + from vllm_omni.config.stage_config import ( + DeployConfig, + PipelineConfig, + StageDeployConfig, + StageExecutionType, + StagePipelineConfig, + merge_pipeline_deploy, + ) + + pipeline = PipelineConfig( + model_type="test_mm", + model_arch="TestModel", + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="ar", + execution_type=StageExecutionType.LLM_AR, + requires_multimodal_data=True, + ), + ), + ) + deploy = DeployConfig(async_chunk=False, stages=[StageDeployConfig(stage_id=0)]) + + stages = merge_pipeline_deploy(pipeline, deploy) + + assert stages[0].yaml_runtime["requires_multimodal_data"] is True + class TestQwen3OmniPipeline: def test_registered(self): diff --git a/vllm_omni/config/pipeline_registry.py b/vllm_omni/config/pipeline_registry.py index 1a129cad8..3d44d1bff 100644 --- a/vllm_omni/config/pipeline_registry.py +++ b/vllm_omni/config/pipeline_registry.py @@ -65,6 +65,18 @@ _OMNI_PIPELINES: dict[str, tuple[str, str]] = { "vllm_omni.model_executor.models.glm_image.pipeline", "GLM_IMAGE_PIPELINE", ), + "hunyuan_image3": ( + "vllm_omni.model_executor.models.hunyuan_image3.pipeline", + "HUNYUAN_IMAGE3_PIPELINE", + ), + "hunyuan_image3_ar": ( + "vllm_omni.model_executor.models.hunyuan_image3.pipeline", + "HUNYUAN_IMAGE3_AR_PIPELINE", + ), + "hunyuan_image3_dit": ( + "vllm_omni.model_executor.models.hunyuan_image3.pipeline", + "HUNYUAN_IMAGE3_DIT_PIPELINE", + ), "voxcpm2": ( "vllm_omni.model_executor.models.voxcpm2.pipeline", "VOXCPM2_PIPELINE", diff --git a/vllm_omni/config/stage_config.py b/vllm_omni/config/stage_config.py index 17c703023..dcc4d5ec9 100644 --- a/vllm_omni/config/stage_config.py +++ b/vllm_omni/config/stage_config.py @@ -863,6 +863,7 @@ def merge_pipeline_deploy( if ds.devices is not None: runtime["devices"] = ds.devices runtime["num_replicas"] = ds.num_replicas + runtime["requires_multimodal_data"] = ps.requires_multimodal_data result.append( StageConfig( diff --git a/vllm_omni/deploy/hunyuan_image3.yaml b/vllm_omni/deploy/hunyuan_image3.yaml new file mode 100644 index 000000000..634165cd3 --- /dev/null +++ b/vllm_omni/deploy/hunyuan_image3.yaml @@ -0,0 +1,118 @@ +# HunyuanImage-3.0-Instruct deploy: AR (stage 0) + DiT (stage 1) +# with AR-to-DiT KV reuse. The base CUDA layout uses 2 GPUs for AR +# and 2 GPUs for DiT. +pipeline: hunyuan_image3 +async_chunk: false +trust_remote_code: true + +connectors: + rdma_connector: + name: MooncakeTransferEngineConnector + extra: + host: "auto" + zmq_port: 50051 + protocol: "rdma" + device_name: "" + memory_pool_size: 4294967296 + memory_pool_device: "cpu" + shared_memory_connector: + name: SharedMemoryConnector + extra: + shm_threshold_bytes: 65536 + +stages: + - stage_id: 0 + is_comprehension: false + final_output: true + final_output_type: text + max_num_seqs: 1 + gpu_memory_utilization: 0.95 + enforce_eager: true + max_num_batched_tokens: 32768 + devices: "0,1" + tensor_parallel_size: 2 + hf_overrides: + rope_parameters: + mrope_section: [0, 32, 32] + rope_type: default + omni_kv_config: + need_send_cache: true + output_connectors: + to_stage_1: shared_memory_connector + default_sampling_params: + temperature: 0.0 + top_p: 1 + top_k: -1 + max_tokens: 8192 + detokenize: true + skip_special_tokens: false + + - stage_id: 1 + max_num_seqs: 1 + enforce_eager: true + devices: "2,3" + distributed_executor_backend: "mp" + omni_kv_config: + need_recv_cache: true + parallel_config: + tensor_parallel_size: 2 + enable_expert_parallel: true + input_connectors: + from_stage_0: shared_memory_connector + default_sampling_params: + num_inference_steps: 50 + guidance_scale: 0 + +edges: + - from: 0 + to: 1 + window_size: -1 + max_inflight: 1 + +platforms: + npu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.8 + devices: "0,1,2,3" + tensor_parallel_size: 4 + max_num_batched_tokens: 8192 + - stage_id: 1 + gpu_memory_utilization: 0.65 + devices: "4,5,6,7" + max_num_batched_tokens: 8192 + parallel_config: + tensor_parallel_size: 4 + enable_expert_parallel: true + + xpu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.95 + devices: "0,1,2,3,4,5,6,7" + tensor_parallel_size: 8 + max_num_batched_tokens: 32784 + quantization: fp8 + enable_expert_parallel: true + worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker + default_sampling_params: + max_tokens: 2048 + seed: 42 + repetition_penalty: 1.1 + - stage_id: 1 + gpu_memory_utilization: 0.9 + devices: "0,1,2,3,4,5,6,7" + quantization: fp8 + parallel_config: + pipeline_parallel_size: 1 + data_parallel_size: 1 + tensor_parallel_size: 8 + enable_expert_parallel: true + sequence_parallel_size: 1 + ulysses_degree: 1 + ring_degree: 1 + cfg_parallel_size: 1 + vae_patch_parallel_size: 1 + use_hsdp: false + hsdp_shard_size: -1 + hsdp_replicate_size: 1 diff --git a/vllm_omni/deploy/hunyuan_image3_ar.yaml b/vllm_omni/deploy/hunyuan_image3_ar.yaml new file mode 100644 index 000000000..a59fbfcc9 --- /dev/null +++ b/vllm_omni/deploy/hunyuan_image3_ar.yaml @@ -0,0 +1,47 @@ +# HunyuanImage-3.0-Instruct AR-only deploy. +# +# Use this when AR and DiT are deployed as independent services. This file +# resolves to stage 0 only, avoiding the default two-stage topology. +pipeline: hunyuan_image3_ar +async_chunk: false + +stages: + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.75 + trust_remote_code: true + enforce_eager: true + enable_prefix_caching: false + max_num_batched_tokens: 32768 + devices: "0,1,2,3" + tensor_parallel_size: 4 + hf_overrides: + rope_parameters: + mrope_section: [0, 32, 32] + rope_type: default + default_sampling_params: + temperature: 0.0 + top_p: 0.95 + top_k: 1024 + max_tokens: 1024 + detokenize: true + +platforms: + npu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.75 + devices: "0,1,2,3,4,5,6,7" + tensor_parallel_size: 8 + max_num_batched_tokens: 8192 + + xpu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.95 + devices: "0,1,2,3,4,5,6,7" + tensor_parallel_size: 8 + max_num_batched_tokens: 32784 + quantization: fp8 + enable_expert_parallel: true + worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker diff --git a/vllm_omni/deploy/hunyuan_image3_dit.yaml b/vllm_omni/deploy/hunyuan_image3_dit.yaml new file mode 100644 index 000000000..3c0ba1901 --- /dev/null +++ b/vllm_omni/deploy/hunyuan_image3_dit.yaml @@ -0,0 +1,64 @@ +# HunyuanImage-3.0-Instruct DiT-only deploy. +# +# Use this for standalone DiT/offline execution. This file resolves to one +# diffusion stage with stage_id 0 and does not depend on the AR stage. +pipeline: hunyuan_image3_dit +async_chunk: false + +stages: + - stage_id: 0 + max_num_seqs: 1 + gpu_memory_utilization: 0.9 + enforce_eager: true + devices: "0,1,2,3" + vae_use_slicing: false + vae_use_tiling: false + cache_backend: + cache_config: + enable_cache_dit_summary: false + parallel_config: + pipeline_parallel_size: 1 + data_parallel_size: 1 + tensor_parallel_size: 4 + enable_expert_parallel: true + sequence_parallel_size: 1 + ulysses_degree: 1 + ring_degree: 1 + cfg_parallel_size: 1 + vae_patch_parallel_size: 1 + use_hsdp: false + hsdp_shard_size: -1 + hsdp_replicate_size: 1 + default_sampling_params: + seed: 42 + +platforms: + npu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.65 + devices: "0,1,2,3" + max_num_batched_tokens: 32768 + parallel_config: + tensor_parallel_size: 4 + enable_expert_parallel: true + + xpu: + stages: + - stage_id: 0 + gpu_memory_utilization: 0.9 + devices: "0,1,2,3,4,5,6,7" + quantization: fp8 + parallel_config: + pipeline_parallel_size: 1 + data_parallel_size: 1 + tensor_parallel_size: 8 + enable_expert_parallel: true + sequence_parallel_size: 1 + ulysses_degree: 1 + ring_degree: 1 + cfg_parallel_size: 1 + vae_patch_parallel_size: 1 + use_hsdp: false + hsdp_shard_size: -1 + hsdp_replicate_size: 1 diff --git a/vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py b/vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py index 7a8be0745..5c0ef1635 100644 --- a/vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py +++ b/vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py @@ -1424,7 +1424,11 @@ class HunyuanImage3Pipeline( model_inputs.update(ar_kv_kwargs) outputs = self._generate(**model_inputs, **kwargs) + custom_output = {} + if any(t is not None for t in cot_text_list): + custom_output["ar_generated_text"] = cot_text_list[0] if len(cot_text_list) == 1 else cot_text_list return DiffusionOutput( output=outputs[0], + custom_output=custom_output, stage_durations=self.stage_durations if hasattr(self, "stage_durations") else None, ) diff --git a/vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py b/vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py index 6e8efac31..bfc0146d8 100644 --- a/vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py +++ b/vllm_omni/diffusion/models/hunyuan_image3/prompt_utils.py @@ -17,17 +17,43 @@ canonical mapping for both flows. from __future__ import annotations +from typing import Any + from .system_prompt import get_system_prompt +# HunyuanImage-3.0-Instruct special token ids from tokenizer.json. +# Keep offline AR prompt/stop-token behavior independent of runtime +# tokenizer lookup for these fixed control tokens. +HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS: dict[str, int] = { + "<|endoftext|>": 127957, + "<|startoftext|>": 127958, + "": 128000, + "": 128001, + "": 128006, + "": 128010, + "": 128018, + "": 128019, + "": 128023, + "": 128024, + "": 128025, + "": 128026, + "": 128037, + "": 128044, + "": 128076, + "": 130103, + "": 130106, +} + # task -> (sys_type, bot_task, trigger_tag) _TASK_PRESETS: dict[str, tuple[str, str | None, str | None]] = { "t2t": ("en_unified", None, None), "i2t": ("en_unified", None, None), "it2i_think": ("en_unified", "think", ""), "it2i_recaption": ("en_unified", "recaption", ""), + "t2i": ("en_unified", "image", None), + "t2i_vanilla": ("en_vanilla", "image", None), "t2i_think": ("en_unified", "think", ""), "t2i_recaption": ("en_unified", "recaption", ""), - "t2i_vanilla": ("en_vanilla", "image", None), } @@ -36,6 +62,14 @@ def available_tasks() -> list[str]: return sorted(_TASK_PRESETS) +def resolve_stop_token_ids( + task: str = "it2i_think", + bot_task: str = "think", + tokenizer: Any | None = None, +): + return [HUNYUAN_IMAGE3_SPECIAL_TOKEN_IDS[""]] + + def build_prompt( user_prompt: str, task: str = "it2i_think", @@ -149,4 +183,4 @@ def build_prompt_tokens( return ids -__all__ = ["build_prompt", "build_prompt_tokens", "available_tasks"] +__all__ = ["build_prompt", "build_prompt_tokens", "resolve_stop_token_ids", _TASK_PRESETS] diff --git a/vllm_omni/entrypoints/cli/serve.py b/vllm_omni/entrypoints/cli/serve.py index 540d5c0cf..b4293d59f 100644 --- a/vllm_omni/entrypoints/cli/serve.py +++ b/vllm_omni/entrypoints/cli/serve.py @@ -593,6 +593,7 @@ def run_headless(args: argparse.Namespace) -> None: model, args_dict.get("stage_configs_path"), args_dict, + deploy_config_path=args_dict.get("deploy_config"), ) # Locate the stage config that matches stage_id. diff --git a/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py b/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py index 88494eda4..1e057a71e 100644 --- a/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py +++ b/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py @@ -1517,14 +1517,9 @@ class HunyuanImage3ForConditionalGeneration(nn.Module, SupportsMultiModal, Suppo # For comprehension mode, block image generation tokens but allow # text structure tokens (, , etc.) so the model can - # follow its natural generation pattern. The yaml stop_token_ids - # for i2t/t2t now includes (128024) so the AR-only output - # terminates after the analysis section, matching HF's - # `bot_task="think"` behavior. Without that stop, the model - # continues into a recaption section even in comprehension mode - # (the stage-transition processor only fires in generation mode, - # but the instruct-tuned model writes recaption on its own from - # internal habit). + # follow its natural generation pattern. Runtime sampling params + # decide stop tokens from the active bot_task, matching the official + # HunyuanImage3 generation path without hard-coded YAML token ids. self._blocked_token_ids: set[int] = set() if self._is_comprehension: self._blocked_token_ids.update( diff --git a/vllm_omni/model_executor/models/hunyuan_image3/pipeline.py b/vllm_omni/model_executor/models/hunyuan_image3/pipeline.py new file mode 100644 index 000000000..3ff53af62 --- /dev/null +++ b/vllm_omni/model_executor/models/hunyuan_image3/pipeline.py @@ -0,0 +1,87 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""HunyuanImage3 pipeline topology.""" + +from vllm_omni.config.stage_config import ( + PipelineConfig, + StageExecutionType, + StagePipelineConfig, +) + +_HUNYUAN_IMAGE3_HF_ARCHS = ( + "HunyuanImage3ForConditionalGeneration", + "HunyuanImage3ForCausalMM", +) +_HUNYUAN_IMAGE3_MODEL_ARCH = "HunyuanImage3ForCausalMM" +_HUNYUAN_IMAGE3_INPUT_PROCESSOR = "vllm_omni.model_executor.stage_input_processors.hunyuan_image3" + + +HUNYUAN_IMAGE3_PIPELINE = PipelineConfig( + model_type="hunyuan_image3", + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + hf_architectures=_HUNYUAN_IMAGE3_HF_ARCHS, + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="AR", + execution_type=StageExecutionType.LLM_AR, + input_sources=(), + final_output=False, + owns_tokenizer=False, + requires_multimodal_data=True, + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + engine_output_type="latent", + ), + StagePipelineConfig( + stage_id=1, + model_stage="dit", + execution_type=StageExecutionType.DIFFUSION, + input_sources=(0,), + final_output=True, + final_output_type="image", + requires_multimodal_data=True, + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + custom_process_input_func=f"{_HUNYUAN_IMAGE3_INPUT_PROCESSOR}.ar2diffusion", + ), + ), +) + + +HUNYUAN_IMAGE3_AR_PIPELINE = PipelineConfig( + model_type="hunyuan_image3_ar", + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + hf_architectures=(), + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="AR", + execution_type=StageExecutionType.LLM_AR, + input_sources=(), + final_output=True, + final_output_type="text", + owns_tokenizer=False, + requires_multimodal_data=True, + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + engine_output_type="latent", + ), + ), +) + + +HUNYUAN_IMAGE3_DIT_PIPELINE = PipelineConfig( + model_type="hunyuan_image3_dit", + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + hf_architectures=(), + stages=( + StagePipelineConfig( + stage_id=0, + model_stage="dit", + execution_type=StageExecutionType.DIFFUSION, + input_sources=(), + final_output=True, + final_output_type="image", + requires_multimodal_data=True, + model_arch=_HUNYUAN_IMAGE3_MODEL_ARCH, + ), + ), +) diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_i2t.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_i2t.yaml deleted file mode 100644 index 0614a9f11..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_i2t.yaml +++ /dev/null @@ -1,41 +0,0 @@ -# Stage config for HunyuanImage-3.0 Image-to-Text (I2T / image understanding). -# Single LLM stage: AR model reads image + text prompt, generates text output. - -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0,1,2,3" - max_batch_size: 1 - requires_multimodal_data: true - engine_args: - model_stage: AR - max_num_seqs: 1 - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.95 - enforce_eager: true - trust_remote_code: true - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - is_comprehension: true - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 0.95 - top_k: 1024 - max_tokens: 2048 - stop_token_ids: [127957, 128024, 128026] # <|endoftext|>, , - detokenize: True - -runtime: - enabled: true diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i.yaml deleted file mode 100644 index 315116973..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i.yaml +++ /dev/null @@ -1,72 +0,0 @@ -# Stage config for HunyuanImage-3.0 Image+Text-to-Image (image editing). -# Stage 0: AR (HunyuanImage3ForConditionalGeneration) — reads (image, text), emits latent tokens -# Stage 1: Diffusion (HunyuanImage3Pipeline / DiT + VAE) — denoise + decode latents → image - -stage_args: - # Stage 0: AR Model - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0,1,2,3" - max_batch_size: 1 - requires_multimodal_data: true # AR needs the original image - engine_args: - model_stage: AR - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.95 - enforce_eager: true - trust_remote_code: true - engine_output_type: latent # AR outputs latent for DiT - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - is_comprehension: false # Generation task, not comprehension - final_output: false # AR is not the final output - default_sampling_params: - temperature: 0.6 - top_p: 0.95 - top_k: 1024 - max_tokens: 4096 - stop_token_ids: [127957] # <|endoftext|> - detokenize: true # DiT bridge consumes ar_generated_text; let the AR engine produce it - - # Stage 1: Diffusion (DiT + VAE) - # Receives latents from AR stage, performs denoising + VAE decode - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "4,5,6,7" - max_batch_size: 1 - requires_multimodal_data: true # May need condition images - engine_args: - model_stage: dit - model_arch: HunyuanImage3ForCausalMM - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - parallel_config: - tensor_parallel_size: 4 - enable_expert_parallel: true - engine_input_source: [0] # Input from AR stage - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.hunyuan_image3.ar2diffusion - final_output: true - final_output_type: image - default_sampling_params: - num_inference_steps: 50 - guidance_scale: 2.5 - -# Top-level runtime config -runtime: - enabled: true - edges: - - from: 0 # AR → Diffusion - to: 1 diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i_kv_reuse.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i_kv_reuse.yaml deleted file mode 100644 index 23c3bbbb2..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_it2i_kv_reuse.yaml +++ /dev/null @@ -1,89 +0,0 @@ -stage_args: - # Stage 0: AR Model - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0,1" - max_batch_size: 1 - requires_multimodal_data: true # AR needs the original image - engine_args: - model_stage: AR - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.95 - enforce_eager: true - trust_remote_code: true - engine_output_type: latent # AR outputs latent for DiT - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 2 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - omni_kv_config: - need_send_cache: true - is_comprehension: false # Generation task, not comprehension - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 1 - top_k: -1 - max_tokens: 8192 - stop_token_ids: [128025] # - detokenize: true # DiT bridge consumes ar_generated_text; let the AR engine produce it - skip_special_tokens: False - output_connectors: - to_stage_1: rdma_connector - - # Stage 1: Diffusion (DiT + VAE) - # Receives latents from AR stage, performs denoising + VAE decode - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "2,3" - max_batch_size: 1 - requires_multimodal_data: true # May need condition images - engine_args: - model_stage: dit - model_arch: HunyuanImage3ForCausalMM - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - omni_kv_config: - need_recv_cache: true # Receive AR KV cache from stage 0 - parallel_config: - tensor_parallel_size: 2 - enable_expert_parallel: true - engine_input_source: [0] # Input from AR stage - custom_process_input_func: vllm_omni.model_executor.stage_input_processors.hunyuan_image3.ar2diffusion - final_output: true - final_output_type: image - default_sampling_params: - num_inference_steps: 50 - guidance_scale: 0 - input_connectors: - from_stage_0: rdma_connector - - -# Top-level runtime config -runtime: - enabled: true - connectors: - rdma_connector: - name: MooncakeTransferEngineConnector - extra: - host: "auto" - zmq_port: 50051 - protocol: "rdma" - device_name: "" - memory_pool_size: 4294967296 - memory_pool_device: "cpu" - edges: - - from: 0 # AR → Diffusion - to: 1 diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe.yaml deleted file mode 100644 index f0797c632..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe.yaml +++ /dev/null @@ -1,96 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 with AR→DiT KV reuse. -# Stage 0: AR Model (vLLM implementation) -# Stage 1: DiT Model (diffusion) -# -# text-to-image flow: AR (stage 0) → KV transfer → DiT (stage 1) -# image-to-text flow: AR (stage 0) only -# -# Compared to hunyuan_image3_t2i.yaml, this config: -# 1. Enables both stages [0, 1] for text-to-image (AR prefill + DiT denoising) -# 2. Adds omni_kv_config to send/receive KV cache between stages - -# The following config has been verified on 8x L40S-48G GPU (4 for AR + 4 for DiT). -stage_args: - - stage_id: 0 - stage_type: llm # Use llm stage type for AR stages - runtime: - process: true # Run this stage in a separate process - devices: "0,1,2,3" # AR stage uses GPU 0-3 - engine_args: - model_stage: AR - max_num_seqs: 1 - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.9 - enforce_eager: true # Now we only support eager mode - trust_remote_code: true - engine_output_type: latent - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - omni_kv_config: - need_send_cache: true - kv_transfer_criteria: - type: prefill_finished # Send KV cache after AR prefill completes - is_comprehension: true - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - max_tokens: 2048 - seed: 42 - detokenize: True - repetition_penalty: 1.1 - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "4,5,6,7" # DiT stage uses GPU 4-7 - max_batch_size: 1 - engine_args: - model_stage: diffusion - enforce_eager: true - distributed_executor_backend: "mp" - vae_use_slicing: false - vae_use_tiling: false - cache_backend: null - cache_config: null - enable_cache_dit_summary: false - omni_kv_config: - need_recv_cache: true # Receive AR KV cache from stage 0 - parallel_config: - pipeline_parallel_size: 1 - data_parallel_size: 1 - tensor_parallel_size: 4 - enable_expert_parallel: false - sequence_parallel_size: 1 - ulysses_degree: 1 - ring_degree: 1 - cfg_parallel_size: 1 - vae_patch_parallel_size: 1 - use_hsdp: false - hsdp_shard_size: -1 - hsdp_replicate_size: 1 - engine_input_source: [0] # Receive input (including KV) from stage 0 - final_output: true - final_output_type: image - -# Top-level runtime config: windows, edges, and connectors -runtime: - enabled: true - defaults: - window_size: -1 # Trigger downstream only after full upstream completion - max_inflight: 1 # Process serially within each stage - - edges: - - from: 0 - to: 1 - window_size: -1 diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe_dit_2gpu_fp8.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe_dit_2gpu_fp8.yaml deleted file mode 100644 index 586b601bc..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_moe_dit_2gpu_fp8.yaml +++ /dev/null @@ -1,32 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 DiT with FP8 online quantization. -# The following config is for 2x H200 GPU. - -# Stage 0: Diffusion (DiT + VAE) -# This stage receives noise and timesteps and performs denoising + VAE decode -stage_args: - - stage_id: 0 - stage_type: diffusion - runtime: - devices: "0,1" - max_batch_size: 1 - engine_args: - model_stage: dit - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - quantization: "fp8" - parallel_config: - tensor_parallel_size: 2 - enable_expert_parallel: true - omni_kv_config: - need_recv_cache: true - - final_output: true - final_output_type: image - is_comprehension: false - default_sampling_params: - seed: 42 - -# Runtime edges -runtime: - enabled: true diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i.yaml deleted file mode 100644 index 1d8c7f481..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i.yaml +++ /dev/null @@ -1,31 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 DiT. -# The following config has been verified on 4x H20 GPU. - -# Stage 0: Diffusion (DiT + VAE) -# This stage receives noise and timesteps and performs denoising + VAE decode -stage_args: - - stage_id: 0 - stage_type: diffusion - runtime: - devices: "0,1,2,3" - engine_args: - max_num_seqs: 1 - model_stage: dit - enforce_eager: true - trust_remote_code: true - distributed_executor_backend: "mp" - parallel_config: - tensor_parallel_size: 4 - enable_expert_parallel: true - omni_kv_config: - need_recv_cache: true - - final_output: true - final_output_type: image - is_comprehension: false - default_sampling_params: - seed: 42 - -# Runtime edges -runtime: - enabled: true diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i_2gpu.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i_2gpu.yaml deleted file mode 100644 index 41ed74ba6..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2i_2gpu.yaml +++ /dev/null @@ -1,41 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 on 2 GPUs with FP8. -# Stage 0: AR Model (vLLM implementation) - -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0,1" - engine_args: - model_stage: AR - max_num_seqs: 1 - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.9 - enforce_eager: true - trust_remote_code: true - engine_output_type: latent - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 2 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - is_comprehension: true - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - max_tokens: 2048 - seed: 42 - detokenize: True - repetition_penalty: 1.1 - -runtime: - enabled: true diff --git a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2t.yaml b/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2t.yaml deleted file mode 100644 index c9daa5e5f..000000000 --- a/vllm_omni/model_executor/stage_configs/hunyuan_image3_t2t.yaml +++ /dev/null @@ -1,42 +0,0 @@ -# Stage config for HunyuanImage-3.0 Text-to-Text (T2T / pure text generation). -# Single LLM stage: AR model reads text prompt only, generates text output. -# Sampling params aligned with official generation_config.json. - -stage_args: - - stage_id: 0 - stage_type: llm - runtime: - process: true - devices: "0,1,2,3" - max_batch_size: 1 - requires_multimodal_data: false - engine_args: - model_stage: AR - max_num_seqs: 1 - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.95 - enforce_eager: true - trust_remote_code: true - enable_prefix_caching: false - max_num_batched_tokens: 32768 - tensor_parallel_size: 4 - pipeline_parallel_size: 1 - hf_overrides: - rope_parameters: - mrope_section: [0, 32, 32] - rope_type: default - is_comprehension: true - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 0.95 - top_k: 1024 - max_tokens: 2048 - stop_token_ids: [127957, 128024, 128026] # <|endoftext|>, , - detokenize: True - -runtime: - enabled: true diff --git a/vllm_omni/platforms/npu/stage_configs/hunyuan_image3_t2i.yaml b/vllm_omni/platforms/npu/stage_configs/hunyuan_image3_t2i.yaml deleted file mode 100644 index 0fd03949d..000000000 --- a/vllm_omni/platforms/npu/stage_configs/hunyuan_image3_t2i.yaml +++ /dev/null @@ -1,35 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 DiT on NPU. -# The following config has been verified on 8x A3-64G NPUs. - -# Stage 0: Diffusion (DiT + VAE) -# This stage receives noise and timesteps and performs denoising + VAE decode. -stage_args: - - stage_id: 0 - stage_type: diffusion - runtime: - devices: "0,1,2,3,4,5,6,7" - engine_args: - max_num_seqs: 1 - model_stage: dit - gpu_memory_utilization: 0.65 - enforce_eager: true - trust_remote_code: true - engine_output_type: image - distributed_executor_backend: "mp" - enable_prefix_caching: false - max_num_batched_tokens: 32768 - parallel_config: - tensor_parallel_size: 8 - enable_expert_parallel: true - omni_kv_config: - need_recv_cache: true - - final_output: true - final_output_type: image - is_comprehension: false - default_sampling_params: - seed: 42 - -# Runtime defaults -runtime: - enabled: true diff --git a/vllm_omni/platforms/xpu/stage_configs/hunyuan_image3_t2i.yaml b/vllm_omni/platforms/xpu/stage_configs/hunyuan_image3_t2i.yaml deleted file mode 100644 index 4e0005f82..000000000 --- a/vllm_omni/platforms/xpu/stage_configs/hunyuan_image3_t2i.yaml +++ /dev/null @@ -1,80 +0,0 @@ -# Stage config for running Hunyuan-Image3.0 with architecture of OmniLLM. -# Stage 0: AR Model (vLLM implementation) - -# The following config has been verified on 8x Max 1550 GPU. -modes: - - mode: text-to-image - stages: [1] - - mode: image-to-text - stages: [0] -stage_args: - - stage_id: 0 - stage_type: llm # Use llm stage type to launch OmniLLM - runtime: - process: true # Run this stage in a separate process - devices: "0,1,2,3,4,5,6,7" # Visible devices for this stage - max_batch_size: 1 - engine_args: - model_stage: AR - model_arch: HunyuanImage3ForCausalMM - worker_cls: vllm_omni.platforms.xpu.worker.xpu_ar_worker.XPUARWorker - scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler - gpu_memory_utilization: 0.95 - enforce_eager: true # Now we only support eager mode - trust_remote_code: true - engine_output_type: latent - enable_prefix_caching: false - max_num_batched_tokens: 32784 - tensor_parallel_size: 8 - pipeline_parallel_size: 1 - enable_expert_parallel: true - quantization: "fp8" - is_comprehension: true - final_output: true - final_output_type: text - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - max_tokens: 2048 - seed: 42 - detokenize: True - repetition_penalty: 1.1 - - stage_id: 1 - stage_type: diffusion - runtime: - process: true - devices: "0,1,2,3,4,5,6,7" - max_batch_size: 1 - engine_args: - model_stage: diffusion - gpu_memory_utilization: 0.9 - enforce_eager: true - engine_output_type: image - distributed_executor_backend: "mp" - enable_prefix_caching: false - vae_use_slicing: false - vae_use_tiling: false - cache_backend: null - cache_config: null - enable_cache_dit_summary: false - quantization: "fp8" - parallel_config: - pipeline_parallel_size: 1 - data_parallel_size: 1 - tensor_parallel_size: 8 - enable_expert_parallel: true - sequence_parallel_size: 1 - ulysses_degree: 1 - ring_degree: 1 - cfg_parallel_size: 1 - vae_patch_parallel_size: 1 - use_hsdp: false - hsdp_shard_size: -1 - hsdp_replicate_size: 1 - final_output: true - final_output_type: image - -# Top-level runtime config (concise): default windows and stage edges -runtime: - enabled: true