[Example] Migrate HunyuanImage-3.0 to model_extras + shared task examples (#5559)

Signed-off-by: suyanli220 <suyanli220@gmail.com>
Signed-off-by: suyan.li <suyan.li@bytedance.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: suyan.li <suyan.li@bytedance.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
authored and GitHub committed 2026-09-11 16:47:25 +08:00
1 parent 96360ccb00
commit a7ab91fb85
17 files changed
+1627 -544

No files matched your search

@@ -1,154 +0,0 @@
# HunyuanImage-3.0-Instruct
This example runs HunyuanImage-3.0-Instruct offline with the unified deploy
YAMLs under `vllm_omni/deploy/`.
## Deploy Configs
| File | Topology | Default use |
| :--- | :--- | :--- |
| `vllm_omni/deploy/hunyuan_image_3_moe.yaml` | AR + DiT | Default for `text2img` and `img2img`. |
| `vllm_omni/deploy/hunyuan_image3_ar.yaml` | AR only | Default for `img2text` and `text2text`. |
| `vllm_omni/deploy/hunyuan_image3_dit.yaml` | DiT only | Standalone diffusion stage. Pass it explicitly with `--deploy-config`. |
The example chooses a deploy config automatically when `--deploy-config` is
omitted:
| `--modality` | `mode` passed to Omni | Default deploy |
| :--- | :--- | :--- |
| `text2img` | `text-to-image` | `hunyuan_image_3_moe.yaml` |
| `img2img` | `image-editing` | `hunyuan_image_3_moe.yaml` |
| `img2text` | `image-to-text` | `hunyuan_image3_ar.yaml` |
| `text2text` | `text-to-text` | `hunyuan_image3_ar.yaml` |
`--modality` is an offline example convenience flag. It maps to the internal
`mode` argument passed to `Omni(...)` by this script. HunyuanImage3 uses
separate deploy YAMLs for AR + DiT, AR-only, and DiT-only topologies, so the
stage topology is selected by the deploy file rather than by YAML mode
overrides.
Online serving does not expose a `--modality` flag or accept `mode` as an API
request field. Choose the deploy topology when starting the server with
`--deploy-config`, then use the OpenAI-compatible endpoint and request shape for
the scenario. The `modalities` request field is used by the chat completions
path; the image endpoints infer the image task from the endpoint and payload.
| Online scenario | Server deploy | Request |
| :--- | :--- | :--- |
| Text to image | `--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml` | `POST /v1/images/generations`, or `POST /v1/chat/completions` with `"modalities": ["image"]`. |
| Image editing | `--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml` | `POST /v1/images/edits`. |
| Image/text to text | `--deploy-config vllm_omni/deploy/hunyuan_image3_ar.yaml` | `POST /v1/chat/completions` for text output, for example with `"modalities": ["text"]`. |
| DiT-only image generation | `--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml` | `POST /v1/images/generations`. |
## Run Examples
Text to image, using the default AR + DiT deploy:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--prompts "A cute cat sitting on a windowsill watching the sunset"
```
Image editing, using the default AR + DiT deploy:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality img2img \
--image-path /path/to/image.png \
--prompts "Make the petals neon pink"
```
Image to text, using the AR-only deploy:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality img2text \
--image-path /path/to/image.jpg \
--prompts "Describe the content of the picture."
```
Text to text, using the AR-only deploy:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2text \
--prompts "What is the capital of France?"
```
Standalone DiT, using the DiT-only deploy explicitly:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \
--prompts "A cinematic portrait of an astronaut in a greenhouse"
```
Override the default full AR + DiT deploy explicitly:
```bash
python examples/offline_inference/hunyuan_image3/end2end.py \
--model tencent/HunyuanImage-3.0-Instruct \
--modality text2img \
--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml \
--prompts "A cute cat"
```
## Additional Config
You can pass diffusion worker `additional_config` from the offline example as a JSON object.
This maps to the upstream vLLM `VllmConfig.additional_config` platform extension field:
https://docs.vllm.ai/en/stable/api/vllm/config/#vllm.config.VllmConfig.additional_config
```bash
python end2end.py --modality text2img \
--prompts "A cute cat" \
--additional-config '{"torchair_graph_config":{"enabled":true}}'
```
## Key Arguments
| Argument | Description |
| :--- | :--- |
| `--deploy-config` | Optional path for unified deploy YAML overrides. |
| `--additional-config` | JSON object forwarded to diffusion worker `additional_config`. |
| `--modality` | Offline-only convenience flag. One of `text2img`, `img2img`, `img2text`, `text2text`. It selects prompt formatting, internal `mode`, and default deploy config for this script. Online serving uses `--deploy-config` plus the endpoint and, for chat completions, request `modalities` instead. |
| `--steps` | Number of diffusion inference steps for image generation. |
| `--guidance-scale` | Classifier-free guidance scale for image generation. |
| `--height`, `--width` | Output image size for `text2img`. |
| `--bot-task` | Override prompt mode. `none`, `think`, `recaption`, `think_recaption`, or `vanilla`. |
| `--sys-type` | Override the system prompt type, for example `en_unified` or `en_vanilla`. |
| `--vae-use-tiling` | Enable VAE tiling for memory reduction. |
## Notes
- `hunyuan_image3_ar.yaml` is a 4-card AR-only text/comprehension deploy.
- `hunyuan_image3_dit.yaml` is a single-stage DiT deploy with `stage_id: 0`.
- Platform-specific overrides are included in the unified deploy YAMLs.
## Prompt Format
HunyuanImage-3.0-Instruct uses an instruct chat template:
```text
<|startoftext|>{system_prompt}
User: {<img>?}{user_prompt}
Assistant: {trigger_tag?}
```
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline).
- Trigger tags: `<think>` for CoT and `<recaption>` for recaptioning, placed after `Assistant: `.
- System prompt: Auto-selected from `task` and `bot_task`.
- `bot_task='vanilla'` with `task='t2i'` uses the bare pretrain template.
The shared `vllm_omni.diffusion.models.hunyuan_image3.prompt_utils.build_prompt_tokens()`
helper handles segment-by-segment tokenization and matches HF `apply_chat_template`.
@@ -1,321 +0,0 @@
"""
HunyuanImage-3.0-Instruct unified end-to-end inference script.
"""
import argparse
import json
import os
from pathlib import Path
from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import (
MAX_IMAGES_PER_REQUEST,
build_prompt_tokens,
resolve_stop_token_ids,
resolve_sys_type,
)
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniPromptType
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image_3_moe.yaml")
_DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml")
_MODALITY_TASK_MAP: dict[str, tuple[str, str | None]] = {
"text2img": ("t2i", "think"),
"img2img": ("it2i", "think"),
"img2text": ("i2t", None),
"text2text": ("t2t", None),
}
_MODALITY_DEFAULT_DEPLOY_CONFIG = {
"text2img": _DEFAULT_DEPLOY_CONFIG,
"img2img": _DEFAULT_DEPLOY_CONFIG,
"img2text": _DEFAULT_AR_DEPLOY_CONFIG,
"text2text": _DEFAULT_AR_DEPLOY_CONFIG,
}
_MODALITY_MODE = {
"text2img": "text-to-image",
"img2img": "image-editing",
"img2text": "image-to-text",
"text2text": "text-to-text",
}
def parse_args():
parser = argparse.ArgumentParser(description="HunyuanImage-3.0-Instruct end-to-end inference.")
parser.add_argument("--model", default="tencent/HunyuanImage-3.0-Instruct", help="Model name or local path.")
parser.add_argument(
"--modality",
default="text2img",
choices=list(_MODALITY_TASK_MAP),
)
parser.add_argument("--prompts", nargs="+", default=None, help="Input text prompts.")
parser.add_argument(
"--image-path",
type=str,
default=None,
help="Input image path(s) for img2img/img2text. Comma-separated for multi-image (up to 3).",
)
parser.add_argument("--output", type=str, default=".", help="Output directory to save results.")
parser.add_argument("--steps", type=int, default=50, help="Number of inference steps.")
parser.add_argument("--guidance-scale", type=float, default=5.0, help="Classifier-free guidance scale.")
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
parser.add_argument("--height", type=int, default=None, help="Output image height.")
parser.add_argument("--width", type=int, default=None, help="Output image width.")
parser.add_argument("--vae-use-tiling", action="store_true", help="Enable VAE tiling.")
parser.add_argument(
"--bot-task",
type=str,
default=None,
choices=["none", "think", "recaption", "think_recaption", "vanilla"],
help="Override prompt mode. Default: auto from --modality.",
)
parser.add_argument("--sys-type", type=str, default=None, help="Override system prompt type.")
parser.add_argument("--deploy-config", type=str, default=None, help="Custom deploy YAML path.")
parser.add_argument(
"--stream",
action="store_true",
default=False,
help="Enable streaming CoT display; print AR text token-by-token in real time.",
)
parser.add_argument("--log-stats", action="store_true", default=False)
parser.add_argument("--init-timeout", type=int, default=300, help="Initialization timeout in seconds.")
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
parser.add_argument(
"--diffusion-kv-cache-dtype",
type=str,
default=None,
help="Diffusion attention KV cache dtype, for example 'fp8'. Separate from vLLM --kv-cache-dtype.",
)
parser.add_argument(
"--diffusion-kv-cache-skip-steps",
type=str,
default=None,
help="Denoising step selector to keep diffusion KV cache in native dtype, for example '0,1,4-6'.",
)
parser.add_argument(
"--diffusion-kv-cache-skip-layers",
type=str,
default=None,
help="Transformer layer selector to keep diffusion KV cache in native dtype, for example '0-2,10'.",
)
parser.add_argument(
"--additional-config",
type=str,
default=None,
help=(
"JSON object forwarded to Omni/additional_config, for example "
'\'{"torchair_graph_config":{"enabled":true}}\'. Different platforms may support different '
"configs. Make sure the configs are valid for the platform you are using. "
"Contents must be hashable."
),
)
return parser.parse_args()
def parse_additional_config(raw_value: str | None) -> dict | None:
"""Parse a JSON string into an additional_config mapping."""
if raw_value is None:
return None
try:
additional_config = json.loads(raw_value)
except json.JSONDecodeError as exc:
raise ValueError(f"Invalid --additional-config JSON: {exc}") from exc
if additional_config is None:
return None
if not isinstance(additional_config, dict):
raise ValueError(f"--additional-config must decode to a JSON object, got {type(additional_config).__name__}")
return additional_config
def main():
args = parse_args()
os.makedirs(args.output, exist_ok=True)
additional_config = parse_additional_config(args.additional_config)
task, default_bot_task = _MODALITY_TASK_MAP[args.modality]
if args.bot_task is None:
bot_task: str | None = default_bot_task
elif args.bot_task == "none":
bot_task = None
else:
bot_task = args.bot_task
deploy_config = args.deploy_config or _MODALITY_DEFAULT_DEPLOY_CONFIG[args.modality]
omni_kwargs = {
"model": args.model,
"vae_use_tiling": args.vae_use_tiling,
"log_stats": args.log_stats,
"init_timeout": args.init_timeout,
"enforce_eager": args.enforce_eager,
"mode": _MODALITY_MODE[args.modality],
"diffusion_kv_cache_dtype": args.diffusion_kv_cache_dtype,
"diffusion_kv_cache_skip_steps": args.diffusion_kv_cache_skip_steps,
"diffusion_kv_cache_skip_layers": args.diffusion_kv_cache_skip_layers,
}
if additional_config is not None:
omni_kwargs["additional_config"] = additional_config
omni_kwargs["deploy_config"] = deploy_config
omni = Omni(**omni_kwargs)
prompts = args.prompts or ["A cute cat"]
input_images: list = []
if args.modality in ("img2img", "img2text"):
if not args.image_path:
raise ValueError(f"--image-path required for {args.modality}, got: {args.image_path}")
from PIL import Image
image_paths = [p.strip() for p in args.image_path.split(",") if p.strip()]
if len(image_paths) > MAX_IMAGES_PER_REQUEST:
raise ValueError(
f"--image-path accepts at most {MAX_IMAGES_PER_REQUEST} images for "
f"HunyuanImage-3.0 IT2I, got {len(image_paths)}: {args.image_path}"
)
for image_path in image_paths:
if not os.path.exists(image_path):
raise ValueError(f"Image path does not exist: {image_path}")
input_images.append(Image.open(image_path).convert("RGB"))
if not input_images:
raise ValueError(f"--image-path produced no usable paths: {args.image_path!r}")
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
mm_image_payload = (input_images[0] if len(input_images) == 1 else input_images) if input_images else None
formatted_prompts: list[OmniPromptType] = []
for prompt in prompts:
build_kwargs: dict = {"task": task, "bot_task": bot_task, "sys_type": args.sys_type}
if input_images:
build_kwargs["num_images"] = len(input_images)
result = build_prompt_tokens(prompt, tokenizer, **build_kwargs)
token_ids = result.token_ids
effective_sys_type = args.sys_type or resolve_sys_type(bot_task)
prompt_dict: dict = {
"prompt_token_ids": token_ids,
"prompt": prompt,
"use_system_prompt": effective_sys_type,
}
if args.modality == "text2img":
prompt_dict["modalities"] = ["image"]
elif args.modality == "img2img":
prompt_dict["modalities"] = ["image"]
prompt_dict["multi_modal_data"] = {"image": mm_image_payload}
prompt_dict["height"] = input_images[0].height
prompt_dict["width"] = input_images[0].width
elif args.modality == "img2text":
prompt_dict["modalities"] = ["text"]
prompt_dict["multi_modal_data"] = {"image": mm_image_payload}
else:
prompt_dict["modalities"] = ["text"]
formatted_prompts.append(prompt_dict)
from vllm.sampling_params import RequestOutputKind
params_list = list(omni.default_sampling_params_list)
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
if (args.height is None) != (args.width is None):
raise ValueError("--height and --width must both be specified or both omitted.")
user_specified_size = args.height is not None and args.width is not None
if args.modality in ("img2text", "text2text"):
ar_image_size = "auto"
elif user_specified_size:
ar_image_size = f"{args.width}x{args.height}"
else:
ar_image_size = None
ar_stop_token_ids = resolve_stop_token_ids(
task=task, bot_task=bot_task, tokenizer=tokenizer, image_size=ar_image_size
)
print(
f"[AR Config] task={task}, bot_task={bot_task}, image_size={ar_image_size}, stop_token_ids={ar_stop_token_ids}"
)
for sp in params_list:
if isinstance(sp, OmniDiffusionSamplingParams):
sp.num_inference_steps = args.steps
sp.guidance_scale = args.guidance_scale
sp.guidance_scale_provided = True
if args.seed is not None:
sp.seed = args.seed
if args.modality == "text2img":
sp.height = args.height
sp.width = args.width
elif hasattr(sp, "stop_token_ids"):
sp.stop_token_ids = ar_stop_token_ids
# When --stream is set, request DELTA output from the AR stage
# so we can display CoT text token-by-token in real time.
if args.stream and hasattr(sp, "output_kind"):
sp.output_kind = RequestOutputKind.DELTA
print(f"\n{'=' * 60}")
print("HunyuanImage-3.0 Generation Configuration:")
print(f" Model: {args.model}")
print(f" Modality: {args.modality}")
print(f" Prompt task: {task}")
print(f" Bot task: {bot_task}")
print(f" Deploy config: {deploy_config}")
print(f" Num stages: {omni.num_stages}")
if args.modality in ("text2img", "img2img"):
print(f" Inference steps: {args.steps}")
print(f" Guidance scale: {args.guidance_scale}")
print(f" Seed: {args.seed}")
print(f" diffusion_kv_cache_dtype: {args.diffusion_kv_cache_dtype}")
print(f" diffusion_kv_cache_skip_steps: {args.diffusion_kv_cache_skip_steps}")
print(f" diffusion_kv_cache_skip_layers: {args.diffusion_kv_cache_skip_layers}")
if args.modality == "text2img":
print(f" Output size: {args.width}x{args.height}")
if args.image_path:
print(f" Input image: {args.image_path}")
if additional_config is not None:
print(f" Additional config: {additional_config}")
print(f" Prompts: {prompts}")
print(f"{'=' * 60}\n")
# When --stream is set, print AR CoT text token-by-token in real time.
# Otherwise, collect and print the full AR text once when stage 0 finishes.
omni_outputs = omni.generate(
prompts=formatted_prompts,
sampling_params_list=params_list,
py_generator=True,
use_tqdm=False,
)
img_idx = 0
for req_output in omni_outputs:
ro = req_output
stage_id = getattr(req_output, "stage_id", None)
# AR stage text — each CompletionOutput.text is already a delta when
# output_kind=DELTA, so we can print it directly (matching the pattern
# in serving_chat.py).
if stage_id == 0 and ro and getattr(ro, "outputs", None):
for o in ro.outputs:
text = getattr(o, "text", "") or ""
if text:
print(text, end="", flush=True)
# Non-streaming: one shot with full text — emit a trailing newline.
if not args.stream:
print(flush=True)
# Collect images from diffusion stage
images = getattr(req_output, "images", None)
if not images and ro and hasattr(ro, "images"):
images = ro.images
if images:
for j, img in enumerate(images):
save_path = os.path.join(args.output, f"output_{img_idx}_{j}.png")
img.save(save_path)
print(f"\n[Output] Saved image to {save_path}")
img_idx += 1
if __name__ == "__main__":
main()
@@ -96,6 +96,7 @@ from typing import Any
import torch
from PIL import Image
from vllm_omni.diffusion.data import logger
from vllm_omni.diffusion.utils.image_output import extract_images_from_outputs
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
@@ -103,6 +104,8 @@ from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.model_extras import (
build_image_to_image_prompt,
get_ar_input_builder,
get_ar_tokenizer_validator,
get_extra_body_params,
get_model_class_name,
should_init_extra_args_for_non_diffusion_stages,
@@ -110,6 +113,91 @@ from vllm_omni.model_extras import (
from vllm_omni.platforms import current_omni_platform
def _apply_ar_stage_inputs(
ar_input_builder: Any,
*,
model: str,
prompt_text: str,
extra_body: dict[str, Any],
num_images: int,
height: int | None,
width: int | None,
prompt_dict: dict[str, Any],
sampling_params_list: list[Any],
text_output: bool = False,
trust_remote_code: bool = False,
validate_tokenizer: Any | None = None,
allow_tokenizer_fallback: bool = False,
) -> None:
"""Apply a model's declared AR-stage inputs to the request in place.
Mirrors the helper in ``text_to_image.py``: loads the tokenizer for
byte-for-byte HF-parity AR tokenization, asks the model's
``ar_input_builder`` for the AR prefill + stop tokens, and writes them
onto ``prompt_dict`` and the non-diffusion (AR) stage params.
Model-agnostic -- only the declared-hook contract is assumed.
``trust_remote_code`` must be threaded in from the caller's own resolved
``--trust-remote-code`` flag -- this is a separate tokenizer load outside
the main engine, so it needs the same explicit user opt-in rather than
defaulting to trusting whatever ``--model`` was passed.
``validate_tokenizer``, when the model declares one via
``get_ar_tokenizer_validator``, is called on a successfully-loaded real
tokenizer -- outside the load's try/except, so a validation failure (a
model/tokenizer revision drifting from hardcoded special-token ids)
raises instead of being swallowed into the string-prompt fallback.
``allow_tokenizer_fallback`` (default ``False``, i.e. fail fast): a
tokenizer load failure normally raises, since silently degrading to the
string-prompt form can produce a request that completes with subtly
wrong stop tokens instead of surfacing the real problem (missing
``--trust-remote-code``, network/cache issue, etc.). Set ``True`` only
for explicit unit tests or a deliberate offline-compat run where the
caller has already decided a degraded prompt is acceptable.
"""
try:
from transformers import AutoTokenizer
ar_tokenizer: Any | None = AutoTokenizer.from_pretrained(model, trust_remote_code=trust_remote_code)
except Exception as exc: # noqa: BLE001 - re-raised unless the caller opts into the fallback
if not allow_tokenizer_fallback:
raise
logger.warning(f"AR tokenizer load failed ({exc}); falling back to string prompt (no BPE parity).")
ar_tokenizer = None
if ar_tokenizer is not None and validate_tokenizer is not None:
validate_tokenizer(ar_tokenizer)
ar_inputs = ar_input_builder(
prompt=prompt_text,
tokenizer=ar_tokenizer,
extra_body=extra_body,
num_images=num_images,
height=height,
width=width,
text_output=text_output,
)
if ar_inputs.prompt_token_ids is not None:
prompt_dict["prompt_token_ids"] = ar_inputs.prompt_token_ids
elif ar_inputs.prompt is not None:
prompt_dict["prompt"] = ar_inputs.prompt
if ar_inputs.use_system_prompt:
prompt_dict["use_system_prompt"] = ar_inputs.use_system_prompt
prompt_dict["modalities"] = ar_inputs.modalities
# Apply stop_token_ids to exactly the stage(s) the builder names -- not
# "whichever stage isn't a diffusion stage," which would misfire on any
# future topology with more than one non-diffusion stage.
for stage_index in ar_inputs.stage_indices:
if stage_index >= len(sampling_params_list):
continue
stage_params = sampling_params_list[stage_index]
if not isinstance(stage_params, OmniDiffusionSamplingParams) and hasattr(stage_params, "stop_token_ids"):
stage_params.stop_token_ids = ar_inputs.stop_token_ids
def parse_profiler_config(value: str) -> dict[str, Any]:
try:
config = json.loads(value)
@@ -419,6 +507,22 @@ def parse_args() -> argparse.Namespace:
default=None,
help='JSON profiler config for torch/cuda profiling, e.g. \'{"profiler":"torch","torch_profiler_dir":"./perf"}\'.',
)
parser.add_argument(
"--trust-remote-code",
action="store_true",
help="Trust and execute custom modeling code from the model repo (required by e.g. HunyuanImage-3.0).",
)
parser.add_argument(
"--allow-tokenizer-fallback",
action="store_true",
help=(
"For models with a declared ar_input_builder (e.g. HunyuanImage-3.0): if loading "
"the AR tokenizer fails, degrade to the string-prompt form (no BPE parity) instead "
"of failing the run. Off by default -- a tokenizer load failure usually means a "
"missing --trust-remote-code or a network/cache issue that's worth surfacing, not "
"silently masking with a possibly-wrong prompt."
),
)
return parser.parse_args()
@@ -492,6 +596,8 @@ def main():
)
if args.enforce_eager is not None:
omni_kwargs["enforce_eager"] = args.enforce_eager
if args.trust_remote_code:
omni_kwargs["trust_remote_code"] = True
if args.deploy_config:
omni_kwargs["deploy_config"] = args.deploy_config
omni = Omni(**omni_kwargs)
@@ -584,6 +690,26 @@ def main():
if not diffusion_replaced and len(sampling_params_list) == 1:
sampling_params_list = [diffusion_params]
# Models with an AR text stage (e.g. HunyuanImage3 image-editing) declare an
# ar_input_builder; build the AR prefill + stop tokens declaratively from the
# plain prompt + reference image count + extra_args. Others are untouched.
ar_input_builder = get_ar_input_builder(model_class_name)
if ar_input_builder is not None:
_apply_ar_stage_inputs(
ar_input_builder,
model=args.model,
prompt_text=args.prompt,
extra_body=extra_args_from_cli,
num_images=len(input_images),
height=args.height,
width=args.width,
prompt_dict=prompt_dict,
sampling_params_list=sampling_params_list,
trust_remote_code=args.trust_remote_code,
validate_tokenizer=get_ar_tokenizer_validator(model_class_name),
allow_tokenizer_fallback=args.allow_tokenizer_fallback,
)
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
generation_end = time.perf_counter()
generation_time = generation_end - generation_start
@@ -24,6 +24,8 @@ from vllm_omni.model_extras import (
build_text_to_image_prompt as build_model_text_to_image_prompt,
)
from vllm_omni.model_extras import (
get_ar_input_builder,
get_ar_tokenizer_validator,
get_extra_body_params,
get_model_class_name,
should_init_extra_args_for_non_diffusion_stages,
@@ -377,10 +379,111 @@ def parse_args() -> argparse.Namespace:
default=None,
help="Supplementary auxiliary text encoder parameters model name or path (especially for Hidream-l1-full).",
)
parser.add_argument(
"--trust-remote-code",
action="store_true",
help="Trust and execute custom modeling code from the model repo (required by e.g. HunyuanImage-3.0).",
)
parser.add_argument(
"--allow-tokenizer-fallback",
action="store_true",
help=(
"For models with a declared ar_input_builder (e.g. HunyuanImage-3.0): if loading "
"the AR tokenizer fails, degrade to the string-prompt form (no BPE parity) instead "
"of failing the run. Off by default -- a tokenizer load failure usually means a "
"missing --trust-remote-code or a network/cache issue that's worth surfacing, not "
"silently masking with a possibly-wrong prompt."
),
)
current_omni_platform.pre_register_and_update(parser)
return parser.parse_args()
def _apply_ar_stage_inputs(
ar_input_builder: Any,
*,
model: str,
prompt_text: str,
extra_body: dict[str, Any],
num_images: int,
height: int | None,
width: int | None,
prompt_dict: dict[str, Any],
sampling_params_list: list[Any],
text_output: bool = False,
trust_remote_code: bool = False,
validate_tokenizer: Any | None = None,
allow_tokenizer_fallback: bool = False,
) -> None:
"""Apply a model's declared AR-stage inputs to the request in place.
Loads the model tokenizer (for byte-for-byte HF-parity segment
tokenization) and asks the model's ``ar_input_builder`` for the AR
prefill + stop tokens, then writes them onto ``prompt_dict`` and the
non-diffusion (AR) stage sampling params. Kept model-agnostic: the
example only knows the declared-hook contract.
``trust_remote_code`` must be threaded in from the caller's own resolved
``--trust-remote-code`` flag -- this is a separate tokenizer load outside
the main engine, so it needs the same explicit user opt-in rather than
defaulting to trusting whatever ``--model`` was passed.
``validate_tokenizer``, when the model declares one via
``get_ar_tokenizer_validator``, is called on a successfully-loaded real
tokenizer -- outside the load's try/except, so a validation failure (a
model/tokenizer revision drifting from hardcoded special-token ids)
raises instead of being swallowed into the string-prompt fallback.
``allow_tokenizer_fallback`` (default ``False``, i.e. fail fast): a
tokenizer load failure normally raises, since silently degrading to the
string-prompt form can produce a request that completes with subtly
wrong stop tokens instead of surfacing the real problem (missing
``--trust-remote-code``, network/cache issue, etc.). Set ``True`` only
for explicit unit tests or a deliberate offline-compat run where the
caller has already decided a degraded prompt is acceptable.
"""
try:
from transformers import AutoTokenizer
ar_tokenizer: Any | None = AutoTokenizer.from_pretrained(model, trust_remote_code=trust_remote_code)
except Exception as exc: # noqa: BLE001 - re-raised unless the caller opts into the fallback
if not allow_tokenizer_fallback:
raise
logger.warning(f"AR tokenizer load failed ({exc}); falling back to string prompt (no BPE parity).")
ar_tokenizer = None
if ar_tokenizer is not None and validate_tokenizer is not None:
validate_tokenizer(ar_tokenizer)
ar_inputs = ar_input_builder(
prompt=prompt_text,
tokenizer=ar_tokenizer,
extra_body=extra_body,
num_images=num_images,
height=height,
width=width,
text_output=text_output,
)
if ar_inputs.prompt_token_ids is not None:
prompt_dict["prompt_token_ids"] = ar_inputs.prompt_token_ids
elif ar_inputs.prompt is not None:
prompt_dict["prompt"] = ar_inputs.prompt
if ar_inputs.use_system_prompt:
prompt_dict["use_system_prompt"] = ar_inputs.use_system_prompt
prompt_dict["modalities"] = ar_inputs.modalities
# Apply stop_token_ids to exactly the stage(s) the builder names -- not
# "whichever stage isn't a diffusion stage," which would misfire on any
# future topology with more than one non-diffusion stage.
for stage_index in ar_inputs.stage_indices:
if stage_index >= len(sampling_params_list):
continue
stage_params = sampling_params_list[stage_index]
if not isinstance(stage_params, OmniDiffusionSamplingParams) and hasattr(stage_params, "stop_token_ids"):
stage_params.stop_token_ids = ar_inputs.stop_token_ids
def main():
args = parse_args()
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
@@ -469,6 +572,8 @@ def main():
omni_kwargs["tensor_parallel_size"] = args.tensor_parallel_size
if args.enforce_eager is not None:
omni_kwargs["enforce_eager"] = args.enforce_eager
if args.trust_remote_code:
omni_kwargs["trust_remote_code"] = True
if args.deploy_config:
omni_kwargs["deploy_config"] = args.deploy_config
if use_nextstep:
@@ -613,6 +718,27 @@ def main():
if not diffusion_replaced and len(sampling_params_list) == 1:
sampling_params_list = [diffusion_params]
# Models with an AR text stage (e.g. HunyuanImage3) declare an
# ar_input_builder. When present, build the AR prefill token-ids and AR
# stop-token-ids declaratively from the plain prompt + extra_body, so this
# example stays model-agnostic. Models without one are untouched.
ar_input_builder = get_ar_input_builder(model_class_name)
if ar_input_builder is not None:
_apply_ar_stage_inputs(
ar_input_builder,
model=args.model,
prompt_text=args.prompt,
extra_body=user_extra,
num_images=0,
height=args.height,
width=args.width,
prompt_dict=prompt_dict,
sampling_params_list=sampling_params_list,
trust_remote_code=args.trust_remote_code,
validate_tokenizer=get_ar_tokenizer_validator(model_class_name),
allow_tokenizer_fallback=args.allow_tokenizer_fallback,
)
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
generation_end = time.perf_counter()