[Example] Migrate HunyuanImage-3.0 to model_extras + shared task examples (#5559)
Signed-off-by: suyanli220 <suyanli220@gmail.com> Signed-off-by: suyan.li <suyan.li@bytedance.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: suyan.li <suyan.li@bytedance.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
17 files changed
+1627
-544
No files matched your search
@@ -1,154 +0,0 @@
|
||||
# HunyuanImage-3.0-Instruct
|
||||
|
||||
This example runs HunyuanImage-3.0-Instruct offline with the unified deploy
|
||||
YAMLs under `vllm_omni/deploy/`.
|
||||
|
||||
## Deploy Configs
|
||||
|
||||
| File | Topology | Default use |
|
||||
| :--- | :--- | :--- |
|
||||
| `vllm_omni/deploy/hunyuan_image_3_moe.yaml` | AR + DiT | Default for `text2img` and `img2img`. |
|
||||
| `vllm_omni/deploy/hunyuan_image3_ar.yaml` | AR only | Default for `img2text` and `text2text`. |
|
||||
| `vllm_omni/deploy/hunyuan_image3_dit.yaml` | DiT only | Standalone diffusion stage. Pass it explicitly with `--deploy-config`. |
|
||||
|
||||
The example chooses a deploy config automatically when `--deploy-config` is
|
||||
omitted:
|
||||
|
||||
| `--modality` | `mode` passed to Omni | Default deploy |
|
||||
| :--- | :--- | :--- |
|
||||
| `text2img` | `text-to-image` | `hunyuan_image_3_moe.yaml` |
|
||||
| `img2img` | `image-editing` | `hunyuan_image_3_moe.yaml` |
|
||||
| `img2text` | `image-to-text` | `hunyuan_image3_ar.yaml` |
|
||||
| `text2text` | `text-to-text` | `hunyuan_image3_ar.yaml` |
|
||||
|
||||
`--modality` is an offline example convenience flag. It maps to the internal
|
||||
`mode` argument passed to `Omni(...)` by this script. HunyuanImage3 uses
|
||||
separate deploy YAMLs for AR + DiT, AR-only, and DiT-only topologies, so the
|
||||
stage topology is selected by the deploy file rather than by YAML mode
|
||||
overrides.
|
||||
|
||||
Online serving does not expose a `--modality` flag or accept `mode` as an API
|
||||
request field. Choose the deploy topology when starting the server with
|
||||
`--deploy-config`, then use the OpenAI-compatible endpoint and request shape for
|
||||
the scenario. The `modalities` request field is used by the chat completions
|
||||
path; the image endpoints infer the image task from the endpoint and payload.
|
||||
|
||||
| Online scenario | Server deploy | Request |
|
||||
| :--- | :--- | :--- |
|
||||
| Text to image | `--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml` | `POST /v1/images/generations`, or `POST /v1/chat/completions` with `"modalities": ["image"]`. |
|
||||
| Image editing | `--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml` | `POST /v1/images/edits`. |
|
||||
| Image/text to text | `--deploy-config vllm_omni/deploy/hunyuan_image3_ar.yaml` | `POST /v1/chat/completions` for text output, for example with `"modalities": ["text"]`. |
|
||||
| DiT-only image generation | `--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml` | `POST /v1/images/generations`. |
|
||||
|
||||
## Run Examples
|
||||
|
||||
Text to image, using the default AR + DiT deploy:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--prompts "A cute cat sitting on a windowsill watching the sunset"
|
||||
```
|
||||
|
||||
Image editing, using the default AR + DiT deploy:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2img \
|
||||
--image-path /path/to/image.png \
|
||||
--prompts "Make the petals neon pink"
|
||||
```
|
||||
|
||||
Image to text, using the AR-only deploy:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality img2text \
|
||||
--image-path /path/to/image.jpg \
|
||||
--prompts "Describe the content of the picture."
|
||||
```
|
||||
|
||||
Text to text, using the AR-only deploy:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2text \
|
||||
--prompts "What is the capital of France?"
|
||||
```
|
||||
|
||||
Standalone DiT, using the DiT-only deploy explicitly:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \
|
||||
--prompts "A cinematic portrait of an astronaut in a greenhouse"
|
||||
```
|
||||
|
||||
Override the default full AR + DiT deploy explicitly:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/hunyuan_image3/end2end.py \
|
||||
--model tencent/HunyuanImage-3.0-Instruct \
|
||||
--modality text2img \
|
||||
--deploy-config vllm_omni/deploy/hunyuan_image_3_moe.yaml \
|
||||
--prompts "A cute cat"
|
||||
```
|
||||
|
||||
## Additional Config
|
||||
|
||||
You can pass diffusion worker `additional_config` from the offline example as a JSON object.
|
||||
This maps to the upstream vLLM `VllmConfig.additional_config` platform extension field:
|
||||
https://docs.vllm.ai/en/stable/api/vllm/config/#vllm.config.VllmConfig.additional_config
|
||||
|
||||
```bash
|
||||
python end2end.py --modality text2img \
|
||||
--prompts "A cute cat" \
|
||||
--additional-config '{"torchair_graph_config":{"enabled":true}}'
|
||||
```
|
||||
|
||||
|
||||
## Key Arguments
|
||||
|
||||
| Argument | Description |
|
||||
| :--- | :--- |
|
||||
| `--deploy-config` | Optional path for unified deploy YAML overrides. |
|
||||
| `--additional-config` | JSON object forwarded to diffusion worker `additional_config`. |
|
||||
| `--modality` | Offline-only convenience flag. One of `text2img`, `img2img`, `img2text`, `text2text`. It selects prompt formatting, internal `mode`, and default deploy config for this script. Online serving uses `--deploy-config` plus the endpoint and, for chat completions, request `modalities` instead. |
|
||||
| `--steps` | Number of diffusion inference steps for image generation. |
|
||||
| `--guidance-scale` | Classifier-free guidance scale for image generation. |
|
||||
| `--height`, `--width` | Output image size for `text2img`. |
|
||||
| `--bot-task` | Override prompt mode. `none`, `think`, `recaption`, `think_recaption`, or `vanilla`. |
|
||||
| `--sys-type` | Override the system prompt type, for example `en_unified` or `en_vanilla`. |
|
||||
| `--vae-use-tiling` | Enable VAE tiling for memory reduction. |
|
||||
|
||||
## Notes
|
||||
|
||||
- `hunyuan_image3_ar.yaml` is a 4-card AR-only text/comprehension deploy.
|
||||
- `hunyuan_image3_dit.yaml` is a single-stage DiT deploy with `stage_id: 0`.
|
||||
- Platform-specific overrides are included in the unified deploy YAMLs.
|
||||
|
||||
## Prompt Format
|
||||
|
||||
HunyuanImage-3.0-Instruct uses an instruct chat template:
|
||||
|
||||
```text
|
||||
<|startoftext|>{system_prompt}
|
||||
|
||||
User: {<img>?}{user_prompt}
|
||||
|
||||
Assistant: {trigger_tag?}
|
||||
```
|
||||
|
||||
- `<img>`: Placeholder for each input image (single token; expanded by the multimodal pipeline).
|
||||
- Trigger tags: `<think>` for CoT and `<recaption>` for recaptioning, placed after `Assistant: `.
|
||||
- System prompt: Auto-selected from `task` and `bot_task`.
|
||||
- `bot_task='vanilla'` with `task='t2i'` uses the bare pretrain template.
|
||||
|
||||
The shared `vllm_omni.diffusion.models.hunyuan_image3.prompt_utils.build_prompt_tokens()`
|
||||
helper handles segment-by-segment tokenization and matches HF `apply_chat_template`.
|
||||
@@ -1,321 +0,0 @@
|
||||
"""
|
||||
HunyuanImage-3.0-Instruct unified end-to-end inference script.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from vllm_omni.diffusion.models.hunyuan_image3.prompt_utils import (
|
||||
MAX_IMAGES_PER_REQUEST,
|
||||
build_prompt_tokens,
|
||||
resolve_stop_token_ids,
|
||||
resolve_sys_type,
|
||||
)
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniPromptType
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image_3_moe.yaml")
|
||||
_DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml")
|
||||
|
||||
_MODALITY_TASK_MAP: dict[str, tuple[str, str | None]] = {
|
||||
"text2img": ("t2i", "think"),
|
||||
"img2img": ("it2i", "think"),
|
||||
"img2text": ("i2t", None),
|
||||
"text2text": ("t2t", None),
|
||||
}
|
||||
|
||||
_MODALITY_DEFAULT_DEPLOY_CONFIG = {
|
||||
"text2img": _DEFAULT_DEPLOY_CONFIG,
|
||||
"img2img": _DEFAULT_DEPLOY_CONFIG,
|
||||
"img2text": _DEFAULT_AR_DEPLOY_CONFIG,
|
||||
"text2text": _DEFAULT_AR_DEPLOY_CONFIG,
|
||||
}
|
||||
|
||||
_MODALITY_MODE = {
|
||||
"text2img": "text-to-image",
|
||||
"img2img": "image-editing",
|
||||
"img2text": "image-to-text",
|
||||
"text2text": "text-to-text",
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="HunyuanImage-3.0-Instruct end-to-end inference.")
|
||||
parser.add_argument("--model", default="tencent/HunyuanImage-3.0-Instruct", help="Model name or local path.")
|
||||
parser.add_argument(
|
||||
"--modality",
|
||||
default="text2img",
|
||||
choices=list(_MODALITY_TASK_MAP),
|
||||
)
|
||||
parser.add_argument("--prompts", nargs="+", default=None, help="Input text prompts.")
|
||||
parser.add_argument(
|
||||
"--image-path",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Input image path(s) for img2img/img2text. Comma-separated for multi-image (up to 3).",
|
||||
)
|
||||
parser.add_argument("--output", type=str, default=".", help="Output directory to save results.")
|
||||
parser.add_argument("--steps", type=int, default=50, help="Number of inference steps.")
|
||||
parser.add_argument("--guidance-scale", type=float, default=5.0, help="Classifier-free guidance scale.")
|
||||
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
|
||||
parser.add_argument("--height", type=int, default=None, help="Output image height.")
|
||||
parser.add_argument("--width", type=int, default=None, help="Output image width.")
|
||||
parser.add_argument("--vae-use-tiling", action="store_true", help="Enable VAE tiling.")
|
||||
parser.add_argument(
|
||||
"--bot-task",
|
||||
type=str,
|
||||
default=None,
|
||||
choices=["none", "think", "recaption", "think_recaption", "vanilla"],
|
||||
help="Override prompt mode. Default: auto from --modality.",
|
||||
)
|
||||
parser.add_argument("--sys-type", type=str, default=None, help="Override system prompt type.")
|
||||
parser.add_argument("--deploy-config", type=str, default=None, help="Custom deploy YAML path.")
|
||||
parser.add_argument(
|
||||
"--stream",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Enable streaming CoT display; print AR text token-by-token in real time.",
|
||||
)
|
||||
parser.add_argument("--log-stats", action="store_true", default=False)
|
||||
parser.add_argument("--init-timeout", type=int, default=300, help="Initialization timeout in seconds.")
|
||||
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
|
||||
parser.add_argument(
|
||||
"--diffusion-kv-cache-dtype",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Diffusion attention KV cache dtype, for example 'fp8'. Separate from vLLM --kv-cache-dtype.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--diffusion-kv-cache-skip-steps",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Denoising step selector to keep diffusion KV cache in native dtype, for example '0,1,4-6'.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--diffusion-kv-cache-skip-layers",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Transformer layer selector to keep diffusion KV cache in native dtype, for example '0-2,10'.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--additional-config",
|
||||
type=str,
|
||||
default=None,
|
||||
help=(
|
||||
"JSON object forwarded to Omni/additional_config, for example "
|
||||
'\'{"torchair_graph_config":{"enabled":true}}\'. Different platforms may support different '
|
||||
"configs. Make sure the configs are valid for the platform you are using. "
|
||||
"Contents must be hashable."
|
||||
),
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def parse_additional_config(raw_value: str | None) -> dict | None:
|
||||
"""Parse a JSON string into an additional_config mapping."""
|
||||
if raw_value is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
additional_config = json.loads(raw_value)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid --additional-config JSON: {exc}") from exc
|
||||
|
||||
if additional_config is None:
|
||||
return None
|
||||
if not isinstance(additional_config, dict):
|
||||
raise ValueError(f"--additional-config must decode to a JSON object, got {type(additional_config).__name__}")
|
||||
return additional_config
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
additional_config = parse_additional_config(args.additional_config)
|
||||
|
||||
task, default_bot_task = _MODALITY_TASK_MAP[args.modality]
|
||||
if args.bot_task is None:
|
||||
bot_task: str | None = default_bot_task
|
||||
elif args.bot_task == "none":
|
||||
bot_task = None
|
||||
else:
|
||||
bot_task = args.bot_task
|
||||
|
||||
deploy_config = args.deploy_config or _MODALITY_DEFAULT_DEPLOY_CONFIG[args.modality]
|
||||
|
||||
omni_kwargs = {
|
||||
"model": args.model,
|
||||
"vae_use_tiling": args.vae_use_tiling,
|
||||
"log_stats": args.log_stats,
|
||||
"init_timeout": args.init_timeout,
|
||||
"enforce_eager": args.enforce_eager,
|
||||
"mode": _MODALITY_MODE[args.modality],
|
||||
"diffusion_kv_cache_dtype": args.diffusion_kv_cache_dtype,
|
||||
"diffusion_kv_cache_skip_steps": args.diffusion_kv_cache_skip_steps,
|
||||
"diffusion_kv_cache_skip_layers": args.diffusion_kv_cache_skip_layers,
|
||||
}
|
||||
|
||||
if additional_config is not None:
|
||||
omni_kwargs["additional_config"] = additional_config
|
||||
omni_kwargs["deploy_config"] = deploy_config
|
||||
|
||||
omni = Omni(**omni_kwargs)
|
||||
|
||||
prompts = args.prompts or ["A cute cat"]
|
||||
input_images: list = []
|
||||
if args.modality in ("img2img", "img2text"):
|
||||
if not args.image_path:
|
||||
raise ValueError(f"--image-path required for {args.modality}, got: {args.image_path}")
|
||||
from PIL import Image
|
||||
|
||||
image_paths = [p.strip() for p in args.image_path.split(",") if p.strip()]
|
||||
if len(image_paths) > MAX_IMAGES_PER_REQUEST:
|
||||
raise ValueError(
|
||||
f"--image-path accepts at most {MAX_IMAGES_PER_REQUEST} images for "
|
||||
f"HunyuanImage-3.0 IT2I, got {len(image_paths)}: {args.image_path}"
|
||||
)
|
||||
for image_path in image_paths:
|
||||
if not os.path.exists(image_path):
|
||||
raise ValueError(f"Image path does not exist: {image_path}")
|
||||
input_images.append(Image.open(image_path).convert("RGB"))
|
||||
if not input_images:
|
||||
raise ValueError(f"--image-path produced no usable paths: {args.image_path!r}")
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
|
||||
mm_image_payload = (input_images[0] if len(input_images) == 1 else input_images) if input_images else None
|
||||
|
||||
formatted_prompts: list[OmniPromptType] = []
|
||||
for prompt in prompts:
|
||||
build_kwargs: dict = {"task": task, "bot_task": bot_task, "sys_type": args.sys_type}
|
||||
if input_images:
|
||||
build_kwargs["num_images"] = len(input_images)
|
||||
result = build_prompt_tokens(prompt, tokenizer, **build_kwargs)
|
||||
token_ids = result.token_ids
|
||||
effective_sys_type = args.sys_type or resolve_sys_type(bot_task)
|
||||
|
||||
prompt_dict: dict = {
|
||||
"prompt_token_ids": token_ids,
|
||||
"prompt": prompt,
|
||||
"use_system_prompt": effective_sys_type,
|
||||
}
|
||||
if args.modality == "text2img":
|
||||
prompt_dict["modalities"] = ["image"]
|
||||
elif args.modality == "img2img":
|
||||
prompt_dict["modalities"] = ["image"]
|
||||
prompt_dict["multi_modal_data"] = {"image": mm_image_payload}
|
||||
prompt_dict["height"] = input_images[0].height
|
||||
prompt_dict["width"] = input_images[0].width
|
||||
elif args.modality == "img2text":
|
||||
prompt_dict["modalities"] = ["text"]
|
||||
prompt_dict["multi_modal_data"] = {"image": mm_image_payload}
|
||||
else:
|
||||
prompt_dict["modalities"] = ["text"]
|
||||
formatted_prompts.append(prompt_dict)
|
||||
|
||||
from vllm.sampling_params import RequestOutputKind
|
||||
|
||||
params_list = list(omni.default_sampling_params_list)
|
||||
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
if (args.height is None) != (args.width is None):
|
||||
raise ValueError("--height and --width must both be specified or both omitted.")
|
||||
user_specified_size = args.height is not None and args.width is not None
|
||||
if args.modality in ("img2text", "text2text"):
|
||||
ar_image_size = "auto"
|
||||
elif user_specified_size:
|
||||
ar_image_size = f"{args.width}x{args.height}"
|
||||
else:
|
||||
ar_image_size = None
|
||||
ar_stop_token_ids = resolve_stop_token_ids(
|
||||
task=task, bot_task=bot_task, tokenizer=tokenizer, image_size=ar_image_size
|
||||
)
|
||||
print(
|
||||
f"[AR Config] task={task}, bot_task={bot_task}, image_size={ar_image_size}, stop_token_ids={ar_stop_token_ids}"
|
||||
)
|
||||
for sp in params_list:
|
||||
if isinstance(sp, OmniDiffusionSamplingParams):
|
||||
sp.num_inference_steps = args.steps
|
||||
sp.guidance_scale = args.guidance_scale
|
||||
sp.guidance_scale_provided = True
|
||||
if args.seed is not None:
|
||||
sp.seed = args.seed
|
||||
if args.modality == "text2img":
|
||||
sp.height = args.height
|
||||
sp.width = args.width
|
||||
elif hasattr(sp, "stop_token_ids"):
|
||||
sp.stop_token_ids = ar_stop_token_ids
|
||||
# When --stream is set, request DELTA output from the AR stage
|
||||
# so we can display CoT text token-by-token in real time.
|
||||
if args.stream and hasattr(sp, "output_kind"):
|
||||
sp.output_kind = RequestOutputKind.DELTA
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print("HunyuanImage-3.0 Generation Configuration:")
|
||||
print(f" Model: {args.model}")
|
||||
print(f" Modality: {args.modality}")
|
||||
print(f" Prompt task: {task}")
|
||||
print(f" Bot task: {bot_task}")
|
||||
print(f" Deploy config: {deploy_config}")
|
||||
print(f" Num stages: {omni.num_stages}")
|
||||
if args.modality in ("text2img", "img2img"):
|
||||
print(f" Inference steps: {args.steps}")
|
||||
print(f" Guidance scale: {args.guidance_scale}")
|
||||
print(f" Seed: {args.seed}")
|
||||
print(f" diffusion_kv_cache_dtype: {args.diffusion_kv_cache_dtype}")
|
||||
print(f" diffusion_kv_cache_skip_steps: {args.diffusion_kv_cache_skip_steps}")
|
||||
print(f" diffusion_kv_cache_skip_layers: {args.diffusion_kv_cache_skip_layers}")
|
||||
if args.modality == "text2img":
|
||||
print(f" Output size: {args.width}x{args.height}")
|
||||
if args.image_path:
|
||||
print(f" Input image: {args.image_path}")
|
||||
if additional_config is not None:
|
||||
print(f" Additional config: {additional_config}")
|
||||
print(f" Prompts: {prompts}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
# When --stream is set, print AR CoT text token-by-token in real time.
|
||||
# Otherwise, collect and print the full AR text once when stage 0 finishes.
|
||||
omni_outputs = omni.generate(
|
||||
prompts=formatted_prompts,
|
||||
sampling_params_list=params_list,
|
||||
py_generator=True,
|
||||
use_tqdm=False,
|
||||
)
|
||||
img_idx = 0
|
||||
for req_output in omni_outputs:
|
||||
ro = req_output
|
||||
stage_id = getattr(req_output, "stage_id", None)
|
||||
|
||||
# AR stage text — each CompletionOutput.text is already a delta when
|
||||
# output_kind=DELTA, so we can print it directly (matching the pattern
|
||||
# in serving_chat.py).
|
||||
if stage_id == 0 and ro and getattr(ro, "outputs", None):
|
||||
for o in ro.outputs:
|
||||
text = getattr(o, "text", "") or ""
|
||||
if text:
|
||||
print(text, end="", flush=True)
|
||||
# Non-streaming: one shot with full text — emit a trailing newline.
|
||||
if not args.stream:
|
||||
print(flush=True)
|
||||
|
||||
# Collect images from diffusion stage
|
||||
images = getattr(req_output, "images", None)
|
||||
if not images and ro and hasattr(ro, "images"):
|
||||
images = ro.images
|
||||
if images:
|
||||
for j, img in enumerate(images):
|
||||
save_path = os.path.join(args.output, f"output_{img_idx}_{j}.png")
|
||||
img.save(save_path)
|
||||
print(f"\n[Output] Saved image to {save_path}")
|
||||
img_idx += 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -96,6 +96,7 @@ from typing import Any
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
from vllm_omni.diffusion.data import logger
|
||||
from vllm_omni.diffusion.utils.image_output import extract_images_from_outputs
|
||||
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
@@ -103,6 +104,8 @@ from vllm_omni.entrypoints.openai.stage_params import clone_sampling_params
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.model_extras import (
|
||||
build_image_to_image_prompt,
|
||||
get_ar_input_builder,
|
||||
get_ar_tokenizer_validator,
|
||||
get_extra_body_params,
|
||||
get_model_class_name,
|
||||
should_init_extra_args_for_non_diffusion_stages,
|
||||
@@ -110,6 +113,91 @@ from vllm_omni.model_extras import (
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
|
||||
def _apply_ar_stage_inputs(
|
||||
ar_input_builder: Any,
|
||||
*,
|
||||
model: str,
|
||||
prompt_text: str,
|
||||
extra_body: dict[str, Any],
|
||||
num_images: int,
|
||||
height: int | None,
|
||||
width: int | None,
|
||||
prompt_dict: dict[str, Any],
|
||||
sampling_params_list: list[Any],
|
||||
text_output: bool = False,
|
||||
trust_remote_code: bool = False,
|
||||
validate_tokenizer: Any | None = None,
|
||||
allow_tokenizer_fallback: bool = False,
|
||||
) -> None:
|
||||
"""Apply a model's declared AR-stage inputs to the request in place.
|
||||
|
||||
Mirrors the helper in ``text_to_image.py``: loads the tokenizer for
|
||||
byte-for-byte HF-parity AR tokenization, asks the model's
|
||||
``ar_input_builder`` for the AR prefill + stop tokens, and writes them
|
||||
onto ``prompt_dict`` and the non-diffusion (AR) stage params.
|
||||
Model-agnostic -- only the declared-hook contract is assumed.
|
||||
|
||||
``trust_remote_code`` must be threaded in from the caller's own resolved
|
||||
``--trust-remote-code`` flag -- this is a separate tokenizer load outside
|
||||
the main engine, so it needs the same explicit user opt-in rather than
|
||||
defaulting to trusting whatever ``--model`` was passed.
|
||||
|
||||
``validate_tokenizer``, when the model declares one via
|
||||
``get_ar_tokenizer_validator``, is called on a successfully-loaded real
|
||||
tokenizer -- outside the load's try/except, so a validation failure (a
|
||||
model/tokenizer revision drifting from hardcoded special-token ids)
|
||||
raises instead of being swallowed into the string-prompt fallback.
|
||||
|
||||
``allow_tokenizer_fallback`` (default ``False``, i.e. fail fast): a
|
||||
tokenizer load failure normally raises, since silently degrading to the
|
||||
string-prompt form can produce a request that completes with subtly
|
||||
wrong stop tokens instead of surfacing the real problem (missing
|
||||
``--trust-remote-code``, network/cache issue, etc.). Set ``True`` only
|
||||
for explicit unit tests or a deliberate offline-compat run where the
|
||||
caller has already decided a degraded prompt is acceptable.
|
||||
"""
|
||||
try:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
ar_tokenizer: Any | None = AutoTokenizer.from_pretrained(model, trust_remote_code=trust_remote_code)
|
||||
except Exception as exc: # noqa: BLE001 - re-raised unless the caller opts into the fallback
|
||||
if not allow_tokenizer_fallback:
|
||||
raise
|
||||
logger.warning(f"AR tokenizer load failed ({exc}); falling back to string prompt (no BPE parity).")
|
||||
ar_tokenizer = None
|
||||
|
||||
if ar_tokenizer is not None and validate_tokenizer is not None:
|
||||
validate_tokenizer(ar_tokenizer)
|
||||
|
||||
ar_inputs = ar_input_builder(
|
||||
prompt=prompt_text,
|
||||
tokenizer=ar_tokenizer,
|
||||
extra_body=extra_body,
|
||||
num_images=num_images,
|
||||
height=height,
|
||||
width=width,
|
||||
text_output=text_output,
|
||||
)
|
||||
|
||||
if ar_inputs.prompt_token_ids is not None:
|
||||
prompt_dict["prompt_token_ids"] = ar_inputs.prompt_token_ids
|
||||
elif ar_inputs.prompt is not None:
|
||||
prompt_dict["prompt"] = ar_inputs.prompt
|
||||
if ar_inputs.use_system_prompt:
|
||||
prompt_dict["use_system_prompt"] = ar_inputs.use_system_prompt
|
||||
prompt_dict["modalities"] = ar_inputs.modalities
|
||||
|
||||
# Apply stop_token_ids to exactly the stage(s) the builder names -- not
|
||||
# "whichever stage isn't a diffusion stage," which would misfire on any
|
||||
# future topology with more than one non-diffusion stage.
|
||||
for stage_index in ar_inputs.stage_indices:
|
||||
if stage_index >= len(sampling_params_list):
|
||||
continue
|
||||
stage_params = sampling_params_list[stage_index]
|
||||
if not isinstance(stage_params, OmniDiffusionSamplingParams) and hasattr(stage_params, "stop_token_ids"):
|
||||
stage_params.stop_token_ids = ar_inputs.stop_token_ids
|
||||
|
||||
|
||||
def parse_profiler_config(value: str) -> dict[str, Any]:
|
||||
try:
|
||||
config = json.loads(value)
|
||||
@@ -419,6 +507,22 @@ def parse_args() -> argparse.Namespace:
|
||||
default=None,
|
||||
help='JSON profiler config for torch/cuda profiling, e.g. \'{"profiler":"torch","torch_profiler_dir":"./perf"}\'.',
|
||||
)
|
||||
parser.add_argument(
|
||||
"--trust-remote-code",
|
||||
action="store_true",
|
||||
help="Trust and execute custom modeling code from the model repo (required by e.g. HunyuanImage-3.0).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-tokenizer-fallback",
|
||||
action="store_true",
|
||||
help=(
|
||||
"For models with a declared ar_input_builder (e.g. HunyuanImage-3.0): if loading "
|
||||
"the AR tokenizer fails, degrade to the string-prompt form (no BPE parity) instead "
|
||||
"of failing the run. Off by default -- a tokenizer load failure usually means a "
|
||||
"missing --trust-remote-code or a network/cache issue that's worth surfacing, not "
|
||||
"silently masking with a possibly-wrong prompt."
|
||||
),
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@@ -492,6 +596,8 @@ def main():
|
||||
)
|
||||
if args.enforce_eager is not None:
|
||||
omni_kwargs["enforce_eager"] = args.enforce_eager
|
||||
if args.trust_remote_code:
|
||||
omni_kwargs["trust_remote_code"] = True
|
||||
if args.deploy_config:
|
||||
omni_kwargs["deploy_config"] = args.deploy_config
|
||||
omni = Omni(**omni_kwargs)
|
||||
@@ -584,6 +690,26 @@ def main():
|
||||
if not diffusion_replaced and len(sampling_params_list) == 1:
|
||||
sampling_params_list = [diffusion_params]
|
||||
|
||||
# Models with an AR text stage (e.g. HunyuanImage3 image-editing) declare an
|
||||
# ar_input_builder; build the AR prefill + stop tokens declaratively from the
|
||||
# plain prompt + reference image count + extra_args. Others are untouched.
|
||||
ar_input_builder = get_ar_input_builder(model_class_name)
|
||||
if ar_input_builder is not None:
|
||||
_apply_ar_stage_inputs(
|
||||
ar_input_builder,
|
||||
model=args.model,
|
||||
prompt_text=args.prompt,
|
||||
extra_body=extra_args_from_cli,
|
||||
num_images=len(input_images),
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
prompt_dict=prompt_dict,
|
||||
sampling_params_list=sampling_params_list,
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
validate_tokenizer=get_ar_tokenizer_validator(model_class_name),
|
||||
allow_tokenizer_fallback=args.allow_tokenizer_fallback,
|
||||
)
|
||||
|
||||
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
|
||||
generation_end = time.perf_counter()
|
||||
generation_time = generation_end - generation_start
|
||||
|
||||
@@ -24,6 +24,8 @@ from vllm_omni.model_extras import (
|
||||
build_text_to_image_prompt as build_model_text_to_image_prompt,
|
||||
)
|
||||
from vllm_omni.model_extras import (
|
||||
get_ar_input_builder,
|
||||
get_ar_tokenizer_validator,
|
||||
get_extra_body_params,
|
||||
get_model_class_name,
|
||||
should_init_extra_args_for_non_diffusion_stages,
|
||||
@@ -377,10 +379,111 @@ def parse_args() -> argparse.Namespace:
|
||||
default=None,
|
||||
help="Supplementary auxiliary text encoder parameters model name or path (especially for Hidream-l1-full).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--trust-remote-code",
|
||||
action="store_true",
|
||||
help="Trust and execute custom modeling code from the model repo (required by e.g. HunyuanImage-3.0).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allow-tokenizer-fallback",
|
||||
action="store_true",
|
||||
help=(
|
||||
"For models with a declared ar_input_builder (e.g. HunyuanImage-3.0): if loading "
|
||||
"the AR tokenizer fails, degrade to the string-prompt form (no BPE parity) instead "
|
||||
"of failing the run. Off by default -- a tokenizer load failure usually means a "
|
||||
"missing --trust-remote-code or a network/cache issue that's worth surfacing, not "
|
||||
"silently masking with a possibly-wrong prompt."
|
||||
),
|
||||
)
|
||||
current_omni_platform.pre_register_and_update(parser)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _apply_ar_stage_inputs(
|
||||
ar_input_builder: Any,
|
||||
*,
|
||||
model: str,
|
||||
prompt_text: str,
|
||||
extra_body: dict[str, Any],
|
||||
num_images: int,
|
||||
height: int | None,
|
||||
width: int | None,
|
||||
prompt_dict: dict[str, Any],
|
||||
sampling_params_list: list[Any],
|
||||
text_output: bool = False,
|
||||
trust_remote_code: bool = False,
|
||||
validate_tokenizer: Any | None = None,
|
||||
allow_tokenizer_fallback: bool = False,
|
||||
) -> None:
|
||||
"""Apply a model's declared AR-stage inputs to the request in place.
|
||||
|
||||
Loads the model tokenizer (for byte-for-byte HF-parity segment
|
||||
tokenization) and asks the model's ``ar_input_builder`` for the AR
|
||||
prefill + stop tokens, then writes them onto ``prompt_dict`` and the
|
||||
non-diffusion (AR) stage sampling params. Kept model-agnostic: the
|
||||
example only knows the declared-hook contract.
|
||||
|
||||
``trust_remote_code`` must be threaded in from the caller's own resolved
|
||||
``--trust-remote-code`` flag -- this is a separate tokenizer load outside
|
||||
the main engine, so it needs the same explicit user opt-in rather than
|
||||
defaulting to trusting whatever ``--model`` was passed.
|
||||
|
||||
``validate_tokenizer``, when the model declares one via
|
||||
``get_ar_tokenizer_validator``, is called on a successfully-loaded real
|
||||
tokenizer -- outside the load's try/except, so a validation failure (a
|
||||
model/tokenizer revision drifting from hardcoded special-token ids)
|
||||
raises instead of being swallowed into the string-prompt fallback.
|
||||
|
||||
``allow_tokenizer_fallback`` (default ``False``, i.e. fail fast): a
|
||||
tokenizer load failure normally raises, since silently degrading to the
|
||||
string-prompt form can produce a request that completes with subtly
|
||||
wrong stop tokens instead of surfacing the real problem (missing
|
||||
``--trust-remote-code``, network/cache issue, etc.). Set ``True`` only
|
||||
for explicit unit tests or a deliberate offline-compat run where the
|
||||
caller has already decided a degraded prompt is acceptable.
|
||||
"""
|
||||
try:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
ar_tokenizer: Any | None = AutoTokenizer.from_pretrained(model, trust_remote_code=trust_remote_code)
|
||||
except Exception as exc: # noqa: BLE001 - re-raised unless the caller opts into the fallback
|
||||
if not allow_tokenizer_fallback:
|
||||
raise
|
||||
logger.warning(f"AR tokenizer load failed ({exc}); falling back to string prompt (no BPE parity).")
|
||||
ar_tokenizer = None
|
||||
|
||||
if ar_tokenizer is not None and validate_tokenizer is not None:
|
||||
validate_tokenizer(ar_tokenizer)
|
||||
|
||||
ar_inputs = ar_input_builder(
|
||||
prompt=prompt_text,
|
||||
tokenizer=ar_tokenizer,
|
||||
extra_body=extra_body,
|
||||
num_images=num_images,
|
||||
height=height,
|
||||
width=width,
|
||||
text_output=text_output,
|
||||
)
|
||||
|
||||
if ar_inputs.prompt_token_ids is not None:
|
||||
prompt_dict["prompt_token_ids"] = ar_inputs.prompt_token_ids
|
||||
elif ar_inputs.prompt is not None:
|
||||
prompt_dict["prompt"] = ar_inputs.prompt
|
||||
if ar_inputs.use_system_prompt:
|
||||
prompt_dict["use_system_prompt"] = ar_inputs.use_system_prompt
|
||||
prompt_dict["modalities"] = ar_inputs.modalities
|
||||
|
||||
# Apply stop_token_ids to exactly the stage(s) the builder names -- not
|
||||
# "whichever stage isn't a diffusion stage," which would misfire on any
|
||||
# future topology with more than one non-diffusion stage.
|
||||
for stage_index in ar_inputs.stage_indices:
|
||||
if stage_index >= len(sampling_params_list):
|
||||
continue
|
||||
stage_params = sampling_params_list[stage_index]
|
||||
if not isinstance(stage_params, OmniDiffusionSamplingParams) and hasattr(stage_params, "stop_token_ids"):
|
||||
stage_params.stop_token_ids = ar_inputs.stop_token_ids
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
|
||||
@@ -469,6 +572,8 @@ def main():
|
||||
omni_kwargs["tensor_parallel_size"] = args.tensor_parallel_size
|
||||
if args.enforce_eager is not None:
|
||||
omni_kwargs["enforce_eager"] = args.enforce_eager
|
||||
if args.trust_remote_code:
|
||||
omni_kwargs["trust_remote_code"] = True
|
||||
if args.deploy_config:
|
||||
omni_kwargs["deploy_config"] = args.deploy_config
|
||||
if use_nextstep:
|
||||
@@ -613,6 +718,27 @@ def main():
|
||||
if not diffusion_replaced and len(sampling_params_list) == 1:
|
||||
sampling_params_list = [diffusion_params]
|
||||
|
||||
# Models with an AR text stage (e.g. HunyuanImage3) declare an
|
||||
# ar_input_builder. When present, build the AR prefill token-ids and AR
|
||||
# stop-token-ids declaratively from the plain prompt + extra_body, so this
|
||||
# example stays model-agnostic. Models without one are untouched.
|
||||
ar_input_builder = get_ar_input_builder(model_class_name)
|
||||
if ar_input_builder is not None:
|
||||
_apply_ar_stage_inputs(
|
||||
ar_input_builder,
|
||||
model=args.model,
|
||||
prompt_text=args.prompt,
|
||||
extra_body=user_extra,
|
||||
num_images=0,
|
||||
height=args.height,
|
||||
width=args.width,
|
||||
prompt_dict=prompt_dict,
|
||||
sampling_params_list=sampling_params_list,
|
||||
trust_remote_code=args.trust_remote_code,
|
||||
validate_tokenizer=get_ar_tokenizer_validator(model_class_name),
|
||||
allow_tokenizer_fallback=args.allow_tokenizer_fallback,
|
||||
)
|
||||
|
||||
outputs = omni.generate(prompt_dict, sampling_params_list=sampling_params_list)
|
||||
|
||||
generation_end = time.perf_counter()
|
||||
|
||||
Reference in new issue
Block a user