[Feat] Add support and testcase for HunyuanImage-3.0-Instruct-Distil (#4048)

Signed-off-by: zengchuang <zengchuang3@huawei.com>
Signed-off-by: zengchuang-hw <zengchuang1005@gmail.com>
This commit is contained in:
Zeng Chuang authored and GitHub committed 2026-09-03 02:43:13 +00:00
1 parent 4927fd5412
commit b5e898f295
12 files changed
+751 -102

No files matched your search

@@ -0,0 +1 @@
首先,我分析所有输入图像:图像1是一个圆形的logo,设计现代且抽象。它由不同色调的蓝色(深蓝、中蓝、浅蓝)和白色构成,这些色块以流畅的曲线相互交织,形成一个动态的、类似旋涡或波浪的图案。整个logo是扁平化的矢量图形,背景为纯黑色。图像2展示了四个并排摆放的卡通动物造型冰箱贴,<recaption>这幅图像以产品摄影的精致风格,呈现了一枚根据`image_1`标志定制的圆形珐琅冰箱贴。最终图像使用`image_2`的分辨率。冰箱贴居中放置在`image_2`的灰色织物背景上,其设计完美复刻了`image_1`中由深蓝、中蓝、浅蓝和白色构成的动态旋涡图案。整个冰箱贴被赋予了`image_2`中冰箱贴特有的高级质感:表面覆盖着一层光滑如镜的珐琅釉面,反射出柔和而清晰的高光;图案的每一个色块边缘都由纤细的抛光金属边框精确勾勒,增强了立体感。柔和的顶光在冰箱贴的弧形边缘上形成平滑的过渡,并在其下方投下淡淡的、轮廓模糊的阴影,使其与织物背景无缝融合,营造出一种真实、静谧的视觉效果。<relation_1>最终图像完整保留了`image_1`中标志的全部设计元素。这包括其完美的圆形轮廓,以及内部由深蓝、中蓝、浅蓝和白色组成的精确旋涡状图案布局、形状和色彩关系。</relation_1><relation_2>最终图像的分辨率、背景和材质均来自`image_2`。背景中灰色织物的纹理和质感被完整保留。冰箱贴的材质被完美重构,精确复刻了`image_2`中冰箱贴所展示的光滑珐琅质感、抛光金属边框的视觉效果,以及整体柔和、均匀的布光环境和由此产生的自然阴影。</relation_2></recaption><answer><boi><img_size_1024><img_ratio_36><timestep>[<img>]{3600}<eoi></answer>
@@ -0,0 +1 @@
首先,我分析所有输入图像:图像1是一个圆形logo,背景为黑色,主体由深蓝、中蓝、浅蓝和白色组成的抽象曲线构成,整体呈现出一种动态的、类似旋涡的视觉效果。图像2展示了四个并排摆放在灰色织物表面上的冰箱贴。这些冰箱贴的造型分别是长颈鹿、小鸟、蜜蜂和海豚,其关键特征是材质:它们具有光滑、高反光的珐琅质感,边缘有明显的银色金属包边,呈现出立体感。用户的指令是“基于图一的logo,参考图二中冰箱贴的材质,制作一个新的冰箱贴”。这个指令要求将一个二维的平面设计(logo)转化为一个具有特定三维物理属性(冰箱贴材质)的实体物品。核心在于保留logo的设计精髓,同时精确复刻参考图像中冰箱贴的材质和工艺感。为了构建最终的答案图像,我会将图一的圆形logo作为新冰箱贴的主体图案。这个圆形冰箱贴将拥有与图二冰箱贴一致的厚度和立体感。其表面会覆盖一层高光泽的透明釉质,完美再现珐琅的质感。logo内部的蓝色和白色区域将保持其原有的色彩,但会因为表面的光泽而出现明亮的高光反射。整个圆形的边缘会包裹一圈纤细而有光泽的银色金属边框,这不仅增加了产品的精致感,也与图二中的金属包边工艺相呼应。最后,将这个新制作的圆形冰箱贴放置在图二所示的灰色织物背景上,确保其光照、阴影和反射效果与周围环境和其他冰箱贴完全一致,使其看起来就像是和它们一起被拍摄的,从而创造出一个逼真且风格统一的视觉效果。</think><recaption>这幅图像以产品摄影的精准风格,展示了一枚根据`image_1`的logo设计定制的圆形冰箱贴。最终图像使用`image_2`的分辨率。该冰箱贴居中放置于`image_2`的灰色织物背景之上,其设计完美复刻了`image_1`中由深蓝、中蓝、浅蓝及白色构成的动态旋涡状抽象图案。整个冰箱贴被赋予了与`image_2`中冰箱贴完全一致的高级珐琅质感,表面覆盖着一层晶莹剔透的高光釉面,在柔和的顶光照射下,呈现出清晰而温润的高光反射。一圈纤薄而精致的银色金属边框紧密包裹着圆形主体,其金属光泽与釉面的质感形成精妙对比。冰箱贴在织物背景上投下了一层极其自然的柔和阴影,证明了其与环境光影的无缝融合,整体视觉效果真实可信,宛如一件精心制作的实体工艺品。<relation_1>最终图像完整保留了`image_1`中logo的全部核心设计元素。这包括其完美的圆形轮廓,以及内部由深蓝、中蓝、浅蓝和白色四种颜色构成的抽象旋涡图案的精确形态、色彩和布局。</relation_1><relation_2>最终图像的分辨率、背景和材质风格完全来自`image_2`。这包括其作为背景的灰色织物表面的独特纹理,以及应用于新冰箱贴上的、与`image_2`中冰箱贴完全相同的珐琅材质(高光釉面质感)和银色金属边框的物理属性。画面的整体光照环境,包括柔和的光源方向和质感,也与`image_2`保持一致。</relation_2></recaption><answer><boi><img_size_1024><img_ratio_36><timestep><guidance><timestep_r>[<img>]{3600}<eoi></answer>
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

@@ -24,6 +24,7 @@ from vllm_omni.diffusion.models.hunyuan_image3.request_layout import (
build_hunyuan_diffusion_kv_requests,
extract_hunyuan_prompt_inputs,
hunyuan_num_image_tokens,
hunyuan_num_special_tokens,
normalize_hunyuan_cot_text,
prepare_hunyuan_layout,
)
@@ -114,7 +115,7 @@ class _FakeImageProcessor:
self.image_sizes: list[tuple[int, int]] = []
self.vision_encoder_processor = SimpleNamespace(patch_size=1)
def build_image_info(self, image_size):
def build_image_info(self, image_size, **kwargs):
self.image_sizes.append(image_size)
return ImageInfo(
image_type="gen_image",
@@ -125,6 +126,7 @@ class _FakeImageProcessor:
image_token_length=self.image_token_length,
base_size=1024,
ratio_index=0,
**kwargs,
)
@@ -166,6 +168,29 @@ def _request(
)
def test_distilled_layout_uses_embedded_cfg_and_meanflow_tokens() -> None:
tokenizer, image_processor = _components([12])
request = _request(guidance_scale=2.5)
prepared_layout = prepare_hunyuan_layout(
request,
tokenizer_wrapper=tokenizer,
image_processor=image_processor,
generation_config=SimpleNamespace(sequence_template="instruct", drop_think=False),
image_base_size=1024,
cfg_distilled=True,
use_meanflow=True,
)
image_info = prepared_layout.generated_image_info
assert image_info.add_guidance_token
assert image_info.add_timestep_r_token
assert tokenizer.calls[0]["cfg_factor"] == 1
assert hunyuan_num_special_tokens(image_info) == 3
assert hunyuan_num_image_tokens(image_info) == 19
assert len(build_hunyuan_diffusion_kv_requests(request, prepared_layout)) == 1
def test_builds_kv_request_lengths_without_model_execution() -> None:
tokenizer, image_processor = _components([12])
request = _request(guidance_scale=1.0)
@@ -385,9 +410,13 @@ def test_prepared_model_inputs_match_local_tokenization(
torch.testing.assert_close(prepared_mask, local_mask)
assert prepared_inputs["full_attn_spans"] == local_inputs["full_attn_spans"]
num_image_tokens = hunyuan_num_image_tokens(prepared_layout.generated_image_info)
local_inputs.update(attention_mask=local_mask, num_image_tokens=num_image_tokens)
prepared_inputs.update(attention_mask=prepared_mask, num_image_tokens=num_image_tokens)
image_info = prepared_layout.generated_image_info
generation_token_counts = {
"num_image_tokens": hunyuan_num_image_tokens(image_info),
"num_special_tokens": hunyuan_num_special_tokens(image_info),
}
local_inputs.update(attention_mask=local_mask, **generation_token_counts)
prepared_inputs.update(attention_mask=prepared_mask, **generation_token_counts)
local_step_inputs = pipeline._update_model_kwargs_for_generation(ModelOutput(), local_inputs)
prepared_step_inputs = pipeline._update_model_kwargs_for_generation(ModelOutput(), prepared_inputs)
@@ -0,0 +1,294 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Unit tests for HunyuanImage3 prepare_inputs_for_generation to ensure
distilled model parameters (guidance, timesteps_r) are properly passed."""
import pytest
import torch
from vllm_omni.diffusion.models.hunyuan_image3.hunyuan_image3_transformer import (
HunyuanImage3Text2ImagePipeline,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class MockHunyuanImage3Pipeline:
"""Minimal mock that replicates prepare_inputs_for_generation method."""
def __init__(self):
# Mock config for distilled models
self.config = type("Config", (), {})()
self.hf_config = type("HFConfig", (), {"cfg_distilled": True, "use_meanflow": True})()
# Bind the real method from the pipeline class
from vllm_omni.diffusion.models.hunyuan_image3.pipeline_hunyuan_image3 import (
HunyuanImage3Pipeline as _Real,
)
prepare_inputs_for_generation = _Real.prepare_inputs_for_generation
_update_model_kwargs_for_generation = _Real._update_model_kwargs_for_generation
class TestPrepareInputsForGenerationDistilled:
"""Tests that distilled model parameters are passed through
prepare_inputs_for_generation."""
def setup_method(self):
self.pipeline = MockHunyuanImage3Pipeline()
def _create_mock_kwargs(self, include_distilled_params: bool = True) -> dict:
"""Create mock kwargs for prepare_inputs_for_generation."""
kwargs = {
"position_ids": torch.tensor([[0, 1, 2]]),
"custom_pos_emb": (torch.tensor([[0.0]]), torch.tensor([[1.0]])),
"mode": "gen_image",
"images": torch.randn(1, 4, 64, 64),
"image_mask": torch.ones(1, 64, 64),
"timestep": torch.tensor([0.5]),
"gen_timestep_scatter_index": torch.tensor([[5]]),
"cond_vae_images": None,
"cond_timestep": None,
"cond_vae_image_mask": None,
"cond_vit_images": None,
"cond_vit_image_mask": None,
"vit_kwargs": None,
"cond_timestep_scatter_index": None,
"query_lens": [10],
"seq_lens": [100],
"num_image_tokens": 4096,
"num_special_tokens": 3,
"ar_kv_reuse_len": 0,
"full_attn_spans": None,
"use_cache": True,
}
if include_distilled_params:
# CFG distilled model parameters
kwargs["guidance"] = torch.tensor([2500.0])
kwargs["guidance_scatter_index"] = torch.tensor([[4]])
# MeanFlow distilled model parameters
kwargs["timesteps_r"] = torch.tensor([0.3])
kwargs["timesteps_r_scatter_index"] = torch.tensor([[3]])
else:
kwargs["guidance"] = None
kwargs["guidance_scatter_index"] = None
kwargs["timesteps_r"] = None
kwargs["timesteps_r_scatter_index"] = None
return kwargs
def test_guidance_passed_to_model_inputs(self):
"""Test that guidance is passed through prepare_inputs_for_generation."""
kwargs = self._create_mock_kwargs(include_distilled_params=True)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
assert "guidance" in model_inputs, "guidance should be in model_inputs"
assert model_inputs["guidance"] is not None, "guidance should not be None"
assert torch.equal(model_inputs["guidance"], kwargs["guidance"])
def test_guidance_scatter_index_passed_to_model_inputs(self):
"""Test that guidance_scatter_index is passed through."""
kwargs = self._create_mock_kwargs(include_distilled_params=True)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
assert "guidance_scatter_index" in model_inputs
assert model_inputs["guidance_scatter_index"] is not None
assert torch.equal(model_inputs["guidance_scatter_index"], kwargs["guidance_scatter_index"])
def test_timesteps_r_passed_to_model_inputs(self):
"""Test that timesteps_r is passed through for MeanFlow models."""
kwargs = self._create_mock_kwargs(include_distilled_params=True)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
assert "timesteps_r" in model_inputs, "timesteps_r should be in model_inputs"
assert model_inputs["timesteps_r"] is not None
assert torch.equal(model_inputs["timesteps_r"], kwargs["timesteps_r"])
def test_timesteps_r_scatter_index_passed_to_model_inputs(self):
"""Test that timesteps_r_scatter_index is passed through."""
kwargs = self._create_mock_kwargs(include_distilled_params=True)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
assert "timesteps_r_scatter_index" in model_inputs
assert model_inputs["timesteps_r_scatter_index"] is not None
assert torch.equal(model_inputs["timesteps_r_scatter_index"], kwargs["timesteps_r_scatter_index"])
def test_num_special_tokens_passed_to_model_inputs(self):
"""Test that num_special_tokens is passed through."""
kwargs = self._create_mock_kwargs(include_distilled_params=True)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
assert "num_special_tokens" in model_inputs
assert model_inputs["num_special_tokens"] == kwargs["num_special_tokens"]
def test_decode_update_keeps_all_special_token_positions(self):
"""Test decode step uses timestep, guidance, and timestep_r positions."""
model_kwargs = self._create_mock_kwargs(include_distilled_params=True)
model_kwargs.update(
{
"tokenizer_output": object(),
"attention_mask": torch.ones(1, 1, 7, 7, dtype=torch.bool),
"image_mask": torch.tensor([[False, False, False, False, True, True, True]]),
"gen_timestep_scatter_index": torch.tensor([[1]]),
"guidance_scatter_index": torch.tensor([[2]]),
"timesteps_r_scatter_index": torch.tensor([[3]]),
}
)
updated = self.pipeline._update_model_kwargs_for_generation({}, model_kwargs)
assert torch.equal(updated["position_ids"], torch.tensor([[1, 2, 3, 4, 5, 6]]))
assert updated["num_special_tokens"] == 3
assert updated["guidance_scatter_index"] is model_kwargs["guidance_scatter_index"]
assert updated["timesteps_r_scatter_index"] is model_kwargs["timesteps_r_scatter_index"]
assert updated["attention_mask"].shape == (1, 1, 6, 7)
def test_distilled_params_none_when_not_provided(self):
"""Test that distilled params are None when not provided."""
kwargs = self._create_mock_kwargs(include_distilled_params=False)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
# All distilled params should be present but None
assert "guidance" in model_inputs
assert model_inputs["guidance"] is None
assert "guidance_scatter_index" in model_inputs
assert model_inputs["guidance_scatter_index"] is None
assert "timesteps_r" in model_inputs
assert model_inputs["timesteps_r"] is None
assert "timesteps_r_scatter_index" in model_inputs
assert model_inputs["timesteps_r_scatter_index"] is None
def test_all_distilled_keys_present_in_model_inputs(self):
"""Test that all 4 distilled keys are always present (even if None)."""
kwargs = self._create_mock_kwargs(include_distilled_params=False)
input_ids = torch.tensor([[1, 2, 3]])
attention_mask = torch.ones(1, 3)
model_inputs = self.pipeline.prepare_inputs_for_generation(
input_ids=input_ids,
past_key_values=None,
attention_mask=attention_mask,
inputs_embeds=None,
tokenizer_output=None,
batch_gen_image_info=None,
generator=None,
**kwargs,
)
required_distilled_keys = [
"guidance",
"guidance_scatter_index",
"timesteps_r",
"timesteps_r_scatter_index",
]
for key in required_distilled_keys:
assert key in model_inputs, f"{key} must be in model_inputs dict"
def test_ar_kv_reuse_rebases_distilled_scatter_indices():
pipeline = object.__new__(HunyuanImage3Text2ImagePipeline)
model_kwargs = {
"query_lens": [8],
"attention_mask": torch.ones(1, 1, 8, 12, dtype=torch.bool),
"position_ids": torch.arange(8).unsqueeze(0),
"image_mask": torch.ones(1, 8, dtype=torch.bool),
"gen_timestep_scatter_index": torch.tensor([[5]]),
"guidance_scatter_index": torch.tensor([[6]]),
"timesteps_r_scatter_index": torch.tensor([[7]]),
}
truncated = pipeline._truncate_reused_prefix(
input_ids=torch.arange(8).unsqueeze(0),
model_kwargs=model_kwargs,
positive_reuse_len=3,
)
assert truncated.tolist() == [[3, 4, 5, 6, 7]]
assert model_kwargs["gen_timestep_scatter_index"].tolist() == [[2]]
assert model_kwargs["guidance_scatter_index"].tolist() == [[3]]
assert model_kwargs["timesteps_r_scatter_index"].tolist() == [[4]]
def test_cfg_parallel_splits_meanflow_scatter_indices():
model_kwargs = {
"timesteps_r_scatter_index": torch.tensor([[1], [2], [3], [4]]),
}
HunyuanImage3Text2ImagePipeline._split_model_kwargs_for_cfg_parallel(
model_kwargs,
batch_size=2,
cfg_rank=1,
)
assert model_kwargs["timesteps_r_scatter_index"].tolist() == [[3], [4]]
@@ -38,6 +38,7 @@ def _pipeline():
cache_backend=None,
diffusion_kv_cache_skip_step_indices=None,
)
pipeline.hf_config = SimpleNamespace(cfg_distilled=False, use_meanflow=False)
pipeline._pipeline = SimpleNamespace()
return pipeline
@@ -451,3 +452,44 @@ def test_denoise_step_uses_input_batch_group_order_and_splits_back(monkeypatch):
assert states[1].extra[_STEP_MODEL_KWARGS]["attention_mask"].shape == (2, 1, 2, 6)
assert states[0].extra[_STEP_MODEL_KWARGS]["full_attn_spans"] == [[(2, 4)], [(2, 4)]]
assert states[1].extra[_STEP_MODEL_KWARGS]["full_attn_spans"] == [[(4, 6)], [(4, 6)]]
def test_distilled_step_supplies_guidance_and_meanflow_timestep(monkeypatch):
pipeline = _pipeline()
pipeline.hf_config = SimpleNamespace(cfg_distilled=True, use_meanflow=True)
monkeypatch.setattr(HunyuanImage3Pipeline, "device", property(lambda self: torch.device("cpu")))
state = _state("distilled", 1)
state.latents = torch.zeros(1, 1)
state.extra[_STEP_GUIDANCE_SCALE] = 2.5
state.extra[_STEP_MODEL_KWARGS].update(
{
"attention_mask": torch.ones(1, 1, 2, 4, dtype=torch.bool),
"full_attn_spans": [[(2, 4)]],
}
)
state.extra[_STEP_PROMPT_KV] = [
{
"key": torch.zeros(1, 2, 1, 1),
"value": torch.zeros(1, 2, 1, 1),
"lens": torch.tensor([2]),
}
]
state.scheduler = SimpleNamespace(get_timestep_r=lambda _timestep: torch.tensor(0.25))
captured = {}
pipeline._restore_prompt_kv_cache = lambda *_args: None
def fake_prepare_inputs(input_ids, images, timestep, **model_kwargs):
del input_ids, images, timestep
captured.update(model_kwargs)
return {"model_kwargs": model_kwargs}
pipeline.prepare_inputs_for_generation = fake_prepare_inputs
pipeline.forward_call = lambda **_kwargs: {"diffusion_prediction": torch.tensor([[1.0]])}
pipeline._update_model_kwargs_for_generation = lambda _output, model_kwargs: model_kwargs
output = pipeline.denoise_step(InputBatch.make_batch([state]))
torch.testing.assert_close(output, torch.tensor([[1.0]]))
torch.testing.assert_close(captured["guidance"], torch.tensor([2500.0], dtype=torch.bfloat16))
torch.testing.assert_close(captured["timesteps_r"], torch.tensor([0.25]))
+102 -69
View File
@@ -94,15 +94,28 @@ def _default_ar_dit_devices() -> tuple[str, str]:
AR_DEVICES, DIT_DEVICES = _default_ar_dit_devices()
MODEL_NAME = "tencent/HunyuanImage-3.0-Instruct"
HUNYUAN_IMAGE_REF_PATH = get_asset_path("hunyuan/hunyuan_image_ref.png")
NUM_INFERENCE_STEPS = 50
# Model configuration based on HUNYUAN_MODEL_PATH
DISTIL_MODEL_NAME = "tencent/HunyuanImage-3.0-Instruct-Distil"
INSTRUCT_MODEL_NAME = "tencent/HunyuanImage-3.0-Instruct"
MODEL_PATH = os.environ.get("HUNYUAN_MODEL_PATH", DISTIL_MODEL_NAME)
# Determine model type and set parameters accordingly
IS_DISTIL = "distil" in MODEL_PATH.lower()
MODEL_NAME = DISTIL_MODEL_NAME if IS_DISTIL else INSTRUCT_MODEL_NAME
NUM_INFERENCE_STEPS = 8 if IS_DISTIL else 50
GUIDANCE_SCALE = 2.5
# Asset paths based on model type
ASSET_SUBDIR = "hunyuan_image3"
REF_IMAGE_NAME = "hunyuan_image_distill_ref.png" if IS_DISTIL else "hunyuan_image_instruct_ref.png"
COT_REF_NAME = "hunyuan_image_distill_cot_ref.txt" if IS_DISTIL else "hunyuan_image3_instruct_cot_ref.txt"
HUNYUAN_IMAGE_REF_PATH = get_asset_path(f"{ASSET_SUBDIR}/{REF_IMAGE_NAME}")
HUNYUAN_COT_REF_PATH = get_asset_path(f"{ASSET_SUBDIR}/{COT_REF_NAME}")
# ============================================================================
# Constants
# ============================================================================
MODEL_PATH = os.environ.get("HUNYUAN_MODEL_PATH", MODEL_NAME)
# Test input
PROMPT = "基于图一的logo,参考图二中冰箱贴的材质,制作一个新的冰箱贴"
TEST_IMAGE_URLS = [
@@ -113,23 +126,52 @@ SEED = 42
AR_TP_SIZE = len(AR_DEVICES.split(","))
DIT_TP_SIZE = len(DIT_DEVICES.split(","))
# Precision thresholds
# Precision thresholds organized by model variant and test type
THRESHOLDS = {
# AR text comparison
"text_prefix_match": 10, # First 10 characters must match exactly
"cot_semantic_sim": 0.9, # Full CoT semantic similarity
# Image comparison
"clip_score": 90, # CLIP image semantic similarity
"ssim": 0.26, # Structural similarity
"psnr": 12.5, # Peak signal-to-noise ratio (dB)
"instruct": {
# AR text comparison
"text_prefix_match": 10, # First 10 characters must match exactly
"cot_semantic_sim": 0.9, # Full CoT semantic similarity
# Image comparison
"clip_score": 90, # CLIP image semantic similarity
"ssim": 0.26, # Structural similarity
"psnr": 12.5, # Peak signal-to-noise ratio (dB)
},
# The 8-step distilled model has more structural variance across GPU
# runtimes; CLIP and PSNR retain the same semantic and pixel safeguards.
"distil": {
"text_prefix_match": 10,
"cot_semantic_sim": 0.9,
"clip_score": 90,
"ssim": 0.23,
"psnr": 12.5,
},
# Thresholds for quantized DiT accuracy tests (bf16 vs quant comparison)
"quant": {
"psnr": 10.0,
"ssim": 0.20,
"clip_score": 20.0, # Minimum CLIP score for quantized output
"clip_score_drop": float(
os.environ.get("HUNYUAN_IMAGE3_QUANT_CLIP_SCORE_DROP_THRESHOLD", "5.0")
), # Max allowed drop from bf16
},
}
# Helper to get thresholds for current model variant
def _get_thresholds() -> dict:
"""Return appropriate thresholds based on test context."""
return THRESHOLDS["distil" if IS_DISTIL else "instruct"]
def _get_quant_thresholds() -> dict:
"""Return thresholds for quantization accuracy tests."""
return THRESHOLDS["quant"]
# Quant test configuration
QUANT_PROMPT = "A brown and white dog is running on the grass."
QUANT_HEIGHT, QUANT_WIDTH = 1024, 1024
QUANT_PSNR_THRESHOLD = 10.0
QUANT_SSIM_THRESHOLD = 0.20
QUANT_CLIP_SCORE_THRESHOLD = 20.0
QUANT_CLIP_SCORE_DROP_THRESHOLD = float(os.environ.get("HUNYUAN_IMAGE3_QUANT_CLIP_SCORE_DROP_THRESHOLD", "5.0"))
QUANT_RUN_ENV = "HUNYUAN_IMAGE3_RUN_QUANT_ACCURACY"
QUANT_BF16_ENV = "HUNYUAN_IMAGE3_BF16_MODEL"
QUANT_FP8_ENV = "HUNYUAN_IMAGE3_FP8_MODEL"
@@ -167,7 +209,6 @@ _DEPLOY_CONFIG = {
"top_p": 1,
"top_k": -1,
"max_tokens": 8192,
"stop_token_ids": [128025],
"detokenize": True,
"skip_special_tokens": False,
"include_stop_str_in_output": True,
@@ -246,19 +287,6 @@ def _quant_accuracy_cases() -> list[pytest.ParameterSet]:
return params
# fmt: off
COT_REF = ("首先,我分析所有输入图像:图像1是一个圆形的logo,设计现代且抽象。它由不同色调的蓝色(深蓝、中蓝、浅蓝)和白色构成,这些色块以流畅的曲线相互交织,形成一个动态的、类似旋涡或波浪的图案。整个logo是扁平化的矢量图形,背景为纯黑色。图像2展示了四个并排摆放的卡通动物造型冰箱贴,"
"它们被放置在灰色的织物背景上。这些冰箱贴的关键特征是其材质:它们具有光滑、高光的珐琅或烤漆质感,边缘有明显的金属包边,整体呈现出一种立体的、有厚度的实体感。用户的指令是“基于图一的logo,参考图二中冰箱贴的材质,制作一个新的冰箱贴”。这个指令要求将一个二维的平面设计(logo)"
"转化为一个具有特定物理属性(材质和立体感)的三维物体。核心任务是保留logo的视觉识别性,同时赋予其冰箱贴的实体质感。为了构建答案图像,我会将图一的圆形logo作为基础形状。然后,我会将图二中冰箱贴的材质特性应用到这个logo上。具体来说,logo中的每一个色块(深蓝、中蓝、浅蓝、白色)"
"都会被渲染成具有高光泽度的珐琅质感,表面会反射出柔和的环境光,形成自然的高光。logo中不同颜色区域之间的分界线,将被处理成纤细的、带有金属光泽的凸起边缘,这既能清晰地勾勒出图案,也符合珐琅工艺品的典型特征。整个冰箱贴会呈现出轻微的厚度和圆润的边缘,使其看起来像一个真实的、可触摸的物体。"
"最后,将这个制作完成的冰箱贴放置在图二所示的灰色织物背景上,并为其添加一个微妙的、柔和的阴影,以增强其立体感和与背景的融合度,最终呈现出一个精致、逼真的产品展示图。</think><recaption>这幅图像以产品摄影的精致风格,呈现了一枚根据`image_1`标志定制的圆形珐琅冰箱贴。最终图像使用`image_2`的分辨率。"
"冰箱贴居中放置在`image_2`的灰色织物背景上,其设计完美复刻了`image_1`中由深蓝、中蓝、浅蓝和白色构成的动态旋涡图案。整个冰箱贴被赋予了`image_2`中冰箱贴特有的高级质感:表面覆盖着一层光滑如镜的珐琅釉面,反射出柔和而清晰的高光;图案的每一个色块边缘都由纤细的抛光金属边框精确勾勒,增强了立体感。"
"柔和的顶光在冰箱贴的弧形边缘上形成平滑的过渡,并在其下方投下淡淡的、轮廓模糊的阴影,使其与织物背景无缝融合,营造出一种真实、静谧的视觉效果。<relation_1>最终图像完整保留了`image_1`中标志的全部设计元素。这包括其完美的圆形轮廓,以及内部由深蓝、中蓝、浅蓝和白色组成的精确旋涡状图案布局、形状和色彩关系。"
"</relation_1><relation_2>最终图像的分辨率、背景和材质均来自`image_2`。背景中灰色织物的纹理和质感被完整保留。冰箱贴的材质被完美重构,精确复刻了`image_2`中冰箱贴所展示的光滑珐琅质感、抛光金属边框的视觉效果,以及整体柔和、均匀的布光环境和由此产生的自然阴影。</relation_2></recaption><answer><boi>"
"<img_size_1024><img_ratio_36><timestep>[<img>]{3600}<eoi></answer>")
# fmt: on
def _make_config(enable_kv_reuse: bool, path: Path) -> None:
config = copy.deepcopy(_DEPLOY_CONFIG)
config["stages"][0]["omni_kv_config"] = {"need_send_cache": enable_kv_reuse}
@@ -432,34 +460,36 @@ def test_image_to_image_alignment_online(accuracy_artifact_root: Path) -> None:
online_cot = online_cot.lstrip("\n")
scorer = SemanticSimilarityScorer()
clip_scorer = CLIPScorer()
cot_results = scorer.text_similarity(online_cot, COT_REF)
cot_ref = HUNYUAN_COT_REF_PATH.read_text(encoding="utf-8")
cot_results = scorer.text_similarity(online_cot, cot_ref)
image_ref = Image.open(HUNYUAN_IMAGE_REF_PATH).convert("RGB")
image_clip_score = clip_scorer.image_image_score(online_image, image_ref)
ssim_value, psnr_value = compute_image_ssim_psnr(prediction=online_image, reference=image_ref, compare_mode="RGB")
thresholds = _get_thresholds()
table = [
["COT similarity to reference", f"{cot_results['cot_semantic_sim']:.4f}", 0.9644],
["COT prefix match", f"{cot_results['text_prefix_match_count']:.4f}", 29],
["Image-Image similarity", f"{image_clip_score:.4f}", 94.5538],
["SSIM", f"{ssim_value:.4f}", 0.242],
["PSNR (dB)", f"{psnr_value:.2f}", 14.1],
["COT similarity to reference", f"{cot_results['cot_semantic_sim']:.4f}", thresholds["cot_semantic_sim"]],
["COT prefix match", f"{cot_results['text_prefix_match_count']:.4f}", thresholds["text_prefix_match"]],
["Image-Image similarity", f"{image_clip_score:.4f}", thresholds["clip_score"]],
["SSIM", f"{ssim_value:.4f}", thresholds["ssim"]],
["PSNR (dB)", f"{psnr_value:.2f}", thresholds["psnr"]],
]
print("[ONLINE] " + tabulate(table, headers=["Metric", "Value", "L20x Reference"], tablefmt="grid"))
print("[ONLINE] " + tabulate(table, headers=["Metric", "Value", "Threshold"], tablefmt="grid"))
_assert_cot_structural_markers(online_cot, "ONLINE")
assert cot_results["cot_semantic_sim"] >= THRESHOLDS["cot_semantic_sim"], (
f"[ONLINE] COT semantic similarity {cot_results['cot_semantic_sim']:.4f} below threshold {THRESHOLDS['cot_semantic_sim']}"
assert cot_results["cot_semantic_sim"] >= thresholds["cot_semantic_sim"], (
f"[ONLINE] COT semantic similarity {cot_results['cot_semantic_sim']:.4f} below threshold {thresholds['cot_semantic_sim']}"
)
assert cot_results["text_prefix_match_count"] >= THRESHOLDS["text_prefix_match"], (
f"[ONLINE] COT prefix match {cot_results['text_prefix_match_count']} below threshold {THRESHOLDS['text_prefix_match']}"
assert cot_results["text_prefix_match_count"] >= thresholds["text_prefix_match"], (
f"[ONLINE] COT prefix match {cot_results['text_prefix_match_count']} below threshold {thresholds['text_prefix_match']}"
)
assert image_clip_score >= THRESHOLDS["clip_score"], (
f"[ONLINE] Image-Image similarity {image_clip_score:.4f} below threshold {THRESHOLDS['clip_score']}"
assert image_clip_score >= thresholds["clip_score"], (
f"[ONLINE] Image-Image similarity {image_clip_score:.4f} below threshold {thresholds['clip_score']}"
)
assert ssim_value >= THRESHOLDS["ssim"], f"[ONLINE] SSIM {ssim_value:.4f} below threshold {THRESHOLDS['ssim']}"
assert psnr_value >= THRESHOLDS["psnr"], (
f"[ONLINE] PSNR {psnr_value:.2f} dB below threshold {THRESHOLDS['psnr']} dB"
assert ssim_value >= thresholds["ssim"], f"[ONLINE] SSIM {ssim_value:.4f} below threshold {thresholds['ssim']}"
assert psnr_value >= thresholds["psnr"], (
f"[ONLINE] PSNR {psnr_value:.2f} dB below threshold {thresholds['psnr']} dB"
)
@@ -546,34 +576,36 @@ def test_image_to_image_alignment(accuracy_artifact_root: Path) -> None:
scorer = SemanticSimilarityScorer()
clip_scorer = CLIPScorer()
cot_results = scorer.text_similarity(omni_cot, COT_REF)
cot_ref = HUNYUAN_COT_REF_PATH.read_text(encoding="utf-8")
cot_results = scorer.text_similarity(omni_cot, cot_ref)
image_ref = Image.open(HUNYUAN_IMAGE_REF_PATH).convert("RGB")
image_clip_score = clip_scorer.image_image_score(omni_image, image_ref)
ssim_value, psnr_value = compute_image_ssim_psnr(prediction=omni_image, reference=image_ref, compare_mode="RGB")
thresholds = _get_thresholds()
table = [
["COT similarity to reference", f"{cot_results['cot_semantic_sim']:.4f}", 0.9644],
["COT prefix match", f"{cot_results['text_prefix_match_count']:.4f}", 29],
["Image-Image similarity", f"{image_clip_score:.4f}", 94.5538],
["SSIM", f"{ssim_value:.4f}", 0.242],
["PSNR (dB)", f"{psnr_value:.2f}", 14.1],
["COT similarity to reference", f"{cot_results['cot_semantic_sim']:.4f}", thresholds["cot_semantic_sim"]],
["COT prefix match", f"{cot_results['text_prefix_match_count']:.4f}", thresholds["text_prefix_match"]],
["Image-Image similarity", f"{image_clip_score:.4f}", thresholds["clip_score"]],
["SSIM", f"{ssim_value:.4f}", thresholds["ssim"]],
["PSNR (dB)", f"{psnr_value:.2f}", thresholds["psnr"]],
]
print(tabulate(table, headers=["Metric", "Value", "L20x Reference"], tablefmt="grid"))
print(tabulate(table, headers=["Metric", "Value", "Threshold"], tablefmt="grid"))
_assert_cot_structural_markers(omni_cot, "OFFLINE")
assert cot_results["cot_semantic_sim"] >= THRESHOLDS["cot_semantic_sim"], (
f"COT semantic similarity {cot_results['cot_semantic_sim']:.4f} is below threshold {THRESHOLDS['cot_semantic_sim']}"
assert cot_results["cot_semantic_sim"] >= thresholds["cot_semantic_sim"], (
f"COT semantic similarity {cot_results['cot_semantic_sim']:.4f} is below threshold {thresholds['cot_semantic_sim']}"
)
assert cot_results["text_prefix_match_count"] >= THRESHOLDS["text_prefix_match"], (
f"COT prefix match count {cot_results['text_prefix_match_count']} is below threshold {THRESHOLDS['text_prefix_match']}"
assert cot_results["text_prefix_match_count"] >= thresholds["text_prefix_match"], (
f"COT prefix match count {cot_results['text_prefix_match_count']} is below threshold {thresholds['text_prefix_match']}"
)
assert image_clip_score >= THRESHOLDS["clip_score"], (
f"Image-Image similarity{image_clip_score:.4f} is below threshold {THRESHOLDS['clip_score']}"
assert image_clip_score >= thresholds["clip_score"], (
f"Image-Image similarity{image_clip_score:.4f} is below threshold {thresholds['clip_score']}"
)
assert ssim_value >= THRESHOLDS["ssim"], f"SSIM {ssim_value:.4f} is below threshold {THRESHOLDS['ssim']}"
assert psnr_value >= THRESHOLDS["psnr"], f"PSNR {psnr_value:.2f} dB is below threshold {THRESHOLDS['psnr']} dB"
assert ssim_value >= thresholds["ssim"], f"SSIM {ssim_value:.4f} is below threshold {thresholds['ssim']}"
assert psnr_value >= thresholds["psnr"], f"PSNR {psnr_value:.2f} dB is below threshold {thresholds['psnr']} dB"
@pytest.mark.parametrize("case", _quant_accuracy_cases())
@@ -607,12 +639,13 @@ def test_quantized_dit_matches_bf16_accuracy(
prediction=quant_image,
reference=bf16_image,
)
quant_thresholds = _get_quant_thresholds()
assert_similarity(
model_name=f"{MODEL_NAME} {case.name} vs bf16",
vllm_image=quant_image,
diffusers_image=bf16_image,
ssim_threshold=QUANT_SSIM_THRESHOLD,
psnr_threshold=QUANT_PSNR_THRESHOLD,
ssim_threshold=quant_thresholds["ssim"],
psnr_threshold=quant_thresholds["psnr"],
width=QUANT_WIDTH,
height=QUANT_HEIGHT,
)
@@ -647,15 +680,15 @@ def test_quantized_dit_matches_bf16_accuracy(
print(f" bf16 model: {bf16_model}")
print(f" quant model: {quant_model}")
print(f" BF16 CLIP score: {bf16_clip_score:.4f}")
print(f" quant CLIP score: {quant_clip_score:.4f} threshold>={QUANT_CLIP_SCORE_THRESHOLD:.4f}")
print(f" CLIP score drop: {clip_score_drop:.4f} threshold<={QUANT_CLIP_SCORE_DROP_THRESHOLD:.4f}")
print(f" quant CLIP score: {quant_clip_score:.4f} threshold>={quant_thresholds['clip_score']:.4f}")
print(f" CLIP score drop: {clip_score_drop:.4f} threshold<={quant_thresholds['clip_score_drop']:.4f}")
print(f" metrics: {metrics_path}")
assert quant_clip_score >= QUANT_CLIP_SCORE_THRESHOLD, (
assert quant_clip_score >= quant_thresholds["clip_score"], (
f"{case.name} CLIP score below threshold: got {quant_clip_score:.4f}, "
f"expected >= {QUANT_CLIP_SCORE_THRESHOLD:.4f}"
f"expected >= {quant_thresholds['clip_score']:.4f}"
)
assert clip_score_drop <= QUANT_CLIP_SCORE_DROP_THRESHOLD, (
assert clip_score_drop <= quant_thresholds["clip_score_drop"], (
f"{case.name} CLIP score drop too large: got {clip_score_drop:.4f}, "
f"expected <= {QUANT_CLIP_SCORE_DROP_THRESHOLD:.4f}"
f"expected <= {quant_thresholds['clip_score_drop']:.4f}"
)