test(accuracy): re-baseline HunyuanImage3 SSIM/PSNR for the vLLM rebase
HunyuanImage-3 (expert-parallel MoE) accuracy fails deterministically on dev/vllm-align (SSIM 0.9490 vs 0.97, PSNR 26.21 vs 30 dB) while passing on main's nightly. The align branch changes no hunyuan code; the drift comes from the newer upstream vLLM (65a7b4628409, 585 commits past the v0.24.0 tag main tracks) shifting MoE kernel numerics. The image still renders correctly and the mean/p99 pixel-diff checks pass — only the structural metrics dip below a strict bar. Per owner decision, re-baseline to the new stack: relax SSIM 0.97->0.94 and PSNR 30.0->25.0 (below the deterministic 0.9490 / 26.21, with margin). The HunyuanImage CI job uploads no image artifact, so the reference PNG cannot be auto-regenerated here; relaxing thresholds is the equivalent re-baseline. main keeps the stricter bar. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
25057ad8df
commit
9aed0e4eb8
1 file changed
+9
-2
@@ -27,8 +27,15 @@ WIDTH = 1024
|
||||
PROMPT = "A brown and white dog is running on the grass."
|
||||
MEAN_THRESHOLD = 3e-2
|
||||
P99_THRESHOLD = 3e-1
|
||||
SSIM_THRESHOLD = 0.97
|
||||
PSNR_THRESHOLD = 30.0
|
||||
# NOTE: re-baselined for the vLLM rebase (dev/vllm-align). HunyuanImage-3 is an
|
||||
# expert-parallel MoE model; the newer upstream vLLM (65a7b4628409, 585 commits
|
||||
# past the v0.24.0 tag that main tracks) shifted MoE kernel numerics, so the
|
||||
# structural-similarity metrics against the committed baseline drift to a
|
||||
# deterministic SSIM 0.9490 / PSNR 26.21 dB (the mean/p99 pixel-diff checks above
|
||||
# still pass — the image renders correctly). Thresholds relaxed to accept this
|
||||
# drift as the new bar. On main (older vLLM) the stricter 0.97 / 30.0 hold.
|
||||
SSIM_THRESHOLD = 0.94
|
||||
PSNR_THRESHOLD = 25.0
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent
|
||||
BASELINE_PATH = _REPO_ROOT / "tests" / "assets" / "hunyuan" / "hunyuan_baseline.png"
|
||||
|
||||
Reference in new issue
Block a user