vLLM 0.25 reworked the pooling serving base: PoolingServing.__init__ now
requires a `chat_template_config: ChatTemplateConfig` keyword and no longer
accepts the old `chat_template` / `chat_template_content_format` /
`trust_request_chat_template` kwargs. Build a ChatTemplateConfig once and pass
it to the embedding / pooling / classification / score / responses serving
objects. ServingTokenization keeps the flat kwargs (unchanged upstream).
Without this, omni_init_app_state crashes at startup (missing required arg /
unexpected kwargs) whenever an embed/classify/score task is served.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Qwen3-Omni /v1/realtime accuracy check transcribed the generated
audio with whisper-small, which mishears short Chinese TTS clips
(observed: 北京->韦京 plus a dropped leading sentence, sim=0.443) and
produced spurious sim<threshold failures under async_chunk codec
variability even though audio generation was correct. Switch the grader
to large-v3 so a failure here points at the model, not the ASR check.
Also parameterize the threshold per mode (0.65 async_chunk, 0.8 sync).
Verified on 2x H100 (GPU 0,1), both realtime tests pass:
test_streaming_audio_input_pcm_output_async_chunk[async_chunk] PASSED
test_streaming_audio_input_pcm_output[sync] PASSED
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each (stage_id, replica_id) route allocated its 3 handshake/input/output
ports via a separate get_open_ports_list(count=3) call. That helper only
guarantees uniqueness *within* a single call, so across routes the OS could
hand back a just-closed ephemeral port. Two stages then drew the same port
and the second engine subprocess to bind() it died with
"zmq.error.ZMQError: Address already in use", surfacing as
"Orchestrator initialization failed" / "Stage N replica M exited before API
server became ready". This flaked multi-stage startup (e.g. Qwen3-Omni
thinker/talker/code2wav) — one CI shard passed while a parallel shard failed
on the same commit (build 2630, dev/vllm-align).
OmniMasterServer now tracks every port it hands out in self._allocated_ports
(seeded with the registration port and the coordinator ROUTER port) and a new
_alloc_unique_ports() redraws any collision with a bounded retry budget,
raising loudly on port exhaustion instead of spinning. Adds _port_from_zmq_address()
helper and two CPU-only regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HunyuanImage-3 (expert-parallel MoE) accuracy fails deterministically on
dev/vllm-align (SSIM 0.9490 vs 0.97, PSNR 26.21 vs 30 dB) while passing on
main's nightly. The align branch changes no hunyuan code; the drift comes
from the newer upstream vLLM (65a7b4628409, 585 commits past the v0.24.0
tag main tracks) shifting MoE kernel numerics. The image still renders
correctly and the mean/p99 pixel-diff checks pass — only the structural
metrics dip below a strict bar.
Per owner decision, re-baseline to the new stack: relax SSIM 0.97->0.94
and PSNR 30.0->25.0 (below the deterministic 0.9490 / 26.21, with margin).
The HunyuanImage CI job uploads no image artifact, so the reference PNG
cannot be auto-regenerated here; relaxing thresholds is the equivalent
re-baseline. main keeps the stricter bar.
Co-Authored-By: Claude <noreply@anthropic.com>
The previous commit (93730cfd) replaced the blanket module.to() in
_process_weights_after_loading with an unconditional per-parameter move.
That broke the FSDP/HSDP path: after apply_hsdp_to_model the params are
FSDP-sharded DTensors whose storage cannot be cross-device reassigned via
`param.data = param.data.to(cpu)`, raising at load time:
RuntimeError: Attempted to set the storage of a tensor on device
"cuda:0" to a storage on different device "cpu". This is no longer
allowed; the devices must match.
Regressed nightly test_zimage[layerwise_hsdp] (Diffusion X2I Function
Test with L4), which passed pre-fix. The FP8 online-quant variants it was
meant to fix (qwen/bagel fp8, parallel_*_fp8_*) all passed.
Meta-device params only ever arise from online quant (uses_meta_device),
so gate the meta-safe per-parameter move to that case and keep the
original FSDP/HSDP-aware whole-module module.to() for everything else.
Online quant initializes on the accelerator, so its params are normally
already on the target device (the per-param loop is a no-op move) — the
only behavioral change vs. the original is skipping meta stragglers.
Co-Authored-By: Claude <noreply@anthropic.com>
Newer upstream vLLM (65a7b4628409, 585 commits past the v0.24.0 tag that
vllm-omni main tracks) reworked online quantization: online-quant linear
methods now declare `uses_meta_device=True` and create weights on the
`meta` device, materializing them just-in-time via the layerwise
online-process loader as each layer's weights finish loading.
The diffusion loader's `_process_weights_after_loading` did a blanket
`module.to(target_device)`, which raises
`NotImplementedError: Cannot copy out of meta tensor; no data!` on any
layer whose weights were not fully materialized during load (padded /
partially-loaded "straggler" layers), crashing diffusion worker startup.
This regressed the nightly diffusion FP8 tests on dev/vllm-align (all
green on main nightly): Quantization Test (H100 + L4,
test_single_stage_qwen_image_fp8 / test_bagel_fp8_generates_image) and
the Qwen-Image Multi-GPU vae_patch_parallel_2 case (fp8).
Mirror upstream's loader contract:
- Materialize online-quant stragglers via finalize_layerwise_processing()
before processing (imported lazily, only when online quant is in use,
so older vLLM is unaffected). model_config=None is safe: it is only
used for vLLM Attention/MLAAttention layers, which diffusion DiT models
do not have.
- Replace the blanket module.to() with a per-parameter move that mirrors
upstream's device_loading_context: move only real (non-meta) params to
the target device for processing and restore them afterward, leaving
meta params for the quant method / finalize step.
Co-Authored-By: Claude <noreply@anthropic.com>
Dockerfile.ci build failed at the nvjitlink step:
uv pip install --system "nvidia-cuda-nvjitlink-cu13"
-> No solution found ... nvidia-cuda-nvjitlink-cu13 was not found in the registry
That package name does not exist (spurious 'cuda-'; real wheel is
nvidia-nvjitlink-cu13), and the install is redundant: the cu130 torch
force-reinstall one step above already pulls nvidia-nvjitlink==13.0.88 into
site-packages, which provides libnvJitLink.so.13. The step only needs to
register that lib dir with ldconfig.
Locate the already-installed .so.13 and ldconfig-register it; fall back to a
correctly-named nvidia-nvjitlink-cu13 install only if torch deps did not bring
it in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three regressions surfaced only on dev/vllm-align in Buildkite build 2520
(26 failing jobs vs 5 on main); ~15 of the 19 align-unique failures trace to these:
1. numpy 2.5 vs numba: Dockerfile.ci pinned numpy==2.2.6 before the flashinfer
and torch cu130 reinstalls, which resolved numpy back to 2.5 and broke
'import numba' (needs numpy <= 2.4). Re-pin numpy as the final dependency op.
2. libnvJitLink.so.13 not found: the CUDA-13 JIT linker lives in the
nvidia-cuda-nvjitlink-cu13 wheel, not /usr/local/cuda (CUDA 12) as the old
LD_LIBRARY_PATH assumed; bitsandbytes/torch.compile subprocesses could not
dlopen it (broke Entrypoint L4, a multi-GPU diffusion job, both Quantization
jobs via hang/timeout). Install the wheel and register its lib dir via ldconfig.
3. transformers >= 5.5 AutoProcessor.register now requires the config class, not a
model_type string; processing_gr00t_n1d7.py passed "Gr00tN1d7" -> AttributeError
'str' object has no attribute '__module__' -> pytest collection errors (exit 2).
Pass Gr00tN1d7Config, matching AutoModel.register in modeling/gr00t_n1d7.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added installation of vLLM from a precompiled wheel with a specific commit.
- Set LD_LIBRARY_PATH for CUDA compatibility with bitsandbytes.
- Upgraded flashinfer dependencies and ensured proper installation order for torch to avoid symbol issues.
