Newer upstream vLLM (65a7b4628409, 585 commits past the v0.24.0 tag that
vllm-omni main tracks) reworked online quantization: online-quant linear
methods now declare `uses_meta_device=True` and create weights on the
`meta` device, materializing them just-in-time via the layerwise
online-process loader as each layer's weights finish loading.
The diffusion loader's `_process_weights_after_loading` did a blanket
`module.to(target_device)`, which raises
`NotImplementedError: Cannot copy out of meta tensor; no data!` on any
layer whose weights were not fully materialized during load (padded /
partially-loaded "straggler" layers), crashing diffusion worker startup.
This regressed the nightly diffusion FP8 tests on dev/vllm-align (all
green on main nightly): Quantization Test (H100 + L4,
test_single_stage_qwen_image_fp8 / test_bagel_fp8_generates_image) and
the Qwen-Image Multi-GPU vae_patch_parallel_2 case (fp8).
Mirror upstream's loader contract:
- Materialize online-quant stragglers via finalize_layerwise_processing()
before processing (imported lazily, only when online quant is in use,
so older vLLM is unaffected). model_config=None is safe: it is only
used for vLLM Attention/MLAAttention layers, which diffusion DiT models
do not have.
- Replace the blanket module.to() with a per-parameter move that mirrors
upstream's device_loading_context: move only real (non-meta) params to
the target device for processing and restore them afterward, leaving
meta params for the quant method / finalize step.
Co-Authored-By: Claude <noreply@anthropic.com>
Dockerfile.ci build failed at the nvjitlink step:
uv pip install --system "nvidia-cuda-nvjitlink-cu13"
-> No solution found ... nvidia-cuda-nvjitlink-cu13 was not found in the registry
That package name does not exist (spurious 'cuda-'; real wheel is
nvidia-nvjitlink-cu13), and the install is redundant: the cu130 torch
force-reinstall one step above already pulls nvidia-nvjitlink==13.0.88 into
site-packages, which provides libnvJitLink.so.13. The step only needs to
register that lib dir with ldconfig.
Locate the already-installed .so.13 and ldconfig-register it; fall back to a
correctly-named nvidia-nvjitlink-cu13 install only if torch deps did not bring
it in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three regressions surfaced only on dev/vllm-align in Buildkite build 2520
(26 failing jobs vs 5 on main); ~15 of the 19 align-unique failures trace to these:
1. numpy 2.5 vs numba: Dockerfile.ci pinned numpy==2.2.6 before the flashinfer
and torch cu130 reinstalls, which resolved numpy back to 2.5 and broke
'import numba' (needs numpy <= 2.4). Re-pin numpy as the final dependency op.
2. libnvJitLink.so.13 not found: the CUDA-13 JIT linker lives in the
nvidia-cuda-nvjitlink-cu13 wheel, not /usr/local/cuda (CUDA 12) as the old
LD_LIBRARY_PATH assumed; bitsandbytes/torch.compile subprocesses could not
dlopen it (broke Entrypoint L4, a multi-GPU diffusion job, both Quantization
jobs via hang/timeout). Install the wheel and register its lib dir via ldconfig.
3. transformers >= 5.5 AutoProcessor.register now requires the config class, not a
model_type string; processing_gr00t_n1d7.py passed "Gr00tN1d7" -> AttributeError
'str' object has no attribute '__module__' -> pytest collection errors (exit 2).
Pass Gr00tN1d7Config, matching AutoModel.register in modeling/gr00t_n1d7.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added installation of vLLM from a precompiled wheel with a specific commit.
- Set LD_LIBRARY_PATH for CUDA compatibility with bitsandbytes.
- Upgraded flashinfer dependencies and ensured proper installation order for torch to avoid symbol issues.