3.7 KiB
Upstream report draft: Marlin 8-bit W8A16 yields NaN on sm_86 while the capability query accepts it
Status: DRAFT — prepared for the #3 upstream lane; file when the owner calls it.
Evidence: issue 5 round one (this directory) + PR #4's online-FP8 isolation.
Reproduction: probe/issue-5/marlin_isolate.py --synthetic (no checkpoint needed).
Summary
On vLLM 0.31.0 (torch 2.13.0+cu130, CUDA 13.4, RTX 3090 / sm_86, driver
615.71.09), the 8-bit Marlin W8A16 path selected for compressed-tensors
int8 weight-only checkpoints produces NaN outputs for every tested layer
geometry, activation dtype, and input distribution — including synthetic
random weights — even though query_marlin_supported_quant_types accepts
uint8b128 on the device and choose_mp_linear_kernel has no other W8A16
candidate on sm_86 (Machete is sm90+, ExLlama int4-only).
Isolation chain (each step independently verified)
- Checkpoint exonerated. The int8 weight-only compressed-tensors
artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 using
compressed-tensors' own
unpack_from_int32+ dequantize (verify_artifact.py, output inevidence/rtx3090/artifact-verify.txt). - Loading exonerated.
marlin_isolate.pyloads the artifact'sweight_packedtensors through the production vLLM weight-loader machinery intoQKVParallelLinear/RowParallelLinear/ColumnParallelLinearbuilt exactly as the qwen_image_21 transformer builds them; after loading, the parameters are bitwise-equal to the checkpoint (including the fused q/k/v stacked shards), and scales are finite before and afterprocess_weights_after_loading. - Kernel indicted.
layer(x)is NaN for all four layer geometries (fused 12288-out QKV, 4096-out row, 12288-out column, wide-in row), for both bf16 and fp16 activations, at M=512. - Checkpoint-independence.
--synthetic(random int8 weights + random per-channel scales, quantized and packed in-memory) reproduces the NaN identically — no checkpoint, quantizer, or scale file involved. - Same-family serve evidence. PR #4's online-FP8 path (also Marlin W8A16 on sm_86) produces structurally broken output (≈97% transparent RGBA, PSNR 7 dB vs BF16) across every reference count; transformer-only FP8 isolates the collapse to the Marlin-quantized DiT. Two independent quantizers, one kernel family.
Suspicion boundary (what is and is not pinned)
The NaN arises between gptq_marlin_repack/marlin_permute_scales and
torch.ops.vllm.marlin_gemm — the isolate stops at that boundary. Not yet
dissected: whether the repacked weight bytes are wrong (repack/layout bug
for 8-bit on sm_86) or the kernel itself miscomputes from correct bytes.
The 4-bit Marlin path was not tested (out of scope for issue 5).
Minimal reproduction (from this repo)
python probe/issue-5/marlin_isolate.py --synthetic # NaN, all geometries
Requires: vLLM 0.31.0 importable with CUDA visible; ~2 GiB GPU memory.
Expected on an unaffected stack: rel-L2 ≈ 0.01 per layer.
Observed on the affected stack: NaN for every layer, both activation dtypes
(evidence/rtx3090/marlin-isolate-synthetic.json).
Context that may matter upstream
- torch 2.13.0+cu130 / CUDA 13.4 is a bleeding-edge combination; if CI covers Marlin int8 on sm_80/86 with an older stack, a torch/CUDA 13 interaction is a candidate.
- The 4-bit Marlin path is heavily exercised by CI; the 8-bit
(
uint8b128) path is comparatively new. If 8-bit Marlin support was validated only on sm_80/sm_90, sm_86 may be untested. - The capability query accepting
uint8b128on sm_86 while the kernel yields NaN is the dispatch-overclaim angle: either the query or the kernel is wrong for this cell.