Files
vllm-omni/probe/issue-5/UPSTREAM-REPORT-DRAFT.md
T
admin e37c5aed40
Build Wheel / build (3.12) (pull_request) Failing after 36s
Build Wheel / build (3.11) (pull_request) Failing after 49s
pre-commit / pre-commit (pull_request) Failing after 44s
probe(evidence): round two - W8A8 int8-dynamic passes the structure gate at BF16-parity PSNR; verdict flips to YES; upstream Marlin report draft staged
2026-10-08 09:06:49 -04:00

3.7 KiB

Upstream report draft: Marlin 8-bit W8A16 yields NaN on sm_86 while the capability query accepts it

Status: DRAFT — prepared for the #3 upstream lane; file when the owner calls it. Evidence: issue 5 round one (this directory) + PR #4's online-FP8 isolation. Reproduction: probe/issue-5/marlin_isolate.py --synthetic (no checkpoint needed).

Summary

On vLLM 0.31.0 (torch 2.13.0+cu130, CUDA 13.4, RTX 3090 / sm_86, driver 615.71.09), the 8-bit Marlin W8A16 path selected for compressed-tensors int8 weight-only checkpoints produces NaN outputs for every tested layer geometry, activation dtype, and input distribution — including synthetic random weights — even though query_marlin_supported_quant_types accepts uint8b128 on the device and choose_mp_linear_kernel has no other W8A16 candidate on sm_86 (Machete is sm90+, ExLlama int4-only).

Isolation chain (each step independently verified)

  1. Checkpoint exonerated. The int8 weight-only compressed-tensors artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 using compressed-tensors' own unpack_from_int32 + dequantize (verify_artifact.py, output in evidence/rtx3090/artifact-verify.txt).
  2. Loading exonerated. marlin_isolate.py loads the artifact's weight_packed tensors through the production vLLM weight-loader machinery into QKVParallelLinear / RowParallelLinear / ColumnParallelLinear built exactly as the qwen_image_21 transformer builds them; after loading, the parameters are bitwise-equal to the checkpoint (including the fused q/k/v stacked shards), and scales are finite before and after process_weights_after_loading.
  3. Kernel indicted. layer(x) is NaN for all four layer geometries (fused 12288-out QKV, 4096-out row, 12288-out column, wide-in row), for both bf16 and fp16 activations, at M=512.
  4. Checkpoint-independence. --synthetic (random int8 weights + random per-channel scales, quantized and packed in-memory) reproduces the NaN identically — no checkpoint, quantizer, or scale file involved.
  5. Same-family serve evidence. PR #4's online-FP8 path (also Marlin W8A16 on sm_86) produces structurally broken output (≈97% transparent RGBA, PSNR 7 dB vs BF16) across every reference count; transformer-only FP8 isolates the collapse to the Marlin-quantized DiT. Two independent quantizers, one kernel family.

Suspicion boundary (what is and is not pinned)

The NaN arises between gptq_marlin_repack/marlin_permute_scales and torch.ops.vllm.marlin_gemm — the isolate stops at that boundary. Not yet dissected: whether the repacked weight bytes are wrong (repack/layout bug for 8-bit on sm_86) or the kernel itself miscomputes from correct bytes. The 4-bit Marlin path was not tested (out of scope for issue 5).

Minimal reproduction (from this repo)

python probe/issue-5/marlin_isolate.py --synthetic   # NaN, all geometries

Requires: vLLM 0.31.0 importable with CUDA visible; ~2 GiB GPU memory. Expected on an unaffected stack: rel-L2 ≈ 0.01 per layer. Observed on the affected stack: NaN for every layer, both activation dtypes (evidence/rtx3090/marlin-isolate-synthetic.json).

Context that may matter upstream

  • torch 2.13.0+cu130 / CUDA 13.4 is a bleeding-edge combination; if CI covers Marlin int8 on sm_80/86 with an older stack, a torch/CUDA 13 interaction is a candidate.
  • The 4-bit Marlin path is heavily exercised by CI; the 8-bit (uint8b128) path is comparatively new. If 8-bit Marlin support was validated only on sm_80/sm_90, sm_86 may be untested.
  • The capability query accepting uint8b128 on sm_86 while the kernel yields NaN is the dispatch-overclaim angle: either the query or the kernel is wrong for this cell.