vllm-omni/probe/issue-5/FINDINGS.md

Issue 5 probe: int8 quantized DiT serve paths on the RTX 3090 (24 GB)

Round two (owner directive + amendment): W8A8 int8-dynamic — the capacity route is YES

The 08:37 amendment narrowed the remaining work to one decisive experiment: a compressed-tensors W8A8 int8-dynamic artifact (int8 weights + dynamic per-token symmetric int8 activations, int-quantized format) whose kernel on sm_86 is the cutlass int8 scaled-MM — no Marlin anywhere in the path.

Result: the non-Marlin int8 path works end-to-end. The artifact serves, passes the structure gate on all 16 matrix requests with healthy numbers, and lands at BF16-parity PSNR — including the first usable four-reference quality comparison this card has ever produced:

Refs	Wall (mean, s)	Peak VRAM	Opacity / alpha	PSNR vs sealed Diffusers (dB)
1	20.5	23,879 MiB (run max)	0.984–0.986 / 255.0	33.72
2	25.1	”	0.947–0.966 / 254.9–255.0	28.34
3	29.4	”	0.965–0.988 / 255.0	(no sealed row; derived fixture)
4	34.0	”	0.967–0.976 / 255.0	27.50

Comparison rows: BF16 served 1–2 refs at 34.2/36.7 s and OOMs at 4; online-FP8 and int8-wo-Marlin run 29.9–46.4 s but produce collapsed output; W8A8-dynamic is both the fastest row measured (4-ref 34.0 s ≈ 4.3× the sealed Diffusers nocache baseline's 145.8 s) and the only structurally sound quantized row. 3–4 references at cached speed on one 3090: delivered.

Gate calibration note: round one's pass rule (opaque ≥ 0.99) was an instrument calibration derived from the broken (0.008–0.20) vs BF16 (1.0) contrast. The W8A8 outputs sit at 0.95–0.99 opaque with alpha mean ≈255 and healthy RGB spread — soft-edge alpha, not the broken signature — so the recorded verdict rests on the raw numbers, visual inspection (four-character apothecary scene and the 1-ref portrait both render faithfully), and PSNR in the 26–29 dB BF16 band. The raw per-request structure metrics are in evidence/rtx3090-round2/ unedited.

Kernel-path verification for this round (w8a8_isolate.py, cutlass_ops_probe.py): the raw ops (scaled_int8_quant + cutlass_scaled_mm) match a float32 reference at 0.17% rel-L2 across (64,256,128) / (512,4096,4096) / (512,4096,12288), and the full layer path (load → transpose-repack → apply, fused QKV and row-parallel, synthetic weights, loaded bitwise-verified) matches at 0.17% too. (Round-two lesson recorded for the instrument: an intermediate 97%-error scare in w8a8_isolate.py was a missing ×act_scale factor in the probe's own reference, not a kernel defect — the out/ref norm ratio equaled E[act_scale] exactly, which is what gave it away.)

Artifacts (outside Git): …/artifacts/qwen-image-2.1-dit-w8a8dyn/ (int8 weights unpacked + fp32 per-channel scales, 224 tensors, 7.26 GiB, manifest with digests). The round-one int8-wo artifact stays for the Marlin evidence.

The upstream Marlin report is drafted and ready to file in the #3 lane on the owner's call: UPSTREAM-REPORT-DRAFT.md (capability-query overclaim + NaN kernel + synthetic repro + the PR #4 FP8 convergence). The hard-stop branch did not trigger.

Helper-system follow-through for multi-model-helper-server #27 (input only): the 3090's serving-profile candidacy now extends to 3–4-reference profiles via the W8A8-dynamic artifact — 4-ref at 34.0 s/image, peak 23.9 GiB, BF16-parity PSNR (27.5 dB 4-ref vs the sealed int8-DiT Diffusers baseline), provided serving ships the artifact and the branch's two loader/factory fixes. This supersedes round one's "blocked on Marlin" conclusion.

Round one: int8 weight-only via compressed-tensors — capacity yes, Marlin W8A16 collapses

Probed 2026-10-08 on cachyos-alexander (RTX 3090, sm_86, driver 615.71.09, CUDA 13.4), at upstream main 8484d85d1 plus this branch's two loader/factory fixes, vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130, transformers 5.14.1, diffusers git 122b1e11, compressed-tensors 0.17.0). The DiT was quantized offline from the sealed snapshot Qwen/Qwen-Image-2.1@790c9263 to int8 weight-only (W8A16), per-output-channel symmetric, stored as a compressed-tensors pack-quantized checkpoint; encoder and VAE stay BF16 on disk via the artifact's per-component config wrapper. The probe mirrors the sealed Diffusers attempt 2026-10-07T170729Z-q3-4r fixture-for-fixture (same frozen references-suite-2 images, same rendered prompts, seeds 2301–2304, 20 steps, true_cfg_scale 1.0).

Verdict (superseded by round two above — kept as the round-one record)

Capacity: delivered. Quality: collapsed in the serve kernel — the int8 sidestep does not rescue 3–4 references on this card with this stack, and the blocker is now sharply isolated: the 8-bit Marlin W8A16 path itself.

• Every step of the artifact chain is verified correct: the quantizer's round-trip reproduces the sealed BF16 weights at ~1.1% rel-L2 (evidence/rtx3090/artifact-verify.txt), and the serve path loads the packed tensors bitwise-identically — including the fused q/k/v stacked shards — with finite scales before and after the Marlin repack.
• Serving works mechanically: init ~40 s, all 16 capacity-matrix requests (1/2/3/4 references × seeds 2301–2304) complete at BF16-comparable speed (4-ref 46.4 s; BF16 OOMs at 4 refs, online-FP8 was 44.3 s), peak 23,935 MiB.
• But the output is structurally broken on every counted run (gate FAIL 16/16: opaque 0.0–0.20 vs 1.0 at BF16; PSNR vs the sealed baseline 7.1–7.5 dB — the same collapsed band as online FP8). The images are horizontal-stripe noise.
• A one-layer offline isolation (marlin_isolate.py) pins it: vLLM parallel linears built exactly as the qwen_image_21 transformer builds them, loaded with the artifact's verified tensors, emit NaN for all four layer geometries (fused QKV, to_out, img_mlp proj/out), for both bf16 and fp16 activations — and do the same for synthetic random int8 weights, so the checkpoint and its quantization are exonerated. The 8-bit Marlin W8A16 path on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) is broken even though query_marlin_supported_quant_types accepts uint8b128 on the card, and choose_mp_linear_kernel offers no non-Marlin W8A16 fallback on sm_86 (Machete is sm90+, ExLlama int4-only).

The torchao-int8-in-Diffusers prior (multi-model-helper-server #160, blinded review within label noise of BF16) stands unchallenged: int8 quantization is fine; the serve-path kernel is not. The online-FP8 collapse from PR #4 and this int8 collapse now share one root suspect — the Marlin mixed-precision path on this fork/card — with the fp8 case previously isolated to the Marlin-quantized DiT and this run extending it to int8 with a fully verified checkpoint on the far side of the kernel.

Two serve-path fixes this issue required (in this PR)

1. Quant factory could not build compressed-tensors at all. _normalize_method_name canonicalizes compressed-tensors to compressed_tensors, which then missed vLLM's hyphenated registry — so both --quantization compressed-tensors and checkpoint auto-detection raised "Unknown quantization method". The registry lookup now resolves both spellings, and registry configs whose __init__ does not take the serialized schema fall back to from_config (vllm_omni/quantization/factory.py, tests in tests/quantization/test_quant_factory.py).
2. The diffusion loader misclassified pre-quantized checkpoints as online quantization under CPU offload. With --enable-cpu-offload it loaded every component on the GPU before offloading (encoder 17 GiB + VAE + the packed DiT skeleton > 24 GiB → init OOM). Serialized formats are now detected (is_prequantized_config, component-wrapper aware) and load directly on the host, with each quantized module round-tripped through the accelerator only for the post-load Marlin repack (vllm_omni/diffusion/model_loader/diffusers_loader.py). With these two fixes the artifact serves: init healthy in ~40 s, logs Per-component quantization: {'transformer': 'compressed-tensors'}.

Artifact (outside Git, under the helper state root)

~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo/ — quantized transformer/ (6.76 GiB packed int8 + BF16 boundary layers; 224 tensors = 32 blocks × {to_q,to_k,to_v,to_out.0,proj,gate_layer,out}; 6.98 B params quantized, 0.14 B BF16), model_index.json/scheduler/ text_encoder/vae/processor symlinked to the sealed snapshot, and MANIFEST.json with inputs (snapshot path, revision, tree digest), versions, per-file sha256, and quantization stats (mean rel-L2 1.10%, max 1.73%). Built by quantize_dit_int8wo.py using compressed-tensors' own pack_to_int32 (uint8 biased by 128, 4 per int32) and QuantizationConfig serialization, so the storage convention matches llm-compressor bit-for-bit. transformer/config.json carries the quantization metadata wrapped per-component ({"transformer": {...}}) so auto-detection quantizes the DiT only — the encoder and VAE resolve to None and stay BF16 (the plain global form would hand the encoder a compressed-tensors config it has no scales for).

Run matrix — RTX 3090

Single run int8wo: vllm serve <artifact> --omni --port 8091 --served-model-name Qwen/Qwen-Image-2.1 --enforce-eager --enable-cpu-offload, readiness via /health, VRAM via the 1 Hz nvidia-smi series (issue 2's instrument). Two precursor runs OOM'd at init for reasons fixed on this branch (see Verdict item 2); their logs survive under runs/int8wo-expandable/ on the host (the first attempt's log was overwritten by the successful rerun in the same label).

Refs	Fixture	Wall (mean, s)	Peak (MiB)	Gate	PSNR vs Diffusers (dB)
1	edit-expression-ysolde	31.0	23,935 (run max)	FAIL 4/4	7.14
2	edit-compose-pair	36.6	”	FAIL 4/4	7.46
3	edit-compose-three	41.4	”	FAIL 4/4	no sealed row (derived fixture)
4	edit-compose-four	46.4	”	FAIL 4/4	7.11

BF16 comparison rows (issue 2): 1-ref 34.2 s, 2-ref 36.7 s, 4-ref OOM. Online-fp8 (broken output, capacity reference): 29.9 / 34.1 / 38.9 / 44.3 s. So the int8 artifact matches the fp8 capacity/speed profile exactly — including its broken-output signature (PSNR band 7.0–7.5 dB) — while the checkpoint side is proven numerically sound, moving the defect wholly into the kernel.

Structure gate and quality read

• Gate rule (issue 1's lesson): opaque_fraction ≥ 0.99 and mean(rgb_std) ≥ 10; BF16 outputs measure 1.0 opaque, 26–29 dB. Every int8-artifact run fails (opaque 0.0–0.20, alpha mean 114–137; rgb_std spans 3–93 — several outputs have healthy RGB spread yet near-total transparency, and one (seed 2302) is near-flat grey like the fp8 signature). Outputs are horizontal-stripe noise on visual inspection.
• PSNR vs the sealed 4-reference Diffusers baseline (RGBA-on-RGBA, #1's comparator): 7.14 / 7.46 / 7.11 dB mean for 1/2/4 refs — the collapsed band, not the 26–29 dB BF16-row parity the issue hoped for.

Kernel isolation (the issue's "different kernels" prior, answered)

marlin_isolate.py builds vLLM QKVParallelLinear / RowParallelLinear / ColumnParallelLinear exactly as qwen_image_21_transformer.py does, loads the artifact tensors through the production weight-loader machinery (bitwise-verified after load, fused q/k/v shards included), runs process_weights_after_loading (the Marlin repack) on the GPU, and compares layer(x) against float32 x @ dequant(W)ᵀ computed from the artifact's own unpacked weights:

Layer	bf16 acts	fp16 acts	packed bitwise	scales post-repack
attn.to_qkv (fused)	NaN	NaN	equal	finite (6.6e-4…1.2e-2)
attn.to_out.0 (row)	NaN	NaN	equal	finite
img_mlp.proj (column)	NaN	NaN	equal	finite
img_mlp.out (row, wide-in)	NaN	NaN	equal	finite

Synthetic random int8 weights reproduce the NaN identically, and the runtime capability query accepts uint8b128 on this card — so this is a defect of the 8-bit Marlin W8A16 path on this stack (repack or kernel; sm_86, vllm 0.31.0, torch 2.13.0+cu130), not of the checkpoint, the quantizer, the loader, or the fused-QKV stacking. No alternative W8A16 kernel exists on sm_86 in this build to switch to.

Reproducibility

Scripts in this directory: quantize_dit_int8wo.py (artifact + manifest), verify_artifact.py (round-trip check), serve.sh / probe.py (adapted issue-2 kit; probe records the structure gate inline per request), gate.py (gate + capacity summarizer), psnr.py, marlin_isolate.py (kernel isolation, --synthetic for the stack-only variant). All were pushed verbatim to ~/Projects/vllm-omni-val/issue5/ on the host and run from the validation clone checked out on this branch. Per-run serve/vram logs and all output PNGs live under ~/Projects/vllm-omni-val/issue5/runs/int8wo/; condensed evidence in evidence/rtx3090/.

Helper-system follow-through (input to multi-model-helper-server #27 — not implemented here)

• The 3090's 3–4 reference serving path remains blocked, now with a single named suspect: the 8-bit Marlin W8A16 kernel path on sm_86 (this fork's vllm 0.31.0 / torch 2.13.0 stack). Both quantized routes (online FP8, offline int8 compressed-tensors with a verified checkpoint) collapse at the same place. The unlock order for #27 candidacy is now: upstream Marlin 8-bit fix (or a non-Marlin W8A16 kernel on sm_86), then serving — capacity and speed are already proven sufficient (4-ref at 46.4 s, 2.9× the sealed Diffusers nocache baseline).
• The two vllm-omni loader/factory fixes on this branch are prerequisites for any compressed-tensors artifact serving under CPU offload and are independent of the kernel defect.

Out-of-scope confirmation

No upstream PRs; no W8A8/int4/NVFP4 artifacts built; no helper-system deployment evidence. Root-causing the Marlin 8-bit defect remains the separate lane (PR #4 carries the fp8 isolation; this branch adds the int8 + synthetic + bitwise-load isolation). The only code changes are the two serve-path fixes required to make the issue's plan steps executable at all.