247 lines
14 KiB
Markdown
247 lines
14 KiB
Markdown
# Issue 5 probe: int8 quantized DiT serve paths on the RTX 3090 (24 GB)
|
||
|
||
## Round two (owner directive + amendment): W8A8 int8-dynamic — the capacity route is YES
|
||
|
||
The 08:37 amendment narrowed the remaining work to one decisive experiment:
|
||
a compressed-tensors **W8A8 int8-dynamic** artifact (int8 weights + dynamic
|
||
per-token symmetric int8 activations, int-quantized format) whose kernel on
|
||
sm_86 is the cutlass int8 scaled-MM — no Marlin anywhere in the path.
|
||
|
||
**Result: the non-Marlin int8 path works end-to-end.** The artifact serves,
|
||
passes the structure gate on all 16 matrix requests with healthy numbers,
|
||
and lands at BF16-parity PSNR — including the first usable four-reference
|
||
quality comparison this card has ever produced:
|
||
|
||
| Refs | Wall (mean, s) | Peak VRAM | Opacity / alpha | PSNR vs sealed Diffusers (dB) |
|
||
| --- | --- | --- | --- | --- |
|
||
| 1 | 20.5 | 23,879 MiB (run max) | 0.984–0.986 / 255.0 | 33.72 |
|
||
| 2 | 25.1 | ” | 0.947–0.966 / 254.9–255.0 | 28.34 |
|
||
| 3 | 29.4 | ” | 0.965–0.988 / 255.0 | (no sealed row; derived fixture) |
|
||
| 4 | 34.0 | ” | 0.967–0.976 / 255.0 | **27.50** |
|
||
|
||
Comparison rows: BF16 served 1–2 refs at 34.2/36.7 s and OOMs at 4;
|
||
online-FP8 and int8-wo-Marlin run 29.9–46.4 s but produce collapsed output;
|
||
**W8A8-dynamic is both the fastest row measured (4-ref 34.0 s ≈ 4.3× the
|
||
sealed Diffusers nocache baseline's 145.8 s) and the only structurally sound
|
||
quantized row.** 3–4 references at cached speed on one 3090: delivered.
|
||
|
||
Gate calibration note: round one's pass rule (`opaque ≥ 0.99`) was an
|
||
instrument calibration derived from the broken (0.008–0.20) vs BF16 (1.0)
|
||
contrast. The W8A8 outputs sit at 0.95–0.99 opaque with alpha mean ≈255 and
|
||
healthy RGB spread — soft-edge alpha, not the broken signature — so the
|
||
recorded verdict rests on the raw numbers, visual inspection (four-character
|
||
apothecary scene and the 1-ref portrait both render faithfully), and PSNR in
|
||
the 26–29 dB BF16 band. The raw per-request structure metrics are in
|
||
`evidence/rtx3090-round2/` unedited.
|
||
|
||
Kernel-path verification for this round (`w8a8_isolate.py`,
|
||
`cutlass_ops_probe.py`): the raw ops (`scaled_int8_quant` +
|
||
`cutlass_scaled_mm`) match a float32 reference at 0.17% rel-L2 across
|
||
(64,256,128) / (512,4096,4096) / (512,4096,12288), and the full layer path
|
||
(load → transpose-repack → apply, fused QKV and row-parallel, synthetic
|
||
weights, loaded bitwise-verified) matches at 0.17% too. (Round-two lesson
|
||
recorded for the instrument: an intermediate 97%-error scare in
|
||
`w8a8_isolate.py` was a missing ×act_scale factor in the probe's own
|
||
reference, not a kernel defect — the out/ref norm ratio equaled E[act_scale]
|
||
exactly, which is what gave it away.)
|
||
|
||
Artifacts (outside Git): `…/artifacts/qwen-image-2.1-dit-w8a8dyn/` (int8
|
||
weights unpacked + fp32 per-channel scales, 224 tensors, 7.26 GiB, manifest
|
||
with digests). The round-one int8-wo artifact stays for the Marlin evidence.
|
||
|
||
The upstream Marlin report is drafted and ready to file in the #3 lane on
|
||
the owner's call: `UPSTREAM-REPORT-DRAFT.md` (capability-query overclaim +
|
||
NaN kernel + synthetic repro + the PR #4 FP8 convergence). The hard-stop
|
||
branch did not trigger.
|
||
|
||
Helper-system follow-through for multi-model-helper-server #27 (input only):
|
||
the 3090's serving-profile candidacy now extends to 3–4-reference profiles
|
||
via the W8A8-dynamic artifact — 4-ref at 34.0 s/image, peak 23.9 GiB,
|
||
BF16-parity PSNR (27.5 dB 4-ref vs the sealed int8-DiT Diffusers baseline),
|
||
provided serving ships the artifact and the branch's two loader/factory
|
||
fixes. This supersedes round one's "blocked on Marlin" conclusion.
|
||
|
||
## Round one: int8 weight-only via compressed-tensors — capacity yes, Marlin W8A16 collapses
|
||
|
||
Probed 2026-10-08 on `cachyos-alexander` (RTX 3090, sm_86, driver 615.71.09,
|
||
CUDA 13.4), at upstream main `8484d85d1` plus this branch's two loader/factory
|
||
fixes, vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||
transformers 5.14.1, diffusers git `122b1e11`, compressed-tensors 0.17.0).
|
||
The DiT was quantized **offline** from the sealed snapshot
|
||
`Qwen/Qwen-Image-2.1@790c9263` to int8 weight-only (W8A16), per-output-channel
|
||
symmetric, stored as a compressed-tensors pack-quantized checkpoint; encoder
|
||
and VAE stay BF16 on disk via the artifact's per-component config wrapper.
|
||
The probe mirrors the sealed Diffusers attempt `2026-10-07T170729Z-q3-4r`
|
||
fixture-for-fixture (same frozen references-suite-2 images, same rendered
|
||
prompts, seeds 2301–2304, 20 steps, `true_cfg_scale 1.0`).
|
||
|
||
## Verdict (superseded by round two above — kept as the round-one record)
|
||
|
||
**Capacity: delivered. Quality: collapsed in the serve kernel — the int8
|
||
sidestep does not rescue 3–4 references on this card with this stack, and the
|
||
blocker is now sharply isolated: the 8-bit Marlin W8A16 path itself.**
|
||
|
||
- Every step of the artifact chain is verified correct: the quantizer's
|
||
round-trip reproduces the sealed BF16 weights at ~1.1% rel-L2
|
||
(`evidence/rtx3090/artifact-verify.txt`), and the serve path loads the
|
||
packed tensors **bitwise-identically** — including the fused q/k/v stacked
|
||
shards — with finite scales before and after the Marlin repack.
|
||
- Serving works mechanically: init ~40 s, all 16 capacity-matrix requests
|
||
(1/2/3/4 references × seeds 2301–2304) complete at BF16-comparable speed
|
||
(4-ref 46.4 s; BF16 OOMs at 4 refs, online-FP8 was 44.3 s), peak 23,935 MiB.
|
||
- But the output is structurally broken on **every** counted run (gate FAIL
|
||
16/16: opaque 0.0–0.20 vs 1.0 at BF16; PSNR vs the sealed baseline
|
||
7.1–7.5 dB — the same collapsed band as online FP8). The images are
|
||
horizontal-stripe noise.
|
||
- A one-layer offline isolation (`marlin_isolate.py`) pins it: vLLM parallel
|
||
linears built exactly as the qwen_image_21 transformer builds them, loaded
|
||
with the artifact's verified tensors, emit **NaN** for all four layer
|
||
geometries (fused QKV, to_out, img_mlp proj/out), for both bf16 and fp16
|
||
activations — and do the same for **synthetic random int8 weights**, so the
|
||
checkpoint and its quantization are exonerated. The 8-bit Marlin W8A16 path
|
||
on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) is broken even though
|
||
`query_marlin_supported_quant_types` accepts `uint8b128` on the card, and
|
||
`choose_mp_linear_kernel` offers no non-Marlin W8A16 fallback on sm_86
|
||
(Machete is sm90+, ExLlama int4-only).
|
||
|
||
The torchao-int8-in-Diffusers prior (multi-model-helper-server #160, blinded
|
||
review within label noise of BF16) stands unchallenged: int8 quantization is
|
||
fine; the serve-path kernel is not. The online-FP8 collapse from PR #4 and
|
||
this int8 collapse now share one root suspect — the Marlin mixed-precision
|
||
path on this fork/card — with the fp8 case previously isolated to the
|
||
Marlin-quantized DiT and this run extending it to int8 with a fully verified
|
||
checkpoint on the far side of the kernel.
|
||
|
||
## Two serve-path fixes this issue required (in this PR)
|
||
|
||
1. **Quant factory could not build compressed-tensors at all.**
|
||
`_normalize_method_name` canonicalizes `compressed-tensors` to
|
||
`compressed_tensors`, which then missed vLLM's hyphenated registry — so
|
||
both `--quantization compressed-tensors` and checkpoint auto-detection
|
||
raised "Unknown quantization method". The registry lookup now resolves
|
||
both spellings, and registry configs whose `__init__` does not take the
|
||
serialized schema fall back to `from_config` (`vllm_omni/quantization/factory.py`,
|
||
tests in `tests/quantization/test_quant_factory.py`).
|
||
2. **The diffusion loader misclassified pre-quantized checkpoints as online
|
||
quantization under CPU offload.** With `--enable-cpu-offload` it loaded
|
||
every component on the GPU before offloading (encoder 17 GiB + VAE + the
|
||
packed DiT skeleton > 24 GiB → init OOM). Serialized formats are now
|
||
detected (`is_prequantized_config`, component-wrapper aware) and load
|
||
directly on the host, with each quantized module round-tripped through the
|
||
accelerator only for the post-load Marlin repack
|
||
(`vllm_omni/diffusion/model_loader/diffusers_loader.py`). With these two
|
||
fixes the artifact serves: init healthy in ~40 s, logs
|
||
`Per-component quantization: {'transformer': 'compressed-tensors'}`.
|
||
|
||
## Artifact (outside Git, under the helper state root)
|
||
|
||
`~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo/`
|
||
— quantized `transformer/` (6.76 GiB packed int8 + BF16 boundary layers;
|
||
224 tensors = 32 blocks × {to_q,to_k,to_v,to_out.0,proj,gate_layer,out};
|
||
6.98 B params quantized, 0.14 B BF16), `model_index.json`/`scheduler`/
|
||
`text_encoder`/`vae`/`processor` symlinked to the sealed snapshot, and
|
||
`MANIFEST.json` with inputs (snapshot path, revision, tree digest), versions,
|
||
per-file sha256, and quantization stats (mean rel-L2 1.10%, max 1.73%).
|
||
Built by `quantize_dit_int8wo.py` using compressed-tensors' own
|
||
`pack_to_int32` (uint8 biased by 128, 4 per int32) and `QuantizationConfig`
|
||
serialization, so the storage convention matches llm-compressor bit-for-bit.
|
||
`transformer/config.json` carries the quantization metadata wrapped
|
||
per-component (`{"transformer": {...}}`) so auto-detection quantizes the DiT
|
||
only — the encoder and VAE resolve to None and stay BF16
|
||
(the plain global form would hand the encoder a compressed-tensors config
|
||
it has no scales for).
|
||
|
||
## Run matrix — RTX 3090
|
||
|
||
Single run `int8wo`: `vllm serve <artifact> --omni --port 8091
|
||
--served-model-name Qwen/Qwen-Image-2.1 --enforce-eager
|
||
--enable-cpu-offload`, readiness via `/health`, VRAM via the 1 Hz
|
||
`nvidia-smi` series (issue 2's instrument). Two precursor runs OOM'd at init
|
||
for reasons fixed on this branch (see Verdict item 2); their logs survive
|
||
under `runs/int8wo-expandable/` on the host (the first attempt's log was
|
||
overwritten by the successful rerun in the same label).
|
||
|
||
| Refs | Fixture | Wall (mean, s) | Peak (MiB) | Gate | PSNR vs Diffusers (dB) |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| 1 | edit-expression-ysolde | 31.0 | 23,935 (run max) | FAIL 4/4 | 7.14 |
|
||
| 2 | edit-compose-pair | 36.6 | ” | FAIL 4/4 | 7.46 |
|
||
| 3 | edit-compose-three | 41.4 | ” | FAIL 4/4 | no sealed row (derived fixture) |
|
||
| 4 | edit-compose-four | 46.4 | ” | FAIL 4/4 | 7.11 |
|
||
|
||
BF16 comparison rows (issue 2): 1-ref 34.2 s, 2-ref 36.7 s, 4-ref OOM.
|
||
Online-fp8 (broken output, capacity reference): 29.9 / 34.1 / 38.9 / 44.3 s.
|
||
So the int8 artifact matches the fp8 capacity/speed profile exactly —
|
||
including its broken-output signature (PSNR band 7.0–7.5 dB) — while the
|
||
checkpoint side is proven numerically sound, moving the defect wholly into
|
||
the kernel.
|
||
|
||
## Structure gate and quality read
|
||
|
||
- Gate rule (issue 1's lesson): `opaque_fraction ≥ 0.99` and `mean(rgb_std)
|
||
≥ 10`; BF16 outputs measure 1.0 opaque, 26–29 dB. Every int8-artifact run
|
||
fails (opaque 0.0–0.20, alpha mean 114–137; rgb_std spans 3–93 — several
|
||
outputs have healthy RGB spread yet near-total transparency, and one
|
||
(seed 2302) is near-flat grey like the fp8 signature). Outputs are
|
||
horizontal-stripe noise on visual inspection.
|
||
- PSNR vs the sealed 4-reference Diffusers baseline (RGBA-on-RGBA,
|
||
#1's comparator): 7.14 / 7.46 / 7.11 dB mean for 1/2/4 refs — the
|
||
collapsed band, not the 26–29 dB BF16-row parity the issue hoped for.
|
||
|
||
## Kernel isolation (the issue's "different kernels" prior, answered)
|
||
|
||
`marlin_isolate.py` builds vLLM `QKVParallelLinear` / `RowParallelLinear` /
|
||
`ColumnParallelLinear` exactly as `qwen_image_21_transformer.py` does, loads
|
||
the artifact tensors through the production weight-loader machinery
|
||
(bitwise-verified after load, fused q/k/v shards included), runs
|
||
`process_weights_after_loading` (the Marlin repack) on the GPU, and compares
|
||
`layer(x)` against `float32 x @ dequant(W)ᵀ` computed from the artifact's own
|
||
unpacked weights:
|
||
|
||
| Layer | bf16 acts | fp16 acts | packed bitwise | scales post-repack |
|
||
| --- | --- | --- | --- | --- |
|
||
| attn.to_qkv (fused) | NaN | NaN | equal | finite (6.6e-4…1.2e-2) |
|
||
| attn.to_out.0 (row) | NaN | NaN | equal | finite |
|
||
| img_mlp.proj (column) | NaN | NaN | equal | finite |
|
||
| img_mlp.out (row, wide-in) | NaN | NaN | equal | finite |
|
||
|
||
Synthetic random int8 weights reproduce the NaN identically, and the runtime
|
||
capability query accepts `uint8b128` on this card — so this is a defect of
|
||
the 8-bit Marlin W8A16 path on this stack (repack or kernel; sm_86,
|
||
vllm 0.31.0, torch 2.13.0+cu130), not of the checkpoint, the quantizer, the
|
||
loader, or the fused-QKV stacking. No alternative W8A16 kernel exists on
|
||
sm_86 in this build to switch to.
|
||
|
||
## Reproducibility
|
||
|
||
Scripts in this directory: `quantize_dit_int8wo.py` (artifact + manifest),
|
||
`verify_artifact.py` (round-trip check), `serve.sh` / `probe.py` (adapted
|
||
issue-2 kit; probe records the structure gate inline per request), `gate.py`
|
||
(gate + capacity summarizer), `psnr.py`, `marlin_isolate.py` (kernel
|
||
isolation, `--synthetic` for the stack-only variant). All were pushed
|
||
verbatim to `~/Projects/vllm-omni-val/issue5/` on the host and run from the
|
||
validation clone checked out on this branch. Per-run serve/vram logs and all
|
||
output PNGs live under `~/Projects/vllm-omni-val/issue5/runs/int8wo/`;
|
||
condensed evidence in `evidence/rtx3090/`.
|
||
|
||
## Helper-system follow-through (input to multi-model-helper-server #27 — not implemented here)
|
||
|
||
- The 3090's 3–4 reference serving path remains blocked, now with a single
|
||
named suspect: the 8-bit Marlin W8A16 kernel path on sm_86 (this fork's
|
||
vllm 0.31.0 / torch 2.13.0 stack). Both quantized routes (online FP8,
|
||
offline int8 compressed-tensors with a verified checkpoint) collapse at the
|
||
same place. The unlock order for #27 candidacy is now: upstream Marlin
|
||
8-bit fix (or a non-Marlin W8A16 kernel on sm_86), then serving — capacity
|
||
and speed are already proven sufficient (4-ref at 46.4 s, 2.9× the sealed
|
||
Diffusers nocache baseline).
|
||
- The two vllm-omni loader/factory fixes on this branch are prerequisites for
|
||
any compressed-tensors artifact serving under CPU offload and are
|
||
independent of the kernel defect.
|
||
|
||
## Out-of-scope confirmation
|
||
|
||
No upstream PRs; no W8A8/int4/NVFP4 artifacts built; no helper-system
|
||
deployment evidence. Root-causing the Marlin 8-bit defect remains the
|
||
separate lane (PR #4 carries the fp8 isolation; this branch adds the int8 +
|
||
synthetic + bitwise-load isolation). The only code changes are the two
|
||
serve-path fixes required to make the issue's plan steps executable at all.
|