Int8 weight-only compressed-tensors DiT artifact + serve-path fixes (issue 5) #6

2026-10-08T08:34:15-04:00

anodyine commented

2026-10-08 08:34:15 -04:00

Summary

Implements issue 5, both rounds: serve pre-quantized int8 DiT checkpoints via
compressed-tensors on the RTX 3090, structure-gated capacity matrix and
quality read each round.

Final verdict (round two, per the owner's amended directive): YES — the
W8A8 int8-dynamic artifact (cutlass int8 scaled-MM, no Marlin) delivers 1–4
reference capacity at the fastest speed measured, with BF16-parity quality:
walls 20.5/25.1/29.4/34.0 s per reference count, peak 23,879 MiB, all 16
matrix requests structurally sound, PSNR vs the sealed Diffusers baseline
33.7/28.3/27.5 dB at 1/2/4 refs (BF16 band 26–29 dB) — the first usable
4-reference quality comparison this card has produced.

Round one (int8 weight-only, pack-quantized) established the capacity
mechanics but collapsed in the kernel: Marlin W8A16 on sm_86 emits NaN even
for synthetic random int8 weights with bitwise-verified loading — the same
kernel family as PR #4's online-FP8 collapse. That convergence (two
independent quantizers, one kernel family) is recorded on PR #4, and a
ready-to-file upstream report is staged at probe/issue-5/UPSTREAM-REPORT-DRAFT.md
for the #3 lane.

Two serve-path fixes were required to make any compressed-tensors artifact
servable under CPU offload (both in this PR, unit-tested):

vllm_omni/quantization/factory.py — the quant factory could not build
compressed-tensors at all (hyphen/underscore registry miss + missing
from_config fallback); checkpoint auto-detection and the CLI both failed.
vllm_omni/diffusion/model_loader/diffusers_loader.py (+is_prequantized_config
in component_config.py) — pre-quantized checkpoints were misclassified as
online quantization under CPU offload, loading everything on the GPU
(init OOM). Serialized formats now load on the host, round-tripping each
quantized module through the accelerator only for post-load transforms.

Artifacts live outside Git under the helper state root with digest manifests:
qwen-image-2.1-dit-int8wo (round one, Marlin evidence) and
qwen-image-2.1-dit-w8a8dyn (round two, the working path).

sealed BF16 snapshot ──quantize──▶ int8 artifacts ──vllm serve──▶ structure-gated matrix
   int8-wo (Marlin W8A16):    capacity ✓ speed ✓  output ✗ (NaN kernel — isolated, synthetic repro)
   W8A8-dynamic (cutlass):    capacity ✓ speed ✓✓ output ✓ (27.5 dB @ 4 refs, BF16 parity)

Evidence

Round one (Marlin broken): artifact round-trip ≈1.1% rel-L2 vs sealed
BF16 (evidence/rtx3090/artifact-verify.txt); serve loads bitwise-identically;
marlin_isolate.py NaN for real and synthetic int8, all layer geometries,
bf16 and fp16 activations (evidence/rtx3090/marlin-isolate-*.json); matrix
gate FAIL 16/16 at 7.1–7.5 dB.
Round two (cutlass works): raw ops + full layer path at 0.17% rel-L2
(evidence/rtx3090-round2/cutlass-ops-probe.json, w8a8-layer-isolate.json);
matrix 16/16 sound, PSNR 33.7/28.3/27.5 dB, walls/VRAM in the table above
(evidence/rtx3090-round2/w8a8dyn-*). Visual checks: 4-ref apothecary scene
and 1-ref portrait render faithfully.
Full record: probe/issue-5/FINDINGS.md (both rounds, gate-calibration note,
kernel-isolation chain).

Checks run

python -m pytest tests/quantization/test_quant_factory.py -q          # 7 passed
python -m pytest tests/diffusion/quantization/ tests/config/test_omni_config.py -q

Observed: 12 failed, 790 passed, 18 skipped on the branch — the same 12
fail on main on this host (re-run to confirm; GPU-memory-bound fp8/gguf
tests on the 24 GB card), so no regressions from this change.

Documentation disposition

updated — probe/issue-5/FINDINGS.md records both rounds' verdicts, the
run matrices, the kernel-isolation chains, and the gate-calibration note;
helper-system follow-through for multi-model-helper-server #27 is input
only, per the issue's scope.

Merge Danger

Door: two-way. Both fixes are behavior-narrow (name-resolution fallback;
offline-detection for serialized formats) and revert cleanly; the probe kit
and artifacts are additive/independent.

Blast Radius: the quant factory's method resolution (all dict/CLI quant
specs) and the diffusion loader's CPU-offload load-path selection for
quantized checkpoints (compressed-tensors/AutoRound-style artifacts; online
fp8/int8 diffusion paths keep their GPU-side load-quantize-offload flow
unchanged).

Rollback: revert the two fix commits; the artifacts are independent of
the repo and nothing else depends on the new helper.

Reviewer prerequisites

RTX 3090-class host with the validation environment (issue 5's standing
venv/clone/snapshot) to reproduce serve numbers; the unit tests alone need
no GPU beyond what CI provides.
Sealed snapshot Qwen/Qwen-Image-2.1@790c9263 under the helper state root
for artifact rebuild/verification.
Duration: unit tests ~1 min; full reproduction (quantize + serve + matrix)
~35 min per scheme.

Pre-Test Setup

git fetch origin
git switch ISSUE-5
git pull --ff-only origin ISSUE-5
python -m pytest tests/quantization/test_quant_factory.py -q

Expected: 7 passed.

Manual Test Plan

Action: rebuild the working artifact and verify it numerically:

ssh anodyine@192.168.0.39
source ~/Projects/vllm-omni-val/venv/bin/activate
cd ~/Projects/vllm-omni-val/vllm-omni && git switch ISSUE-5
python probe/issue-5/quantize_dit_int8wo.py --scheme w8a8dyn \
  --snapshot ~/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758 \
  --out /tmp/qi21-w8a8dyn-review

Expected: per-tensor rel_l2 ≈ 0.009–0.012, 224 quantized tensors.
Stop if the error is large.

Action: serve and probe one 4-reference request (GPU free):

ARTIFACT=/tmp/qi21-w8a8dyn-review systemd-run --user --unit=issue5-r2 \
  ~/Projects/vllm-omni-val/issue5/serve.sh review-r2
# wait for health: curl -s -o /dev/null -w %{http_code} http://127.0.0.1:8091/health
cd ~/Projects/vllm-omni-val/issue5
python probe.py --out runs/review-r2 --single --fixture edit-compose-four --seeds 2301
python psnr.py --fixture edit-compose-four runs/review-r2
systemctl --user stop issue5-r2

Expected: HTTP 200 in ~34 s; structure metrics opaque ≈0.97,
alpha_mean ≈255; PSNR ≈27–28 dB vs the sealed baseline; the rendered PNG
shows the four-character apothecary scene. Stop and retain logs if PSNR
falls in the single digits or opacity collapses (that would be the broken
Marlin signature, meaning the artifact was built with the wrong scheme).

Action (optional, the negative control): reproduce the Marlin
defect with the synthetic isolation:
```
python marlin_isolate.py --artifact ~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo --synthetic
```
Expected: NaN for every layer — the upstream-report reproduction.

Failure and recovery

Retain runs/<label>/serve.log, vram.log, and the PNGs for any failed
attempt; do not record a run as passing on structure metrics alone — the
visual check and PSNR against the sealed baseline are part of the gate.
Artifact rebuilds are idempotent into fresh directories only. If init OOMs,
check nvidia-smi for stray DiffusionWorker processes before concluding a
loader regression.

Post-Merge Cleanup

git switch main
git pull --ff-only origin main
git branch -d ISSUE-5
git push origin --delete ISSUE-5

## Summary Implements issue 5, both rounds: serve pre-quantized int8 DiT checkpoints via compressed-tensors on the RTX 3090, structure-gated capacity matrix and quality read each round. **Final verdict (round two, per the owner's amended directive): YES — the W8A8 int8-dynamic artifact (cutlass int8 scaled-MM, no Marlin) delivers 1–4 reference capacity at the fastest speed measured, with BF16-parity quality:** walls 20.5/25.1/29.4/34.0 s per reference count, peak 23,879 MiB, all 16 matrix requests structurally sound, PSNR vs the sealed Diffusers baseline 33.7/28.3/**27.5 dB** at 1/2/4 refs (BF16 band 26–29 dB) — the first usable 4-reference quality comparison this card has produced. Round one (int8 weight-only, pack-quantized) established the capacity mechanics but collapsed in the kernel: Marlin W8A16 on sm_86 emits NaN even for synthetic random int8 weights with bitwise-verified loading — the same kernel family as PR #4's online-FP8 collapse. That convergence (two independent quantizers, one kernel family) is recorded on PR #4, and a ready-to-file upstream report is staged at `probe/issue-5/UPSTREAM-REPORT-DRAFT.md` for the #3 lane. Two serve-path fixes were required to make any compressed-tensors artifact servable under CPU offload (both in this PR, unit-tested): 1. `vllm_omni/quantization/factory.py` — the quant factory could not build compressed-tensors at all (hyphen/underscore registry miss + missing `from_config` fallback); checkpoint auto-detection and the CLI both failed. 2. `vllm_omni/diffusion/model_loader/diffusers_loader.py` (+`is_prequantized_config` in `component_config.py`) — pre-quantized checkpoints were misclassified as online quantization under CPU offload, loading everything on the GPU (init OOM). Serialized formats now load on the host, round-tripping each quantized module through the accelerator only for post-load transforms. Artifacts live outside Git under the helper state root with digest manifests: `qwen-image-2.1-dit-int8wo` (round one, Marlin evidence) and `qwen-image-2.1-dit-w8a8dyn` (round two, the working path). ``` sealed BF16 snapshot ──quantize──▶ int8 artifacts ──vllm serve──▶ structure-gated matrix int8-wo (Marlin W8A16): capacity ✓ speed ✓ output ✗ (NaN kernel — isolated, synthetic repro) W8A8-dynamic (cutlass): capacity ✓ speed ✓✓ output ✓ (27.5 dB @ 4 refs, BF16 parity) ``` ## Evidence - **Round one (Marlin broken):** artifact round-trip ≈1.1% rel-L2 vs sealed BF16 (`evidence/rtx3090/artifact-verify.txt`); serve loads bitwise-identically; `marlin_isolate.py` NaN for real and synthetic int8, all layer geometries, bf16 and fp16 activations (`evidence/rtx3090/marlin-isolate-*.json`); matrix gate FAIL 16/16 at 7.1–7.5 dB. - **Round two (cutlass works):** raw ops + full layer path at 0.17% rel-L2 (`evidence/rtx3090-round2/cutlass-ops-probe.json`, `w8a8-layer-isolate.json`); matrix 16/16 sound, PSNR 33.7/28.3/27.5 dB, walls/VRAM in the table above (`evidence/rtx3090-round2/w8a8dyn-*`). Visual checks: 4-ref apothecary scene and 1-ref portrait render faithfully. - Full record: `probe/issue-5/FINDINGS.md` (both rounds, gate-calibration note, kernel-isolation chain). ### Checks run ```sh python -m pytest tests/quantization/test_quant_factory.py -q # 7 passed python -m pytest tests/diffusion/quantization/ tests/config/test_omni_config.py -q ``` Observed: `12 failed, 790 passed, 18 skipped` on the branch — the **same 12** fail on `main` on this host (re-run to confirm; GPU-memory-bound fp8/gguf tests on the 24 GB card), so no regressions from this change. ## Documentation disposition updated — `probe/issue-5/FINDINGS.md` records both rounds' verdicts, the run matrices, the kernel-isolation chains, and the gate-calibration note; helper-system follow-through for multi-model-helper-server #27 is input only, per the issue's scope. ## Merge Danger **Door:** two-way. Both fixes are behavior-narrow (name-resolution fallback; offline-detection for serialized formats) and revert cleanly; the probe kit and artifacts are additive/independent. **Blast Radius:** the quant factory's method resolution (all dict/CLI quant specs) and the diffusion loader's CPU-offload load-path selection for quantized checkpoints (compressed-tensors/AutoRound-style artifacts; online fp8/int8 diffusion paths keep their GPU-side load-quantize-offload flow unchanged). **Rollback:** revert the two fix commits; the artifacts are independent of the repo and nothing else depends on the new helper. ## Reviewer prerequisites - RTX 3090-class host with the validation environment (issue 5's standing venv/clone/snapshot) to reproduce serve numbers; the unit tests alone need no GPU beyond what CI provides. - Sealed snapshot `Qwen/Qwen-Image-2.1@790c9263` under the helper state root for artifact rebuild/verification. - Duration: unit tests ~1 min; full reproduction (quantize + serve + matrix) ~35 min per scheme. ## Pre-Test Setup ```sh git fetch origin git switch ISSUE-5 git pull --ff-only origin ISSUE-5 python -m pytest tests/quantization/test_quant_factory.py -q ``` Expected: `7 passed`. ## Manual Test Plan 1. [ ] **Action:** rebuild the working artifact and verify it numerically: ```sh ssh anodyine@192.168.0.39 source ~/Projects/vllm-omni-val/venv/bin/activate cd ~/Projects/vllm-omni-val/vllm-omni && git switch ISSUE-5 python probe/issue-5/quantize_dit_int8wo.py --scheme w8a8dyn \ --snapshot ~/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758 \ --out /tmp/qi21-w8a8dyn-review ``` **Expected:** per-tensor `rel_l2` ≈ 0.009–0.012, 224 quantized tensors. Stop if the error is large. 2. [ ] **Action:** serve and probe one 4-reference request (GPU free): ```sh ARTIFACT=/tmp/qi21-w8a8dyn-review systemd-run --user --unit=issue5-r2 \ ~/Projects/vllm-omni-val/issue5/serve.sh review-r2 # wait for health: curl -s -o /dev/null -w %{http_code} http://127.0.0.1:8091/health cd ~/Projects/vllm-omni-val/issue5 python probe.py --out runs/review-r2 --single --fixture edit-compose-four --seeds 2301 python psnr.py --fixture edit-compose-four runs/review-r2 systemctl --user stop issue5-r2 ``` **Expected:** HTTP 200 in ~34 s; structure metrics opaque ≈0.97, alpha_mean ≈255; PSNR ≈27–28 dB vs the sealed baseline; the rendered PNG shows the four-character apothecary scene. Stop and retain logs if PSNR falls in the single digits or opacity collapses (that would be the broken Marlin signature, meaning the artifact was built with the wrong scheme). 3. [ ] **Action (optional, the negative control):** reproduce the Marlin defect with the synthetic isolation: ```sh python marlin_isolate.py --artifact ~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo --synthetic ``` **Expected:** NaN for every layer — the upstream-report reproduction. ## Failure and recovery Retain `runs/<label>/serve.log`, `vram.log`, and the PNGs for any failed attempt; do not record a run as passing on structure metrics alone — the visual check and PSNR against the sealed baseline are part of the gate. Artifact rebuilds are idempotent into fresh directories only. If init OOMs, check `nvidia-smi` for stray DiffusionWorker processes before concluding a loader regression. ## Post-Merge Cleanup ```sh git switch main git pull --ff-only origin main git branch -d ISSUE-5 git push origin --delete ISSUE-5 ```

anodyine added 6 commits 2026-10-08 08:34:16 -04:00

fix(quant): resolve hyphenated registry methods and disk-dict from_config in quant factory 2f8363c5fc

_normalize_method_name canonicalizes dashes to underscores for _OVERRIDES
lookup, but vLLM's QUANTIZATION_METHODS keeps historical spellings
('compressed-tensors'), so both --quantization compressed-tensors and
checkpoint auto-detection raised 'Unknown quantization method'. Resolve
the normalized name back to the registry spelling, and fall back to
from_config for registry configs whose __init__ does not take the
serialized schema (compressed-tensors config_groups/format/ignore).

probe: int8 weight-only compressed-tensors DiT artifact kit (issue 5) e778d21f91

quantize_dit_int8wo.py builds the offline W8A16 artifact (per-channel
symmetric int8, pack-quantized via compressed-tensors' own packer,
per-component config wrapper so encoder/VAE stay BF16, digest manifest);
serve.sh/probe.py/gate.py/psnr.py adapt the issue-2 kit for the
structure-gated capacity matrix (1-4 refs x seeds 2301-2304).

fix(quant): load offline-quantized compressed-tensors checkpoints directly on CPU under offload 321e8bb47f

The diffusion loader treated any quant config without the MXFP8/
is_checkpoint_quantized markers as online quantization: it loaded every
component on the accelerator before offloading, so serving the int8
weight-only compressed-tensors artifact OOM'd at model construction
(encoder 17 GiB + VAE + packed DiT skeleton > 24 GiB). compressed-tensors
is always a serialized format with no on-load quantization pass — detect
it (through component wrappers too) and load on the host, round-tripping
each quantized module through the accelerator only for the post-load
Marlin repack, which has no CPU path.

probe: track issue-5 fixtures.json (ignore exception, matches issue-2 kit) 67e177860a

probe: artifact round-trip verifier (unpack+dequant vs sealed BF16) 74b596cd05

probe(evidence): int8-wo capacity matrix, structure gate, PSNR, Marlin 8-bit NaN isolation (issue 5)

Build Wheel / build (3.11) (pull_request) Failing after 42s

Details

Build Wheel / build (3.12) (pull_request) Failing after 42s

Details

pre-commit / pre-commit (pull_request) Failing after 37s

Details

bf65a41ceb

anodyine referenced this pull request

2026-10-08 08:34:26 -04:00