anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Serve a pre-quantized int8 weight-only DiT via compressed-tensors for 3-4 reference capacity on the RTX 3090 #5

New Issue
Open
opened 2026-10-08 06:34:36 -04:00 by anodyine · 4 comments
anodyine commented 2026-10-08 06:34:36 -04:00
Owner
Copy link
Copy source

Context

PR #4 established two things on the RTX 3090 (24 GB): the BF16 serve path tops out at two references cached (~36.7 s), and the online-FP8 path — while it frees exactly the needed ~6 GiB and served all four reference counts (4-ref 44.1-44.4 s, peak 23,522 MB) — produces structurally broken outputs (~97% transparent RGBA, near-flat grey, 7-9 dB), isolated to the Marlin-quantized DiT itself (transformer-only FP8 breaks it; encoder BF16 does not; --ignored-layers inert). Root-causing the Marlin path is a separate lane. This issue takes the sidestep: quantize the DiT offline to int8 weight-only, serve the artifact, and get the 3-4 reference capacity without touching the suspect online-FP8 code path.

Why int8 weight-only has the strong prior: the helper-system already quality-validated a torchao int8 DiT in Diffusers — Q3's blinded review placed it within label noise of the BF16 baseline (multi-model-helper-server #160) — and compressed-tensors is vLLM's most battle-tested quantized-checkpoint route. Same memory win as FP8 (~13.25 -> ~7 GiB DiT), different quantizer, different kernels.

Host and environment facts

  • Host: cachyos-alexander at 192.168.0.39, user anodyine, key-based SSH from the workstation (ssh anodyine@192.168.0.39); user lingering on for systemd-run --user units. RTX 3090 24 GB (sm_86), driver 615.71.09, CUDA 13.4.
  • GPU caution: the host's relayview-inference.service (a vLLM worker that grabs the whole GPU at boot) is owner-controlled and its enabled state varies between sessions — check nvidia-smi first and ask the owner if it holds the card; do not stop it yourself.
  • Environment (standing from #1/#2): Python 3.13 venv at ~/Projects/vllm-omni-val/venv over the clone at ~/Projects/vllm-omni-val/vllm-omni (origin = this mirror, upstream = GitHub; main 8484d85d1). Installed: vllm 0.31.0, torch 2.13.0+cu130, transformers 5.14.1, diffusers git 122b1e11, repo editable, requirements/common.txt runtime deps.
  • Reusable probe kit from #2: ~/Projects/vllm-omni-val/issue2/ holds serve.sh (server lifecycle under systemd units), probe.py (fixed-seed request matrix, STEPS=20), psnr.py (baseline comparison), fixtures.json, and per-run records under runs/. Adapt rather than rewrite; PR #4's branch carries the same kit under probe/issue-2/.
  • Weights: sealed snapshot Qwen/Qwen-Image-2.1@790c9263 at ~/.local/state/multi-model-helper-server/state/image-service/weights — serve from the local snapshot path with HF_HUB_OFFLINE=1 (the sealed cache lacks assets/qr.png, so hub require_all cannot complete offline).
  • Frozen references: ~/.local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2/. Sealed four-reference Diffusers baseline for PSNR: attempt 2026-10-07T170729Z-q3-4r under ~/.local/share/multi-model-helper-server/data/evaluation/media/image/attempts/ (samples.json maps fixture+seed to files and digests).
  • Artifacts stay outside Git under the state root, digest-pinned with a manifest, per the helper repo's recipe discipline (see multi-model-helper-server evaluation/media/recipes/ and the quantization-recipe pattern in evaluation/media/harness/README.md).

Plan

  1. Build the artifact once: quantize the DiT to int8 weight-only (W8A16) and save as a compressed-tensors checkpoint — candidate tooling: llm-compressor (Int8WeightOnly), or compressed-tensors export from a torchao int8 state. Record inputs, versions, digests, and the manifest. The encoder and VAE stay BF16 in the artifact or on disk.
  2. Serve it: vllm serve <artifact-or-snapshot> ... --enable-cpu-offload with the compressed-tensors path (auto-detected or --quantization compressed-tensors), eager, single request.
  3. Structure gate every run (the lesson from #1's FP8 misread): opacity fraction, non-flatness, and PSNR against the sealed baseline — a run only counts if the output is structurally sound.
  4. Capacity matrix: one-, two-, three-, four-reference requests across seeds 2301-2304; record wall time, peak VRAM, server metrics.
  5. Quality read: BF16-row parity is the bar (26-29 dB band vs the Diffusers baselines, visual spot-check); note that a quantized trajectory may legitimately diverge at fixed seed (issue #1's 7 dB teapot lesson) — the structure gate plus visual check decides, and suite-level gates belong to the helper repo if this graduates.
  6. Verdict: does int8-compressed-tensors deliver 3-4 references at cached speed with structurally sound, visually clean output? Record per-reference-count numbers beside PR #4's BF16 and FP8 rows.

Acceptance Criteria

  • The artifact exists with a manifest (inputs, versions, digests) under the external state root; nothing enters Git.
  • The serve path loads it on sm_86 and passes the structure gate on every counted run.
  • The capacity matrix records wall time and peak VRAM per reference count, reproducible from the record.
  • The verdict states whether this is the 3-4 reference serving path on one 3090, with the quality evidence named.
  • Helper-system follow-through (serving-profile candidacy) is recorded as input to multi-model-helper-server #27, not implemented here.

Out of scope

  • Root-causing the online-FP8 Marlin bug (separate lane; PR #4 carries the isolation evidence).
  • Upstream code changes or PRs.
  • W8A8 int8 compute paths, int4, or NVFP4 (follow-ons if int8-wo fails).
  • Helper-system deployment evidence.
## Context PR #4 established two things on the RTX 3090 (24 GB): the BF16 serve path tops out at two references cached (~36.7 s), and the online-FP8 path — while it frees exactly the needed ~6 GiB and served all four reference counts (4-ref 44.1-44.4 s, peak 23,522 MB) — produces structurally broken outputs (~97% transparent RGBA, near-flat grey, 7-9 dB), isolated to the Marlin-quantized DiT itself (transformer-only FP8 breaks it; encoder BF16 does not; `--ignored-layers` inert). Root-causing the Marlin path is a separate lane. This issue takes the sidestep: **quantize the DiT offline to int8 weight-only, serve the artifact, and get the 3-4 reference capacity without touching the suspect online-FP8 code path.** Why int8 weight-only has the strong prior: the helper-system already quality-validated a torchao int8 DiT in Diffusers — Q3's blinded review placed it within label noise of the BF16 baseline (multi-model-helper-server #160) — and compressed-tensors is vLLM's most battle-tested quantized-checkpoint route. Same memory win as FP8 (~13.25 -> ~7 GiB DiT), different quantizer, different kernels. ## Host and environment facts - **Host**: `cachyos-alexander` at `192.168.0.39`, user `anodyine`, key-based SSH from the workstation (`ssh anodyine@192.168.0.39`); user lingering on for `systemd-run --user` units. RTX 3090 24 GB (sm_86), driver 615.71.09, CUDA 13.4. - **GPU caution**: the host's `relayview-inference.service` (a vLLM worker that grabs the whole GPU at boot) is owner-controlled and its enabled state varies between sessions — check `nvidia-smi` first and ask the owner if it holds the card; do not stop it yourself. - **Environment** (standing from #1/#2): Python 3.13 venv at `~/Projects/vllm-omni-val/venv` over the clone at `~/Projects/vllm-omni-val/vllm-omni` (origin = this mirror, upstream = GitHub; main `8484d85d1`). Installed: vllm 0.31.0, torch 2.13.0+cu130, transformers 5.14.1, diffusers git `122b1e11`, repo editable, `requirements/common.txt` runtime deps. - **Reusable probe kit** from #2: `~/Projects/vllm-omni-val/issue2/` holds `serve.sh` (server lifecycle under systemd units), `probe.py` (fixed-seed request matrix, STEPS=20), `psnr.py` (baseline comparison), `fixtures.json`, and per-run records under `runs/`. Adapt rather than rewrite; PR #4's branch carries the same kit under `probe/issue-2/`. - **Weights**: sealed snapshot `Qwen/Qwen-Image-2.1@790c9263` at `~/.local/state/multi-model-helper-server/state/image-service/weights` — serve from the local snapshot path with `HF_HUB_OFFLINE=1` (the sealed cache lacks `assets/qr.png`, so hub `require_all` cannot complete offline). - **Frozen references**: `~/.local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2/`. **Sealed four-reference Diffusers baseline** for PSNR: attempt `2026-10-07T170729Z-q3-4r` under `~/.local/share/multi-model-helper-server/data/evaluation/media/image/attempts/` (samples.json maps fixture+seed to files and digests). - **Artifacts stay outside Git** under the state root, digest-pinned with a manifest, per the helper repo's recipe discipline (see multi-model-helper-server `evaluation/media/recipes/` and the quantization-recipe pattern in `evaluation/media/harness/README.md`). ## Plan 1. **Build the artifact once**: quantize the DiT to int8 weight-only (W8A16) and save as a compressed-tensors checkpoint — candidate tooling: llm-compressor (Int8WeightOnly), or compressed-tensors export from a torchao int8 state. Record inputs, versions, digests, and the manifest. The encoder and VAE stay BF16 in the artifact or on disk. 2. **Serve it**: `vllm serve <artifact-or-snapshot> ... --enable-cpu-offload` with the compressed-tensors path (auto-detected or `--quantization compressed-tensors`), eager, single request. 3. **Structure gate every run** (the lesson from #1's FP8 misread): opacity fraction, non-flatness, and PSNR against the sealed baseline — a run only counts if the output is structurally sound. 4. **Capacity matrix**: one-, two-, three-, four-reference requests across seeds 2301-2304; record wall time, peak VRAM, server metrics. 5. **Quality read**: BF16-row parity is the bar (26-29 dB band vs the Diffusers baselines, visual spot-check); note that a quantized trajectory may legitimately diverge at fixed seed (issue #1's 7 dB teapot lesson) — the structure gate plus visual check decides, and suite-level gates belong to the helper repo if this graduates. 6. **Verdict**: does int8-compressed-tensors deliver 3-4 references at cached speed with structurally sound, visually clean output? Record per-reference-count numbers beside PR #4's BF16 and FP8 rows. ## Acceptance Criteria - The artifact exists with a manifest (inputs, versions, digests) under the external state root; nothing enters Git. - The serve path loads it on sm_86 and passes the structure gate on every counted run. - The capacity matrix records wall time and peak VRAM per reference count, reproducible from the record. - The verdict states whether this is the 3-4 reference serving path on one 3090, with the quality evidence named. - Helper-system follow-through (serving-profile candidacy) is recorded as input to multi-model-helper-server #27, not implemented here. ## Out of scope - Root-causing the online-FP8 Marlin bug (separate lane; PR #4 carries the isolation evidence). - Upstream code changes or PRs. - W8A8 int8 compute paths, int4, or NVFP4 (follow-ons if int8-wo fails). - Helper-system deployment evidence.
anodyine added the type/testarea/backendstatus/ready-for-agent labels 2026-10-08 06:34:36 -04:00
anodyine added status/in-progress and removed status/ready-for-agent labels 2026-10-08 06:40:51 -04:00
anodyine added reference ISSUE-5 2026-10-08 06:40:53 -04:00
anodyine commented 2026-10-08 07:51:11 -04:00
Author
Owner
Copy link
Copy source

Directive after the int8-wo result (2026-10-08, owner-approved): worth one more bounded round — pivot from quantizer variants to kernel-path isolation.

The convergence is the finding: online-FP8 (PR #4) and offline int8-compressed-tensors (this issue, artifact exonerated at ~1% L2 by independent dequantization) both collapse in the serve path, and both route through Marlin W8A16 on sm_86. Quantizers ruled out; kernel family implicated. The stripe texture (parallel output tiles, healthy RGB) suggests a tiling/parallel-reduction bug at DiT-scale M — shapes LLM decoders never produce.

Round two, two experiments:

  1. W8A8 int8-dynamic (the follow-on this issue's out-of-scope held for exactly this failure): compressed-tensors W8A8-Int8-Dynamic artifact, int8 compute on Ampere's native tensor cores — no Marlin in the path. Same ~7 GiB capacity target. Structure gate every run.
  2. Single-GEMM Marlin isolation: one DiT linear layer, Marlin vs reference dequant-then-matmul, M swept from LLM-scale to DiT-scale. A break threshold in M pins the bug and writes the upstream report (possibly with a chunk-the-M fix).

Hard stop: if W8A8 also produces broken structure, record the every-low-bit-path-implicated pattern and descope — capacity falls back to the helper repo's paths (multi-model-helper-server #160 cache-off tier; #164 placement). The Marlin root-cause joins #3's upstream lane, off the product critical path. Either way, update PR #4's record with the convergence: two independent quantizers, one kernel family.

Directive after the int8-wo result (2026-10-08, owner-approved): worth one more bounded round — pivot from quantizer variants to kernel-path isolation. The convergence is the finding: online-FP8 (PR #4) and offline int8-compressed-tensors (this issue, artifact exonerated at ~1% L2 by independent dequantization) both collapse in the serve path, and both route through Marlin W8A16 on sm_86. Quantizers ruled out; kernel family implicated. The stripe texture (parallel output tiles, healthy RGB) suggests a tiling/parallel-reduction bug at DiT-scale M — shapes LLM decoders never produce. Round two, two experiments: 1. **W8A8 int8-dynamic** (the follow-on this issue's out-of-scope held for exactly this failure): compressed-tensors W8A8-Int8-Dynamic artifact, int8 compute on Ampere's native tensor cores — no Marlin in the path. Same ~7 GiB capacity target. Structure gate every run. 2. **Single-GEMM Marlin isolation**: one DiT linear layer, Marlin vs reference dequant-then-matmul, M swept from LLM-scale to DiT-scale. A break threshold in M pins the bug and writes the upstream report (possibly with a chunk-the-M fix). Hard stop: if W8A8 also produces broken structure, record the every-low-bit-path-implicated pattern and descope — capacity falls back to the helper repo's paths (multi-model-helper-server #160 cache-off tier; #164 placement). The Marlin root-cause joins #3's upstream lane, off the product critical path. Either way, update PR #4's record with the convergence: two independent quantizers, one kernel family.
anodyine commented 2026-10-08 08:34:26 -04:00
Author
Owner
Copy link
Copy source

Implemented in PR: #6

Verdict in brief: the int8 weight-only compressed-tensors artifact delivers the 3-4 reference capacity and speed (4-ref 46.4 s, peak 23,935 MiB, all 16 matrix requests complete) but output quality collapses in the serve kernel — structure gate FAIL 16/16, PSNR 7.1-7.5 dB. The checkpoint is verified numerically sound (round-trip ~1.1% rel-L2 vs sealed BF16; bitwise-correct loading incl. fused q/k/v), and marlin_isolate.py pins the defect to the 8-bit Marlin W8A16 path on this stack: NaN output for real and synthetic int8 weights, all layer geometries, both bf16/fp16 activations, on sm_86 where the capability query accepts uint8b128. Two serve-path fixes were required and are in the PR (quant factory compressed-tensors resolution; offline-quantized CPU-offload loading). Helper-system follow-through for multi-model-helper-server #27 is recorded in probe/issue-5/FINDINGS.md as input only.

Implemented in PR: https://tea.kyle-merritt.com/anodyine/vllm-omni/pulls/6 Verdict in brief: the int8 weight-only compressed-tensors artifact delivers the 3-4 reference capacity and speed (4-ref 46.4 s, peak 23,935 MiB, all 16 matrix requests complete) but output quality collapses in the serve kernel — structure gate FAIL 16/16, PSNR 7.1-7.5 dB. The checkpoint is verified numerically sound (round-trip ~1.1% rel-L2 vs sealed BF16; bitwise-correct loading incl. fused q/k/v), and marlin_isolate.py pins the defect to the 8-bit Marlin W8A16 path on this stack: NaN output for real and synthetic int8 weights, all layer geometries, both bf16/fp16 activations, on sm_86 where the capability query accepts uint8b128. Two serve-path fixes were required and are in the PR (quant factory compressed-tensors resolution; offline-quantized CPU-offload loading). Helper-system follow-through for multi-model-helper-server #27 is recorded in probe/issue-5/FINDINGS.md as input only.
anodyine added status/review and removed status/in-progress labels 2026-10-08 08:34:34 -04:00
anodyine commented 2026-10-08 08:37:23 -04:00
Author
Owner
Copy link
Copy source

Amendment to the 08-07 directive, given PR #6's verdict: the isolation experiment already ran and eliminated the directive's stated hypotheses — NaN occurs with synthetic int8 weights (not quantizer/scales), across all layer geometries (not an M-shape edge case), under both bf16 and fp16 activations (not a dtype mismatch). The surviving root-cause shape: the capability query accepts uint8b128 on sm_86 while the kernel itself yields NaN — a dispatch overclaim for the 8-bit Marlin W8A16 path on this architecture.

Direction therefore narrows to:

  1. W8A8 int8-dynamic is now the decisive remaining experiment for the capacity route — with W8A16-Marlin conclusively broken on sm_86, the non-Marlin int8 path is the only low-bit route left through vLLM on this card. Structure gate every run.
  2. The upstream bug report no longer waits on anything: capability-query overclaim + NaN kernel + synthetic-weight repro + the PR #4 online-FP8 collapse (same family) can be drafted for the #3 lane now.
  3. Hard stop unchanged: W8A8 structure-gate failure -> record the pattern and descope; capacity falls back to multi-model-helper-server #160/#164.
Amendment to the 08-07 directive, given PR #6's verdict: the isolation experiment already ran and eliminated the directive's stated hypotheses — NaN occurs with synthetic int8 weights (not quantizer/scales), across all layer geometries (not an M-shape edge case), under both bf16 and fp16 activations (not a dtype mismatch). The surviving root-cause shape: the capability query accepts uint8b128 on sm_86 while the kernel itself yields NaN — a dispatch overclaim for the 8-bit Marlin W8A16 path on this architecture. Direction therefore narrows to: 1. **W8A8 int8-dynamic is now the decisive remaining experiment** for the capacity route — with W8A16-Marlin conclusively broken on sm_86, the non-Marlin int8 path is the only low-bit route left through vLLM on this card. Structure gate every run. 2. **The upstream bug report no longer waits on anything**: capability-query overclaim + NaN kernel + synthetic-weight repro + the PR #4 online-FP8 collapse (same family) can be drafted for the #3 lane now. 3. **Hard stop unchanged**: W8A8 structure-gate failure -> record the pattern and descope; capacity falls back to multi-model-helper-server #160/#164.
anodyine commented 2026-10-08 09:07:02 -04:00
Author
Owner
Copy link
Copy source

Round two (amendment response) — the decisive experiment is a YES: W8A8 int8-dynamic delivers the capacity route on one 3090.

W8A8-dynamic artifact (cutlass int8 scaled-MM, no Marlin): all 16 capacity-matrix requests structurally sound — opaque 0.947–0.988, alpha mean ≈255, healthy RGB spread; visual checks render faithful scenes (4-ref apothecary scene, 1-ref portrait). PSNR vs the sealed Diffusers baseline: 33.72 / 28.34 / 27.50 dB at 1/2/4 refs — the BF16-parity band (26–29 dB), including the first usable 4-reference quality comparison this card has produced. Walls 20.5/25.1/29.4/34.0 s per reference count — the fastest row measured (4-ref ≈ 4.3× the sealed Diffusers nocache baseline), peak 23,879 MiB.

Gate calibration honesty: round one's 0.99-opacity pass rule was my instrument calibration from the broken-vs-BF16 contrast; W8A8 outputs sit at 0.95–0.99 with alpha ~255 (soft-edge alpha, nowhere near the 0.008–0.20 broken band). The verdict rests on the raw numbers, visual checks, and PSNR; per-request metrics are recorded unedited.

Kernel verification this round: raw cutlass ops match a float32 reference at 0.17% rel-L2 across DiT geometries, and the full layer path (load → repack → apply, fused QKV included, bitwise-verified loading) matches at 0.17%. One recorded instrument lesson: an intermediate 97%-error scare was a missing ×act_scale factor in my probe's own reference, not a kernel defect.

Hard stop did not trigger. The upstream Marlin report is drafted and staged (probe/issue-5/UPSTREAM-REPORT-DRAFT.md — capability-query overclaim + NaN kernel + synthetic repro + the PR #4 FP8 convergence), ready for the #3 lane on your call. PR #4's record has the convergence comment; PR #6 carries the full round-two evidence (probe/issue-5/evidence/rtx3090-round2/ + FINDINGS.md).

Round two (amendment response) — the decisive experiment is a **YES**: W8A8 int8-dynamic delivers the capacity route on one 3090. **W8A8-dynamic artifact (cutlass int8 scaled-MM, no Marlin):** all 16 capacity-matrix requests structurally sound — opaque 0.947–0.988, alpha mean ≈255, healthy RGB spread; visual checks render faithful scenes (4-ref apothecary scene, 1-ref portrait). PSNR vs the sealed Diffusers baseline: **33.72 / 28.34 / 27.50 dB at 1/2/4 refs** — the BF16-parity band (26–29 dB), including the first usable 4-reference quality comparison this card has produced. Walls 20.5/25.1/29.4/34.0 s per reference count — the fastest row measured (4-ref ≈ 4.3× the sealed Diffusers nocache baseline), peak 23,879 MiB. Gate calibration honesty: round one's 0.99-opacity pass rule was my instrument calibration from the broken-vs-BF16 contrast; W8A8 outputs sit at 0.95–0.99 with alpha ~255 (soft-edge alpha, nowhere near the 0.008–0.20 broken band). The verdict rests on the raw numbers, visual checks, and PSNR; per-request metrics are recorded unedited. Kernel verification this round: raw cutlass ops match a float32 reference at 0.17% rel-L2 across DiT geometries, and the full layer path (load → repack → apply, fused QKV included, bitwise-verified loading) matches at 0.17%. One recorded instrument lesson: an intermediate 97%-error scare was a missing ×act_scale factor in my probe's own reference, not a kernel defect. Hard stop did not trigger. The upstream Marlin report is drafted and staged (probe/issue-5/UPSTREAM-REPORT-DRAFT.md — capability-query overclaim + NaN kernel + synthetic repro + the PR #4 FP8 convergence), ready for the #3 lane on your call. PR #4's record has the convergence comment; PR #6 carries the full round-two evidence (probe/issue-5/evidence/rtx3090-round2/ + FINDINGS.md).
Sign in to join this conversation.
ISSUE-5
Labels
Clear labels
area/agent-workflow
Agent workflow area
 area/backend
Backend area
 area/docs
Documentation area
 area/frontend
Frontend area
 area/tests
Testing area
 area/workflow
General workflow area
 priority/P0
Exceptional interrupt; not a complete priority taxonomy
 status/done
Work is complete
 status/in-progress
Implementation is in progress
 status/needs-info
Blocked pending more information
 status/needs-triage
Needs initial review and prioritization
 status/ready-for-agent
Ready for an agent to claim
 status/ready-for-human
Ready for human review or action
 status/review
Implementation is ready for review
 status/wontfix
Work will not be pursued
 type/architecture
Architecture or system-design work
 type/bug
Defect or regression
 type/docs
Documentation work
 type/epic
Epic grouping related work
 type/feature
New behavior or capability
 type/refactor
Internal restructuring without behavior change
 type/roadmap
Canonical repository work queue
 type/test
Test coverage or test infrastructure
No labels area/backend status/review type/test
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
anodyine
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anodyine/vllm-omni#5