anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

probe: Qwen-Image conditioning-KV spill probe on RTX 3090 / RTX 5080 (issue 2) #4

Open
anodyine wants to merge 2 commits from ISSUE-2 into main
pull from: ISSUE-2
merge into: anodyine:main
Conversation 4 Commits 2 Files Changed 11
+2498
anodyine commented 2026-10-08 00:58:57 -04:00
Owner
Copy link
Copy source

Summary

Evidence-only change closing issue 2's probe: whether Qwen-Image-2.1's conditioning KV can spill to host DRAM on the RTX 3090 (24 GB) and RTX 5080 (16 GB). No code changes to the stack — the PR carries the probe kit, per-host run records, and the findings report under probe/issue-2/.

Verdict: not-built on both cards.

3090 (24 GB, sm_86)   served ceiling = 2 references, cache ON, 2.2× Diffusers nocache speed
                      4 references OOM in the DiT img-MLP forward under every flag;
                      binding constraint = resident BF16 working set + fixed ~490 MiB
                      activation, NOT the conditioning KV (≈0.3 GiB/ref, per-request, freed)
5080 (16 GB, sm_120)  server cannot initialize under any flag (6 attempts: default,
                      cpu-offload, +layerwise, fp8, fp8+expandable, int8) —
                      init_parameters (diffusion/models/utils.py:159-168) materializes the
                      unquantized parameter skeleton on-device before quantization/offload act

Cache located and documented with file:line, allocator (plain GPU tensor clones, DENSE_LEGACY, not paged), and the exact flags reaching it (prefix_kv_cache_dtype extras, eager/cuda-graph-decode; --swap-space/--cpu-offload-gb do not exist in this fork). Bonus finding: the recipe's deploy-YAML extras route for prefix_kv_cache_dtype does not reach the served transformer on 8484d85d (byte-identical outputs with/without; offline Omni(extras=…) kwarg works).

Evidence

  • Before (the open question): whether vLLM-Omni's merged pipeline pages/swaps/leans enough that multi-reference compositions run at cached speed where Diffusers cannot fit them (#2 body).
  • After: full run matrices per card in probe/issue-2/FINDINGS.md; key numbers:
    • 3090 --enable-cpu-offload: 1-ref OK 34.2 s, 2-ref OK 36.7 s (peak 24,015 MiB), 4-ref HTTP 500 OOM.
    • Layerwise offload strictly worse (1-ref OOMs). expandable_segments:True removes fragmentation (1.84 GiB → 0.5 GiB reserved-unallocated); 4-ref still OOMs with 22.29 GiB genuinely allocated.
    • Offline fp8_v (the one KV-reaching knob, via Omni engine kwargs): 4-ref still OOMs, with and without expandable segments — cache quantization is an order of magnitude too small to matter here.
    • Measured edit-compose-pair seeds 2301–2304: 36.6–36.9 s/image vs Diffusers 81.16 s; PSNR mean 27.57 dB (26.09–28.92, RGBA-on-RGBA) against the sealed attempt 2026-10-07T170729Z-q3-4r.
    • Determinism: seed-2301 outputs byte-identical across three server flag sets.

Checks run

Observed: both hosts' full probe sequences (server lifecycle under systemd-run --user units, 1 Hz nvidia-smi series, one request at a time, fixed seeds); all per-run records in probe/issue-2/evidence/*/runs-summary.json and on-host runs/<label>/ directories. The four-reference PSNR cell is capacity-blocked on every served flag set — recorded honestly; the two-reference row is the highest-count comparison the 3090 can produce.

Documentation disposition

updated — the probe report and evidence records are the deliverable (probe/issue-2/FINDINGS.md).

Merge Danger

Door: one-way (evidence and probe scripts only; trivially revertible as a git revert, hence cheap to undo, but the record should persist).

Blast Radius: none at runtime — no stack code touched. probe/issue-2/ is a new directory nothing references.

Rollback: git revert <merge-commit>.

Reviewer prerequisites

  • For re-running the probe: SSH access to cachyos-alexander (192.168.0.39) and cachyos-9900x-o11d (192.168.0.40), both hosts' ~/Projects/vllm-omni-val/ environments (already standing), sealed weights + frozen suite present on both. The 5080 leg additionally used a userspace CPython 3.13.16 (~/.local/pythons/cpython-3.13) since the host ships only 3.14.
  • Reviewing the record alone needs none of the above.

Pre-Test Setup

git fetch origin
git switch ISSUE-2
git pull --ff-only origin ISSUE-2

Expected: probe/issue-2/ with FINDINGS.md, five scripts/YAML, fixtures.json, and two evidence directories.

Manual Test Plan

  1. Action: Read probe/issue-2/FINDINGS.md end to end.

    Expected: verdicts per card match the issue's three-world taxonomy (not-built both), cache located by file:line, allocator and reachable flags named, both run matrices carry peak VRAM / wall time / PSNR or the honest reason a cell is blocked, inputs reproducible from fixtures.json + flags recorded per run.

  2. Action (optional, re-verify one cell on the 3090):

    ssh anodyine@192.168.0.39 'systemd-run --user --unit=issue2-serve --collect bash ~/Projects/vllm-omni-val/issue2/serve.sh cpuoff --enable-cpu-offload'
# wait for http://127.0.0.1:8091/health == 200, then:
ssh anodyine@192.168.0.39 'cd ~/Projects/vllm-omni-val/issue2 && venv_python=~/Projects/vllm-omni-val/venv/bin/python && $venv_python probe.py --out /tmp/verify --functional'

    Expected: 1-ref OK, 2-ref OK, 4-ref HTTP 500 CUDA OOM; wall times within a few seconds of FINDINGS.md's. Stop the unit afterwards (systemctl --user stop issue2-serve).

Failure and recovery

If a re-verification diverges from the record, retain the new runs/ output and the serve/vram logs; do not mark the review passed on a diverging number — the record would need a re-run matrix, not a note.

Post-Merge Cleanup

git switch main
git pull --ff-only origin main
git branch -d ISSUE-2
git push origin --delete ISSUE-2
## Summary Evidence-only change closing issue 2's probe: whether Qwen-Image-2.1's conditioning KV can spill to host DRAM on the RTX 3090 (24 GB) and RTX 5080 (16 GB). No code changes to the stack — the PR carries the probe kit, per-host run records, and the findings report under `probe/issue-2/`. **Verdict: not-built on both cards.** ``` 3090 (24 GB, sm_86) served ceiling = 2 references, cache ON, 2.2× Diffusers nocache speed 4 references OOM in the DiT img-MLP forward under every flag; binding constraint = resident BF16 working set + fixed ~490 MiB activation, NOT the conditioning KV (≈0.3 GiB/ref, per-request, freed) 5080 (16 GB, sm_120) server cannot initialize under any flag (6 attempts: default, cpu-offload, +layerwise, fp8, fp8+expandable, int8) — init_parameters (diffusion/models/utils.py:159-168) materializes the unquantized parameter skeleton on-device before quantization/offload act ``` Cache located and documented with file:line, allocator (plain GPU tensor clones, `DENSE_LEGACY`, not paged), and the exact flags reaching it (`prefix_kv_cache_dtype` extras, eager/cuda-graph-decode; `--swap-space`/`--cpu-offload-gb` do not exist in this fork). Bonus finding: the recipe's deploy-YAML extras route for `prefix_kv_cache_dtype` does not reach the served transformer on `8484d85d` (byte-identical outputs with/without; offline `Omni(extras=…)` kwarg works). ## Evidence - **Before (the open question):** whether vLLM-Omni's merged pipeline pages/swaps/leans enough that multi-reference compositions run at cached speed where Diffusers cannot fit them (#2 body). - **After:** full run matrices per card in `probe/issue-2/FINDINGS.md`; key numbers: - 3090 `--enable-cpu-offload`: 1-ref OK 34.2 s, 2-ref OK 36.7 s (peak 24,015 MiB), 4-ref HTTP 500 OOM. - Layerwise offload strictly worse (1-ref OOMs). `expandable_segments:True` removes fragmentation (1.84 GiB → 0.5 GiB reserved-unallocated); 4-ref still OOMs with 22.29 GiB genuinely allocated. - Offline fp8_v (the one KV-reaching knob, via Omni engine kwargs): 4-ref still OOMs, with and without expandable segments — cache quantization is an order of magnitude too small to matter here. - Measured `edit-compose-pair` seeds 2301–2304: 36.6–36.9 s/image vs Diffusers 81.16 s; PSNR mean 27.57 dB (26.09–28.92, RGBA-on-RGBA) against the sealed attempt `2026-10-07T170729Z-q3-4r`. - Determinism: seed-2301 outputs byte-identical across three server flag sets. ### Checks run Observed: both hosts' full probe sequences (server lifecycle under `systemd-run --user` units, 1 Hz nvidia-smi series, one request at a time, fixed seeds); all per-run records in `probe/issue-2/evidence/*/runs-summary.json` and on-host `runs/<label>/` directories. The four-reference PSNR cell is capacity-blocked on every served flag set — recorded honestly; the two-reference row is the highest-count comparison the 3090 can produce. ## Documentation disposition updated — the probe report and evidence records are the deliverable (`probe/issue-2/FINDINGS.md`). ## Merge Danger **Door:** one-way (evidence and probe scripts only; trivially revertible as a git revert, hence cheap to undo, but the record should persist). **Blast Radius:** none at runtime — no stack code touched. `probe/issue-2/` is a new directory nothing references. **Rollback:** `git revert <merge-commit>`. ## Reviewer prerequisites - For re-running the probe: SSH access to `cachyos-alexander` (192.168.0.39) and `cachyos-9900x-o11d` (192.168.0.40), both hosts' `~/Projects/vllm-omni-val/` environments (already standing), sealed weights + frozen suite present on both. The 5080 leg additionally used a userspace CPython 3.13.16 (`~/.local/pythons/cpython-3.13`) since the host ships only 3.14. - Reviewing the record alone needs none of the above. ## Pre-Test Setup ```sh git fetch origin git switch ISSUE-2 git pull --ff-only origin ISSUE-2 ``` Expected: `probe/issue-2/` with FINDINGS.md, five scripts/YAML, fixtures.json, and two evidence directories. ## Manual Test Plan 1. [ ] **Action:** Read `probe/issue-2/FINDINGS.md` end to end. **Expected:** verdicts per card match the issue's three-world taxonomy (`not-built` both), cache located by file:line, allocator and reachable flags named, both run matrices carry peak VRAM / wall time / PSNR or the honest reason a cell is blocked, inputs reproducible from `fixtures.json` + flags recorded per run. 2. [ ] **Action (optional, re-verify one cell on the 3090):** ```sh ssh anodyine@192.168.0.39 'systemd-run --user --unit=issue2-serve --collect bash ~/Projects/vllm-omni-val/issue2/serve.sh cpuoff --enable-cpu-offload' # wait for http://127.0.0.1:8091/health == 200, then: ssh anodyine@192.168.0.39 'cd ~/Projects/vllm-omni-val/issue2 && venv_python=~/Projects/vllm-omni-val/venv/bin/python && $venv_python probe.py --out /tmp/verify --functional' ``` **Expected:** 1-ref OK, 2-ref OK, 4-ref HTTP 500 CUDA OOM; wall times within a few seconds of FINDINGS.md's. Stop the unit afterwards (`systemctl --user stop issue2-serve`). ## Failure and recovery If a re-verification diverges from the record, retain the new `runs/` output and the serve/vram logs; do not mark the review passed on a diverging number — the record would need a re-run matrix, not a note. ## Post-Merge Cleanup ```sh git switch main git pull --ff-only origin main git branch -d ISSUE-2 git push origin --delete ISSUE-2 ```
anodyine added 1 commit 2026-10-08 00:58:57 -04:00
probe: Qwen-Image conditioning-KV spill probe on RTX 3090 / RTX 5080 (issue 2)
Build Wheel / build (3.11) (pull_request) Failing after 33s
Details
Build Wheel / build (3.12) (pull_request) Failing after 40s
Details
pre-commit / pre-commit (pull_request) Failing after 41s
Details
67eb7fdb48 
Evidence-only change: probe kit (serve wrapper, API client, PSNR
comparator, offline fp8_v driver), per-host run summaries, and the
findings report. No code changes to the stack.

Verdicts: not-built on both cards. On the 3090 the served reference
ceiling is two condition images with the conditioning cache on
(2.2x the Diffusers nocache baseline, pair PSNR mean 27.57 dB);
four references OOM on the resident BF16 working set, not the KV
(~0.3 GiB/reference, per-request). On the 5080 the server cannot
initialize under any flag: init_parameters materializes the
unquantized skeleton on-device before quantization/offload engage.
Also documents that the recipe's deploy-YAML extras route for
prefix_kv_cache_dtype does not reach the served transformer on
8484d85d (offline Omni engine kwarg does).
anodyine commented 2026-10-08 06:01:26 -04:00
Author
Owner
Copy link
Copy source

Review finding — one configuration is missing from the 3090 serve matrix: --quantization fp8 (weight-only, Marlin on sm_86).

Issue #1's own addendum validated this exact path on this exact card offline: BF16 --enable-cpu-offload peaked 23.0 GB, while --quantization fp8 + offload peaked 18.4 GB — the DiT drops from ~13.25 to ~7.1 GB resident. The 3090 matrix here covers cpu-offload, layerwise, expandable-segments, and the offline fp8_v KV-dtype knob, but never the model-weight fp8 on the serve path (the 5080 matrix includes fp8, but only as one of the six init failures).

The arithmetic says this is precisely the missing headroom: this PR's own finding is that the 4-reference OOM binds on the resident BF16 working set (22.29 GiB genuinely allocated) and not the conditioning KV (~0.3 GiB/ref, per-request, freed). Freeing ~6 GiB of resident BF16 DiT puts that working set near 16.3 GiB — four references should fit on 24 GB, three comfortably, and the 2-ref cached row likely gains a few percent as well (offline fp8 measured 58.56 vs 61.26 s at 50 steps).

Requested before merge:

  1. Add the vllm serve ... --enable-cpu-offload --quantization fp8 row on the 3090; retry one-, two-, three-, and four-reference requests (three refs is currently untested anywhere).
  2. If the server fails at init_parameters under fp8 on the 3090 the way it does on the 5080 — skeleton materializing on-device before quantization — record that as its own finding with the same file:line; check whether a --load-format variant or meta-device init avoids it, and if not, that is a second upstream-worthy bug (sub-skeleton VRAM cannot boot quantized serving at all), separate from the DRAM verdict.

The not-built verdict for DRAM offload itself is well-evidenced (unpaged tensor clones, DENSE_LEGACY, no swap flags in this fork, file:line located) and unaffected — this comment challenges only the reference-capacity ceiling on the 3090, which is currently concluded from a matrix that omits the one quantization its own prior issue proved works on this card.

Review finding — one configuration is missing from the 3090 serve matrix: `--quantization fp8` (weight-only, Marlin on sm_86). Issue #1's own addendum validated this exact path on this exact card offline: BF16 `--enable-cpu-offload` peaked 23.0 GB, while `--quantization fp8` + offload peaked **18.4 GB** — the DiT drops from ~13.25 to ~7.1 GB resident. The 3090 matrix here covers cpu-offload, layerwise, expandable-segments, and the offline `fp8_v` **KV-dtype** knob, but never the **model-weight** fp8 on the serve path (the 5080 matrix includes fp8, but only as one of the six init failures). The arithmetic says this is precisely the missing headroom: this PR's own finding is that the 4-reference OOM binds on the *resident BF16 working set* (22.29 GiB genuinely allocated) and **not** the conditioning KV (~0.3 GiB/ref, per-request, freed). Freeing ~6 GiB of resident BF16 DiT puts that working set near 16.3 GiB — four references should fit on 24 GB, three comfortably, and the 2-ref cached row likely gains a few percent as well (offline fp8 measured 58.56 vs 61.26 s at 50 steps). Requested before merge: 1. Add the `vllm serve ... --enable-cpu-offload --quantization fp8` row on the 3090; retry one-, two-, three-, and four-reference requests (three refs is currently untested anywhere). 2. If the server fails at `init_parameters` under fp8 on the 3090 the way it does on the 5080 — skeleton materializing on-device before quantization — record that as its own finding with the same file:line; check whether a `--load-format` variant or meta-device init avoids it, and if not, that is a second upstream-worthy bug (sub-skeleton VRAM cannot boot quantized serving at all), separate from the DRAM verdict. The **not-built** verdict for DRAM offload itself is well-evidenced (unpaged tensor clones, DENSE_LEGACY, no swap flags in this fork, file:line located) and unaffected — this comment challenges only the reference-capacity ceiling on the 3090, which is currently concluded from a matrix that omits the one quantization its own prior issue proved works on this card.
anodyine added 1 commit 2026-10-08 06:27:37 -04:00
probe(review): fp8 serve row + 3-ref fixture + fp8 quality-collapse diagnosis
Build Wheel / build (3.12) (pull_request) Failing after 50s
Details
Build Wheel / build (3.11) (pull_request) Failing after 50s
Details
pre-commit / pre-commit (pull_request) Failing after 54s
Details
4f5f99d8c6 
Review follow-up on PR #4: run --quantization fp8 through the served
path on the 3090 (the flag issue 1 validated for capacity), add the
untested 3-reference fixture, and measure all counts.

Capacity: confirmed — all of 1/2/3/4 references fit and run (4-ref
44.1-44.4s, server peak 23,522MB, nvidia-smi max 23,983MiB; Marlin
W8A16 is faster than BF16, 3.3x the Diffusers nocache baseline).

Quality: every fp8 output is structurally broken on this commit and
card (~97% transparent RGBA, grey RGB, PSNR 7-9dB) — on the edit path
at every reference count and on T2I alike (issue 1's fp8 artifacts
reinspected: same collapse; that run validated rc+image, not output
structure). img_mlp exclusion does not fix it; transformer-only fp8
also collapses, pinning the culprit on the Marlin-quantized DiT.
3090 verdict stays not-built; upstream fix order is fp8 output
correctness before any capacity role for it.
anodyine commented 2026-10-08 06:27:44 -04:00
Author
Owner
Copy link
Copy source

__zcode_status=$?
if [ "$__zcode_status" -eq 0 ]; then pwd -P > '/tmp/zcode-4c844978-0de0-4cd6-ac3a-7dee0f7f94fe-cwd'; fi
exit "$__zcode_status"

__zcode_status=$? if [ "$__zcode_status" -eq 0 ]; then pwd -P > '/tmp/zcode-4c844978-0de0-4cd6-ac3a-7dee0f7f94fe-cwd'; fi exit "$__zcode_status"
anodyine commented 2026-10-08 06:28:40 -04:00
Author
Owner
Copy link
Copy source

Review answer — fp8 serve row run (commit 4f5f99d8). You were right that the row was missing, and right about the arithmetic: vllm serve … --enforce-eager --enable-cpu-offload --quantization fp8 initializes on the 3090 and all four reference counts fit and run — 1-ref 29.9 s, 2-ref 34.1 s, the previously untested 3-ref 38.9 s, 4-ref 44.3 s (server peak 23,522 MB; nvidia-smi series max 23,983 MiB; measured seeds 2301–2304 for pair/three/four in runs-summary.json). Marlin W8A16 is even faster than BF16 — 4-ref at 3.3× the Diffusers nocache baseline (145.8 s).

But the fp8 row cannot be merged as a working configuration: every fp8 output is structurally broken on this commit and card. The signature is identical everywhere — RGBA ~96.5–97% transparent (opaque fraction 0.008–0.035, alpha mean ≈115), near-flat grey RGB, PSNR 7.03–9.02 dB vs the sealed baseline (BF16 on the same requests: 100% opaque, 26–29 dB). Isolation runs:

  • Serve path, edit fixtures at 1/2/3/4 references: all collapsed (~3% opaque).
  • fp8 minus img_mlp (your --ignored-layers variant — serve-CLI-unreachable, so via offline Omni(quantization_config=…)): still collapsed.
  • Transformer-only fp8 (text encoder stays BF16): still collapsed — this pins the culprit on the Marlin-quantized DiT itself, not the encoder, not the image-MLP.
  • Encoder-only fp8 couldn't initialize (its bf16→fp8 conversion transient lands on GPU atop the resident BF16 DiT).

And a correction to #1's record this uncovered: reinspecting issue #1's own fp8 artifacts (qwen_image_21_fp8_all_rtx3090.png, …_imgmlp_…) shows the same collapse (opaque fraction 0.035 both) — that validation checked rc=0 and image production, not output structure, so "fp8 works on this card" was overstated. The 18.4 GB capacity measurement stands; the implied quality does not. Worth noting upstream never measured DiT-fp8 PSNR either (the recipe quotes PSNR only for text-encoder fp8, 32.1 dB on H200) — this adds the Ampere/Marlin cell: online FP8 DiT weights produce near-noise latents for Qwen-Image-2.1.

Net effect on the verdicts: 3090 stays not-built — the capacity headroom you projected is real, but the only knob that unlocks it ships broken output on this commit, so the upstream order is fp8-output-correctness first, capacity second, and a KV spill remains unwarranted by the arithmetic. The 3-ref and 4-ref PSNR cells under fp8 are recorded in psnr-fp8-rows.json as broken-output data, not usable comparisons. FINDINGS.md carries the full new section.

(Apologies for the stray garbled comment just above — shell-wrapper text from a malformed local command; harmless, one-click delete.)

**Review answer — fp8 serve row run (commit 4f5f99d8).** You were right that the row was missing, and right about the arithmetic: `vllm serve … --enforce-eager --enable-cpu-offload --quantization fp8` initializes on the 3090 and **all four reference counts fit and run** — 1-ref 29.9 s, 2-ref 34.1 s, the previously untested 3-ref 38.9 s, 4-ref 44.3 s (server peak 23,522 MB; nvidia-smi series max 23,983 MiB; measured seeds 2301–2304 for pair/three/four in `runs-summary.json`). Marlin W8A16 is even faster than BF16 — 4-ref at 3.3× the Diffusers nocache baseline (145.8 s). **But the fp8 row cannot be merged as a working configuration: every fp8 output is structurally broken on this commit and card.** The signature is identical everywhere — RGBA ~96.5–97% transparent (opaque fraction 0.008–0.035, alpha mean ≈115), near-flat grey RGB, PSNR 7.03–9.02 dB vs the sealed baseline (BF16 on the same requests: 100% opaque, 26–29 dB). Isolation runs: - Serve path, edit fixtures at 1/2/3/4 references: all collapsed (~3% opaque). - `fp8` minus `img_mlp` (your `--ignored-layers` variant — serve-CLI-unreachable, so via offline `Omni(quantization_config=…)`): still collapsed. - **Transformer-only fp8** (text encoder stays BF16): still collapsed — this pins the culprit on the Marlin-quantized DiT itself, not the encoder, not the image-MLP. - Encoder-only fp8 couldn't initialize (its bf16→fp8 conversion transient lands on GPU atop the resident BF16 DiT). **And a correction to #1's record this uncovered:** reinspecting issue #1's own fp8 artifacts (`qwen_image_21_fp8_all_rtx3090.png`, `…_imgmlp_…`) shows the same collapse (opaque fraction 0.035 both) — that validation checked rc=0 and image production, not output structure, so "fp8 works on this card" was overstated. The 18.4 GB capacity measurement stands; the implied quality does not. Worth noting upstream never measured DiT-fp8 PSNR either (the recipe quotes PSNR only for text-encoder fp8, 32.1 dB on H200) — this adds the Ampere/Marlin cell: online FP8 DiT weights produce near-noise latents for Qwen-Image-2.1. Net effect on the verdicts: 3090 stays `not-built` — the capacity headroom you projected is real, but the only knob that unlocks it ships broken output on this commit, so the upstream order is fp8-output-correctness first, capacity second, and a KV spill remains unwarranted by the arithmetic. The 3-ref and 4-ref PSNR cells under fp8 are recorded in `psnr-fp8-rows.json` as broken-output data, not usable comparisons. FINDINGS.md carries the full new section. (Apologies for the stray garbled comment just above — shell-wrapper text from a malformed local command; harmless, one-click delete.)
anodyine commented 2026-10-08 09:07:41 -04:00
Author
Owner
Copy link
Copy source

Convergence record (from issue 5 / PR #6, per the 08:51 directive): the online-FP8 collapse isolated here and the offline int8-wo collapse from issue 5 are one kernel family's defect — Marlin W8A16 on sm_86 — now with the quantizers fully exonerated on the far side of the kernel.

Issue 5's int8-wo compressed-tensors artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 (compressed-tensors' own unpacker), loads bitwise-identically through the serve path (fused q/k/v shards included), scales finite pre/post Marlin repack — and layer(x) is NaN for real and synthetic int8 weights, all four DiT layer geometries, both bf16/fp16 activations, while the capability query accepts uint8b128 on the card. Two independent quantizers (online FP8 here, offline int8-wo there), one kernel family, one broken cell.

The surviving root-cause shape per issue 5's amendment: a dispatch overclaim for the 8-bit Marlin W8A16 path on this architecture (vllm 0.31.0 / torch 2.13.0+cu130 / CUDA 13.4).

The capacity consequence is resolved elsewhere: a W8A8 int8-dynamic artifact (cutlass int8 scaled-MM, no Marlin) serves 1-4 references at 20.5-34.0 s with BF16-parity PSNR (27.5 dB at 4 refs vs the sealed baseline) — so this FP8 row's brokenness is kernel-path-specific, not an int8/quantization limit.

Ready-to-file upstream report staged in PR #6 at probe/issue-5/UPSTREAM-REPORT-DRAFT.md, with the checkpoint-free reproduction (marlin_isolate.py --synthetic).

Convergence record (from issue 5 / PR #6, per the 08:51 directive): the online-FP8 collapse isolated here and the offline int8-wo collapse from issue 5 are one kernel family's defect — Marlin W8A16 on sm_86 — now with the quantizers fully exonerated on the far side of the kernel. Issue 5's int8-wo compressed-tensors artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 (compressed-tensors' own unpacker), loads bitwise-identically through the serve path (fused q/k/v shards included), scales finite pre/post Marlin repack — and layer(x) is NaN for real and synthetic int8 weights, all four DiT layer geometries, both bf16/fp16 activations, while the capability query accepts uint8b128 on the card. Two independent quantizers (online FP8 here, offline int8-wo there), one kernel family, one broken cell. The surviving root-cause shape per issue 5's amendment: a dispatch overclaim for the 8-bit Marlin W8A16 path on this architecture (vllm 0.31.0 / torch 2.13.0+cu130 / CUDA 13.4). The capacity consequence is resolved elsewhere: a W8A8 int8-dynamic artifact (cutlass int8 scaled-MM, no Marlin) serves 1-4 references at 20.5-34.0 s with BF16-parity PSNR (27.5 dB at 4 refs vs the sealed baseline) — so this FP8 row's brokenness is kernel-path-specific, not an int8/quantization limit. Ready-to-file upstream report staged in PR #6 at probe/issue-5/UPSTREAM-REPORT-DRAFT.md, with the checkpoint-free reproduction (marlin_isolate.py --synthetic).
Some required checks failed
Build Wheel / build (3.12) (pull_request) Failing after 50s
Details
Build Wheel / build (3.11) (pull_request) Failing after 50s
Details
pre-commit / pre-commit (pull_request) Failing after 54s
Details
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin ISSUE-2:ISSUE-2
git checkout ISSUE-2
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
area/agent-workflow
Agent workflow area
 area/backend
Backend area
 area/docs
Documentation area
 area/frontend
Frontend area
 area/tests
Testing area
 area/workflow
General workflow area
 priority/P0
Exceptional interrupt; not a complete priority taxonomy
 status/done
Work is complete
 status/in-progress
Implementation is in progress
 status/needs-info
Blocked pending more information
 status/needs-triage
Needs initial review and prioritization
 status/ready-for-agent
Ready for an agent to claim
 status/ready-for-human
Ready for human review or action
 status/review
Implementation is ready for review
 status/wontfix
Work will not be pursued
 type/architecture
Architecture or system-design work
 type/bug
Defect or regression
 type/docs
Documentation work
 type/epic
Epic grouping related work
 type/feature
New behavior or capability
 type/refactor
Internal restructuring without behavior change
 type/roadmap
Canonical repository work queue
 type/test
Test coverage or test infrastructure
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
anodyine
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anodyine/vllm-omni#4