anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Probe whether the Qwen-Image conditioning KV can spill to host DRAM on the RTX 3090 #2

New Issue
Open
opened 2026-10-07 21:02:38 -04:00 by anodyine · 2 comments
anodyine commented 2026-10-07 21:02:38 -04:00
Owner
Copy link
Copy source

Depends on #1 (validated runtime on sm_86).

Context

The helper-system's four-reference work measured that Diffusers retains the Qwen-Image conditioning KV (~2.8 GiB per reference cached) in VRAM with no spill path, so four references fit only with the cache off, at a 1.7-2.2x latency tax (multi-model-helper-server #160; docs/speculative/research/qwen-image-kv-cache-and-reference-capacity.md). vLLM-Omni's merged Qwen-Image pipeline implements its own conditioning KV cache (their review history names a "KV-cache aliasing bug in decode graphs" and "mixed-phase KV assembly") and natively accepts up to four condition images, but #1's validation exercised only text-to-image — the reference path is functionally untested on our stack, and whether that cache is paged, swappable to host DRAM, or simply leaner than Diffusers' retention is unverified anywhere. If any of those hold, multi-reference compositions could run at cached speed on cards where Diffusers cannot fit them.

Host and environment facts

RTX 3090 host (24 GB leg):

  • cachyos-alexander at 192.168.0.39, user anodyine, key-based SSH from the workstation (ssh anodyine@192.168.0.39); user lingering on for systemd-run --user units.
  • RTX 3090 24 GB (sm_86), driver 615.71.09, CUDA 13.4. The host's relayview-inference.service (vLLM worker that grabs the whole GPU) is owner-controlled and its enabled state varies between sessions — check nvidia-smi first and ask the owner if it holds the card; do not stop it yourself.
  • Working environment from #1: Python 3.13 venv at ~/Projects/vllm-omni-val/venv over the clone at ~/Projects/vllm-omni-val/vllm-omni (origin = this mirror, upstream = GitHub; main 8484d85d1). Installed: vllm 0.31.0 (0.30.0 fails at import on this commit), torch 2.13.0+cu130, transformers 5.14.1, diffusers git 122b1e11, repo editable, plus requirements/common.txt runtime/test deps (diffusers kept at the git pin).
  • Weights: digest-pinned snapshot Qwen/Qwen-Image-2.1@790c9263 at ~/.local/state/multi-model-helper-server/state/image-service/weights — serve from the local snapshot path with HF_HUB_OFFLINE=1 (the sealed cache lacks assets/qr.png, so hub require_all cannot complete offline).
  • Frozen suite references: ~/.local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2/. Diffusers four-reference baseline for PSNR: attempt 2026-10-07T170729Z-q3-4r under .../image/attempts/ (samples.json maps fixture+seed to files).

RTX 5080 host (16 GB leg; powered on for multi-model-helper-server #164):

  • cachyos-9900x-o11d at 192.168.0.40, user anodyine, key-based SSH from the workstation. Often powered off; the owner starts it. 30.45 GiB non-expandable host RAM, zram swap at high swappiness.
  • RTX 5080 16 GB (sm_120, Blackwell — native FP8 compute), same driver 615.71.09 / CUDA 13.4. GPU-in-Docker configured.
  • No vLLM-Omni environment yet: stand one up with #1's recipe (Python 3.13 venv — check /usr/bin/python3.13 exists there; vllm 0.31.0; torch cu13x wheel; transformers 5.14.1; diffusers git 122b1e11; repo editable; requirements/common.txt minus the diffusers pin). Clone into ~/Projects/vllm-omni-val/ with origin = this mirror, upstream = GitHub.
  • Weights already on host from #155: the same pinned snapshot, the frozen references-suite-2 set, and the Q1d baseline outputs under the identical roots (~/.local/state/multi-model-helper-server/state/image-service/weights, ~/.local/share/multi-model-helper-server/data/evaluation/media/image/). Verify with the helper repo's media-eval cache-check and python3 -m evaluation.media.image check from a checkout at merged main.
  • The helper-system checkout on that host also carries the B-series quantized artifacts (FP8 dynamic DiT) — irrelevant to this probe but do not disturb them.

Both legs: eager mode, single request at a time, fixed seeds; record peak VRAM (nvidia-smi series), wall time, and server-reported metrics. Nothing posts upstream without owner review.

Scope

Two distinct questions, in order, on both cards:

  1. Functional first: serve one-, two-, and four-condition-image requests through the API using the frozen suite references. The reference path may fail before memory is ever the question; capture that honestly if so.
  2. Locate the cache: find the conditioning-KV implementation and its allocator in the clone (file and line), and which server flags reach it, if any (--swap-space, memory-utilization, diffusion-specific options).
  3. Baseline at default flags: the four-condition-image request, recording peak VRAM, wall time, and PSNR against the sealed four-reference Diffusers attempt (2026-10-07T170729Z-q3-4r, same references, 20 steps).
  4. Knob run: restart the server with the candidate offload knobs and repeat the identical request; compare VRAM peak and wall time.
  5. State the verdict per card — exactly one of:
    • fits-resident: the reference count that matters fits VRAM with the cache on at default flags;
    • offload-works: a flag demonstrably moves conditioning KV to host DRAM with acceptable latency (named flag, measured numbers);
    • not-built: no path exists; name file and line where a contribution would go, and whether it still serves us before committing to anything upstream-sized.

Expected readings going in: on the 3090 (24 GB), the question is whether four references can run cached at all; on the 5080 (16 GB), whether even two or three fit — its Diffusers ceiling is one reference (B2g). No code changes; a probe only.

Acceptance Criteria

  • The reference path's functional status is recorded per card before any memory probe.
  • The conditioning-KV implementation is identified by file and line, with its allocator and reachable flags named.
  • Both runs record peak VRAM, wall time, and PSNR against the Diffusers baseline; inputs and flags are reproducible from the record.
  • The verdict names one of the three worlds per card, with the evidence behind it.
  • Any helper-system follow-through (serving-profile candidacy, topology input) is recorded as input to multi-model-helper-server #27 and #164, not implemented here.

Out of scope

  • Code changes or upstream PRs (a contribution decision follows the verdict, separately).
  • Multi-GPU or TP paths (blocked upstream by the chroma-speckle bug; separate investigation in #3).
  • Helper-system deployment evidence.
Depends on #1 (validated runtime on sm_86). ## Context The helper-system's four-reference work measured that Diffusers retains the Qwen-Image conditioning KV (~2.8 GiB per reference cached) in VRAM with no spill path, so four references fit only with the cache off, at a 1.7-2.2x latency tax (multi-model-helper-server #160; `docs/speculative/research/qwen-image-kv-cache-and-reference-capacity.md`). vLLM-Omni's merged Qwen-Image pipeline implements its own conditioning KV cache (their review history names a "KV-cache aliasing bug in decode graphs" and "mixed-phase KV assembly") and natively accepts up to four condition images, but #1's validation exercised only text-to-image — the reference path is functionally untested on our stack, and whether that cache is paged, swappable to host DRAM, or simply leaner than Diffusers' retention is unverified anywhere. If any of those hold, multi-reference compositions could run at cached speed on cards where Diffusers cannot fit them. ## Host and environment facts **RTX 3090 host (24 GB leg):** - `cachyos-alexander` at `192.168.0.39`, user `anodyine`, key-based SSH from the workstation (`ssh anodyine@192.168.0.39`); user lingering on for `systemd-run --user` units. - RTX 3090 24 GB (sm_86), driver 615.71.09, CUDA 13.4. The host's `relayview-inference.service` (vLLM worker that grabs the whole GPU) is owner-controlled and its enabled state varies between sessions — check `nvidia-smi` first and ask the owner if it holds the card; do not stop it yourself. - Working environment from #1: Python 3.13 venv at `~/Projects/vllm-omni-val/venv` over the clone at `~/Projects/vllm-omni-val/vllm-omni` (origin = this mirror, upstream = GitHub; main `8484d85d1`). Installed: vllm 0.31.0 (0.30.0 fails at import on this commit), torch 2.13.0+cu130, transformers 5.14.1, diffusers git `122b1e11`, repo editable, plus `requirements/common.txt` runtime/test deps (diffusers kept at the git pin). - Weights: digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` at `~/.local/state/multi-model-helper-server/state/image-service/weights` — serve from the local snapshot path with `HF_HUB_OFFLINE=1` (the sealed cache lacks `assets/qr.png`, so hub `require_all` cannot complete offline). - Frozen suite references: `~/.local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2/`. Diffusers four-reference baseline for PSNR: attempt `2026-10-07T170729Z-q3-4r` under `.../image/attempts/` (samples.json maps fixture+seed to files). **RTX 5080 host (16 GB leg; powered on for multi-model-helper-server #164):** - `cachyos-9900x-o11d` at `192.168.0.40`, user `anodyine`, key-based SSH from the workstation. Often powered off; the owner starts it. 30.45 GiB non-expandable host RAM, zram swap at high swappiness. - RTX 5080 16 GB (sm_120, Blackwell — native FP8 compute), same driver 615.71.09 / CUDA 13.4. GPU-in-Docker configured. - No vLLM-Omni environment yet: stand one up with #1's recipe (Python 3.13 venv — check `/usr/bin/python3.13` exists there; vllm 0.31.0; torch cu13x wheel; transformers 5.14.1; diffusers git `122b1e11`; repo editable; `requirements/common.txt` minus the diffusers pin). Clone into `~/Projects/vllm-omni-val/` with origin = this mirror, upstream = GitHub. - Weights already on host from #155: the same pinned snapshot, the frozen `references-suite-2` set, and the Q1d baseline outputs under the identical roots (`~/.local/state/multi-model-helper-server/state/image-service/weights`, `~/.local/share/multi-model-helper-server/data/evaluation/media/image/`). Verify with the helper repo's `media-eval cache-check` and `python3 -m evaluation.media.image check` from a checkout at merged main. - The helper-system checkout on that host also carries the B-series quantized artifacts (FP8 dynamic DiT) — irrelevant to this probe but do not disturb them. **Both legs:** eager mode, single request at a time, fixed seeds; record peak VRAM (`nvidia-smi` series), wall time, and server-reported metrics. Nothing posts upstream without owner review. ## Scope Two distinct questions, in order, on both cards: 1. **Functional first**: serve one-, two-, and four-condition-image requests through the API using the frozen suite references. The reference path may fail before memory is ever the question; capture that honestly if so. 2. **Locate the cache**: find the conditioning-KV implementation and its allocator in the clone (file and line), and which server flags reach it, if any (`--swap-space`, memory-utilization, diffusion-specific options). 3. **Baseline at default flags**: the four-condition-image request, recording peak VRAM, wall time, and PSNR against the sealed four-reference Diffusers attempt (`2026-10-07T170729Z-q3-4r`, same references, 20 steps). 4. **Knob run**: restart the server with the candidate offload knobs and repeat the identical request; compare VRAM peak and wall time. 5. **State the verdict per card** — exactly one of: - **fits-resident**: the reference count that matters fits VRAM with the cache on at default flags; - **offload-works**: a flag demonstrably moves conditioning KV to host DRAM with acceptable latency (named flag, measured numbers); - **not-built**: no path exists; name file and line where a contribution would go, and whether it still serves us before committing to anything upstream-sized. Expected readings going in: on the 3090 (24 GB), the question is whether four references can run cached at all; on the 5080 (16 GB), whether even two or three fit — its Diffusers ceiling is one reference (B2g). No code changes; a probe only. ## Acceptance Criteria - The reference path's functional status is recorded per card before any memory probe. - The conditioning-KV implementation is identified by file and line, with its allocator and reachable flags named. - Both runs record peak VRAM, wall time, and PSNR against the Diffusers baseline; inputs and flags are reproducible from the record. - The verdict names one of the three worlds per card, with the evidence behind it. - Any helper-system follow-through (serving-profile candidacy, topology input) is recorded as input to multi-model-helper-server #27 and #164, not implemented here. ## Out of scope - Code changes or upstream PRs (a contribution decision follows the verdict, separately). - Multi-GPU or TP paths (blocked upstream by the chroma-speckle bug; separate investigation in #3). - Helper-system deployment evidence.
anodyine added the type/testarea/backendstatus/ready-for-agent labels 2026-10-07 21:02:38 -04:00
anodyine added status/in-progress and removed status/ready-for-agent labels 2026-10-08 00:15:13 -04:00
anodyine added reference ISSUE-2 2026-10-08 00:15:15 -04:00
anodyine commented 2026-10-08 00:59:16 -04:00
Author
Owner
Copy link
Copy source

Implemented in PR: #4

Verdicts (per the issue's three-world taxonomy)

RTX 3090 (24 GB): not-built. No host-DRAM spill path for the conditioning KV exists in the pipeline (code-verified and confirmed by exhausting every reachable knob). Corollary findings:

  • Default flags cannot even load the server (OOM at init, 23.51 GiB at load). At --enable-cpu-offload the served reference ceiling is two condition images, conditioning cache ON: 1-ref 34.2 s, 2-ref 36.7 s, peak 24,015 MiB. 4-ref OOMs mid-request.
  • The 4-ref failure is not the conditioning KV: the cache is per-request and lean (~0.25–0.3 GiB per reference, freed at request end — nothing like Diffusers' 2.8 GiB/reference persistent retention). The binding constraint is the resident BF16 working set plus a fixed ~490 MiB image-MLP activation (qwen_image_21_transformer.py:293). expandable_segments:True removes fragmentation entirely; 4-ref still OOMs with 22.29 GiB genuinely allocated. Layerwise offload is strictly counterproductive (1-ref OOMs). Offline fp8_v (Omni(extras=...)) — the one knob that reaches the cache — still OOMs: cache quantization is an order of magnitude too small to matter.
  • Measured 2-ref (pair, seeds 2301–2304): 36.6–36.9 s/image vs Diffusers 81.16 s (2.2×), PSNR mean 27.57 dB (26.09–28.92) vs the sealed 170729Z-q3-4r pair samples. The 4-ref PSNR cell is capacity-blocked on every flag set — recorded as such.
  • Bonus finding: the recipe's deploy-YAML extras.prefix_kv_cache_dtype route does not reach the served transformer on 8484d85d (byte-identical outputs with/without; drop verified at build_diffusion_config). The offline Omni(extras=…) engine kwarg works.

RTX 5080 (16 GB): not-built (stronger — server cannot initialize). Six init attempts (default, cpu-offload, +layerwise, fp8, fp8+expandable, int8) all OOM at the same 96 MiB late-init allocation: init_parameters (vllm_omni/diffusion/models/utils.py:159-168) materializes the unquantized parameter skeleton on-device before quantization or offload engage. The reference path is unreachable on this card with this stack; the B-series torchao-int8 Diffusers stack (1-ref ceiling) remains its only image backend.

Cache location (scope item 2)

Created per request at pipeline_qwen_image_21.py:1204-1207, written/read at qwen_image_21_transformer.py:538-598, batch assembly at pipeline_qwen_image_21.py:1218-1325. Allocator: plain GPU torch.Tensor clones (DENSE_LEGACY, not paged; FP8-storage optional). --swap-space/--cpu-offload-gb do not exist in this fork; weight-offload flags never move KV tensors. Contribution seam for a spill path: qwen_image_21_transformer.py:541-598 or pipeline_qwen_image_21.py:1218-1325 — but a spill alone would not change either verdict (the 3090 blocks on the resident working set, the 5080 on init).

Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)

  • #27: vLLM-Omni candidacy holds for ≤2-reference serving profiles on 24 GB (2.2× nocache Diffusers, no cross-request cache accumulation). 4-reference profiles stay Diffusers-only until upstream load/init shrinks.
  • #164: the 5080 cannot host this stack at all until the init path materializes quantized/CPU-side (diffusion/models/utils.py:159-168); topology plans should keep B-series Diffusers on that card.

Full matrices, reproducibility commands, and per-run records: probe/issue-2/FINDINGS.md in PR #4. No code changes, no upstream posts.

Implemented in PR: https://tea.kyle-merritt.com/anodyine/vllm-omni/pulls/4 ## Verdicts (per the issue's three-world taxonomy) **RTX 3090 (24 GB): `not-built`.** No host-DRAM spill path for the conditioning KV exists in the pipeline (code-verified and confirmed by exhausting every reachable knob). Corollary findings: - Default flags cannot even load the server (OOM at init, 23.51 GiB at load). At `--enable-cpu-offload` the served reference ceiling is **two condition images, conditioning cache ON**: 1-ref 34.2 s, 2-ref 36.7 s, peak 24,015 MiB. 4-ref OOMs mid-request. - The 4-ref failure is **not the conditioning KV**: the cache is per-request and lean (~0.25–0.3 GiB per reference, freed at request end — nothing like Diffusers' 2.8 GiB/reference persistent retention). The binding constraint is the resident BF16 working set plus a fixed ~490 MiB image-MLP activation (`qwen_image_21_transformer.py:293`). `expandable_segments:True` removes fragmentation entirely; 4-ref still OOMs with 22.29 GiB genuinely allocated. Layerwise offload is strictly counterproductive (1-ref OOMs). Offline fp8_v (`Omni(extras=...)`) — the one knob that reaches the cache — still OOMs: cache quantization is an order of magnitude too small to matter. - Measured 2-ref (pair, seeds 2301–2304): **36.6–36.9 s/image vs Diffusers 81.16 s (2.2×)**, PSNR mean **27.57 dB** (26.09–28.92) vs the sealed `170729Z-q3-4r` pair samples. The 4-ref PSNR cell is capacity-blocked on every flag set — recorded as such. - Bonus finding: the recipe's deploy-YAML `extras.prefix_kv_cache_dtype` route **does not reach the served transformer** on `8484d85d` (byte-identical outputs with/without; drop verified at `build_diffusion_config`). The offline `Omni(extras=…)` engine kwarg works. **RTX 5080 (16 GB): `not-built` (stronger — server cannot initialize).** Six init attempts (default, cpu-offload, +layerwise, fp8, fp8+expandable, int8) all OOM at the same 96 MiB late-init allocation: `init_parameters` (`vllm_omni/diffusion/models/utils.py:159-168`) materializes the unquantized parameter skeleton on-device **before** quantization or offload engage. The reference path is unreachable on this card with this stack; the B-series torchao-int8 Diffusers stack (1-ref ceiling) remains its only image backend. ## Cache location (scope item 2) Created per request at `pipeline_qwen_image_21.py:1204-1207`, written/read at `qwen_image_21_transformer.py:538-598`, batch assembly at `pipeline_qwen_image_21.py:1218-1325`. Allocator: plain GPU `torch.Tensor` clones (`DENSE_LEGACY`, not paged; FP8-storage optional). `--swap-space`/`--cpu-offload-gb` do not exist in this fork; weight-offload flags never move KV tensors. Contribution seam for a spill path: `qwen_image_21_transformer.py:541-598` or `pipeline_qwen_image_21.py:1218-1325` — but a spill alone would not change either verdict (the 3090 blocks on the resident working set, the 5080 on init). ## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here) - **#27**: vLLM-Omni candidacy holds for ≤2-reference serving profiles on 24 GB (2.2× nocache Diffusers, no cross-request cache accumulation). 4-reference profiles stay Diffusers-only until upstream load/init shrinks. - **#164**: the 5080 cannot host this stack at all until the init path materializes quantized/CPU-side (`diffusion/models/utils.py:159-168`); topology plans should keep B-series Diffusers on that card. Full matrices, reproducibility commands, and per-run records: `probe/issue-2/FINDINGS.md` in PR #4. No code changes, no upstream posts.
anodyine added status/review and removed status/in-progress labels 2026-10-08 00:59:25 -04:00
anodyine commented 2026-10-08 06:28:55 -04:00
Author
Owner
Copy link
Copy source

Review follow-up (PR #4, commit 4f5f99d8): the fp8 serve row is now in the matrix.

  • Capacity — reviewer's projection confirmed: --enable-cpu-offload --quantization fp8 serves all of 1/2/3/4 references on the 3090 (4-ref 44.1–44.4 s, server peak 23,522 MB, nvidia-smi max 23,983 MiB; 3-ref was added as a new fixture and measured too). Marlin W8A16 is faster than BF16 (3.3× the Diffusers nocache baseline at 4 refs).
  • Quality — fp8 output is structurally broken on this commit/card: ~97% transparent RGBA, grey RGB, PSNR 7–9 dB on every fp8 artifact this probe produced (edit path at all reference counts, img_mlp-excluded, and transformer-only fp8 — the last pins the culprit on the Marlin-quantized DiT). BF16 on identical requests: 100% opaque, 26–29 dB.
  • Correction to #1: issue #1's own fp8 T2I artifacts show the same collapse on reinspection; that run validated rc+image, not output structure. The 18.4 GB capacity number stands; the quality implication does not.
  • Verdicts unchanged: 3090 not-built (fp8 unlocks capacity but ships broken output; upstream order is fp8-output correctness before capacity), 5080 not-built (init-blocked).

Details in probe/issue-2/FINDINGS.md § "FP8 quality collapse"; records in evidence/rtx3090/runs-summary.json and psnr-fp8-rows.json.

Review follow-up (PR #4, commit 4f5f99d8): the fp8 serve row is now in the matrix. - **Capacity — reviewer's projection confirmed**: `--enable-cpu-offload --quantization fp8` serves all of 1/2/3/4 references on the 3090 (4-ref 44.1–44.4 s, server peak 23,522 MB, nvidia-smi max 23,983 MiB; 3-ref was added as a new fixture and measured too). Marlin W8A16 is faster than BF16 (3.3× the Diffusers nocache baseline at 4 refs). - **Quality — fp8 output is structurally broken on this commit/card**: ~97% transparent RGBA, grey RGB, PSNR 7–9 dB on every fp8 artifact this probe produced (edit path at all reference counts, `img_mlp`-excluded, and transformer-only fp8 — the last pins the culprit on the Marlin-quantized DiT). BF16 on identical requests: 100% opaque, 26–29 dB. - **Correction to #1**: issue #1's own fp8 T2I artifacts show the same collapse on reinspection; that run validated rc+image, not output structure. The 18.4 GB capacity number stands; the quality implication does not. - Verdicts unchanged: 3090 `not-built` (fp8 unlocks capacity but ships broken output; upstream order is fp8-output correctness before capacity), 5080 `not-built` (init-blocked). Details in `probe/issue-2/FINDINGS.md` § "FP8 quality collapse"; records in `evidence/rtx3090/runs-summary.json` and `psnr-fp8-rows.json`.
Sign in to join this conversation.
ISSUE-2
Labels
Clear labels
area/agent-workflow
Agent workflow area
 area/backend
Backend area
 area/docs
Documentation area
 area/frontend
Frontend area
 area/tests
Testing area
 area/workflow
General workflow area
 priority/P0
Exceptional interrupt; not a complete priority taxonomy
 status/done
Work is complete
 status/in-progress
Implementation is in progress
 status/needs-info
Blocked pending more information
 status/needs-triage
Needs initial review and prioritization
 status/ready-for-agent
Ready for an agent to claim
 status/ready-for-human
Ready for human review or action
 status/review
Implementation is ready for review
 status/wontfix
Work will not be pursued
 type/architecture
Architecture or system-design work
 type/bug
Defect or regression
 type/docs
Documentation work
 type/epic
Epic grouping related work
 type/feature
New behavior or capability
 type/refactor
Internal restructuring without behavior change
 type/roadmap
Canonical repository work queue
 type/test
Test coverage or test infrastructure
No labels area/backend status/review type/test
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
anodyine
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Depends on
#1 Validate Qwen-Image-2.1 serving on Ampere (RTX 3090) and report upstream
anodyine/vllm-omni
Reference: anodyine/vllm-omni#2