vllm-omni/probe/issue-2/FINDINGS.md

# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB)

Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and
`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`,
vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
transformers 5.14.1, diffusers git `122b1e11`), served from the
digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with
`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds,
20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed
Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile
`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same
frozen references-suite-2 images, same rendered brief-prose/1 prompts,
same seeds 2301–2304, same per-fixture resolutions.

## Verdicts

**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the
conditioning KV exists anywhere in the pipeline (verified by code read and
by exhausting every reachable knob). At BF16 the served reference ceiling
is two condition images at `--enable-cpu-offload` — with the conditioning
cache ON, at 2.2× the Diffusers nocache baseline's speed. Four references
OOM inside the DiT forward under every BF16 flag combination, and the
binding constraint is the resident BF16 working set plus a fixed ~490 MiB
image-MLP activation, not the conditioning KV: the per-request cache
contributes only roughly 0.25–0.3 GiB per reference (see "KV size" below).
FP8 weight quantization (`--quantization fp8`) does free the ~6 GiB the
arithmetic calls for — all of one, two, three, and four references then
fit and run (4-ref: 44.1–44.4 s, server peak 23,522 MB, nvidia-smi series
max 23,983 MiB) — **but on this commit and card the FP8 path produces
structurally broken output** (~96.5–97% transparent RGBA, grey noise RGB,
PSNR 7–9 dB), for the edit path at every reference count and for T2I
alike (see "FP8 quality collapse"). So the verdict stands: no usable path
to four references, and none that spills the KV.

**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.**
Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8,
fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB
allocation with the card full. The failure is in
`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the
unquantized parameter skeleton is allocated on-device at construction,
before any quantization or offload path engages, so no runtime flag can
shrink the load-time peak. The reference path is unreachable on this card
with this stack; its one-reference Diffusers ceiling (B2g) relied on a
torchao-int8 + group-offload load path that never materializes the full
BF16 model on the GPU. The conditioning-KV question is moot on this card.

## Where the cache lives (issue scope item 2)

- Created per request at
  `vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207`
  (`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG
  branch `cond`/`uncond`), valid because `causal_condition=True` makes
  text + condition-image K/V timestep-independent.
- Written at prefill (first denoise step) and read at decode in
  `qwen_image_21/.../qwen_image_21_transformer.py:538-598`
  (`branch_cache["key"] = key[:, :cache_write_len].clone()` and the
  decode-side `torch.cat` prepend).
- Batched mixed-phase assembly/scatter at
  `pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and
  `:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in
  `qwen_image_21/decode_graph.py` (inactive here — eager).
- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's
  paged KVCacheManager — the model deliberately runs `DENSE_LEGACY`
  (`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of
  `PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via
  `extras.prefix_kv_cache_dtype`.
- **Lifetime**: per-request only; freed by refcount when the runner drops
  the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`).
  No cross-request retention, no LRU, no dedup — unlike Diffusers' helper
  cache, nothing persists between requests.
- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` /
  `fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the
  propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode`
  (static-buffer copy), `cuda_graph_max_decode_graphs`.
  `--swap-space` and `--cpu-offload-gb` **do not exist** in this fork;
  `--gpu-memory-utilization` sizes only the unrelated paged-scheduler
  path. The weight-offload flags (`--enable-cpu-offload`,
  `--enable-layerwise-offload`) never move KV tensors.
- **Spill path**: none. A host-DRAM contribution would go at the prefill
  write / decode read (`qwen_image_21_transformer.py:541-598` — allocate
  on pinned CPU, prefetch before the decode `torch.cat`) or at the batch
  boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the
  request-owns-independent-storage invariant that the decode-graph weakref
  identity check at `decode_graph.py:350-363` depends on.

## KV size (empirical)

Between one and two references the per-request peak moves by only
~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+
expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image —
roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the
helper-system measured Diffusers retaining (multi-model-helper-server
#160). The cache is per-request and lean; it is not what blocks four
references.

## Run matrix — RTX 3090

All runs: `vllm serve <snapshot> --omni --port 8091 --enforce-eager
--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via
`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under
`~/Projects/vllm-omni-val/issue2/runs/<label>/` on the host.

| Run | Flags | Init | 1-ref | 2-ref | 4-ref | nvidia-smi peak |
| --- | --- | --- | --- | --- | --- | --- |
| default | (none) | **OOM in 14 s** (23.51 GiB in use at load) | — | — | — | 23,939 MiB |
| cpuoff | `--enable-cpu-offload` | ok | OK 34.2 s | OK 36.7 s | **OOM** in request | 24,015 MiB |
| cpuoff-layerwise | `+ --enable-layerwise-offload` | ok | **OOM** | **OOM** | **OOM** | 24,137 MiB |
| cpuoff-fp8v (yaml) | `+ --deploy-config deploy-fp8v.yaml` | ok | OK 34.3 s | OK 36.8 s | **OOM** | 24,015 MiB |
| cpuoff-expandable | `+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ok | OK 33.8 s | OK 36.7 s | **OOM** | 24,095 MiB |
| fp8 | `--enable-cpu-offload --quantization fp8` | ok | OK 29.9 s | OK 34.1 s (+3-ref OK 38.9 s) | OK 44.3 s — **output broken** | 23,983 MiB |

The `fp8` row (review follow-up; #1 validated this flag's capacity
offline) also covered the previously untested three-reference count via
`edit-compose-three` — derived from `edit-compose-four` minus
`ref-shop-hesper` and its Subject line, four→three wording, same
instrument (recorded in `fixtures.json`). Measured under fp8, seeds
2301–2304: four 44.06–44.35 s (peak 22,918 MB), pair 34.62–35.05 s
(peak 21,504 MB), three 39.44–39.69 s (peak 23,532 MB) — every request
succeeds capacity-wise, and Marlin W8A16 is *faster* than BF16
(4-ref 44.3 s vs Diffusers' 145.8 s = 3.3×). The quality outcome is in
the next section.

- 4-ref OOM signature (identical ask across configs): `Tried to allocate
  490.00 MiB` inside the image-branch MLP forward
  (`qwen_image_21_transformer.py:293`, via `predict_noise` — prefill
  encoding has already succeeded). At cpuoff: 21.02 GiB allocated + 1.84
  GiB reserved-unallocated; at cpuoff-expandable: 22.29 GiB allocated,
  505 MiB reserved-unallocated — fragmentation removed, still short:
  genuine capacity gap.
- Layerwise offload is strictly counterproductive here: even one reference
  OOMs (21.76 GiB allocated during the request).
- Determinism: seed-2301 outputs are byte-identical (sha256
  `45fac0b8…` pair, `3c4eb2ef…` single) across cpuoff, cpuoff-fp8v,
  cpuoff-expandable — the frozen-seed instrument is sound.

### FP8 quality collapse (and a correction to issue 1)

Every FP8-weight output this probe produced is structurally broken, not
merely noisier:

- Signature (identical everywhere): RGBA output ~96.5–97% transparent
  (opaque fraction 0.008–0.035, alpha mean ≈115), near-flat grey RGB
  (mean ≈120–130), PSNR vs the Diffusers baseline 6.6–9.0 dB. The BF16
  path on the same requests is 100% opaque and 26–29 dB.
- Scope: the serve path at 1, 2, 3, and 4 references (all seeds measured),
  the offline path with `fp8` minus `img_mlp`, and the offline path with
  **transformer-only** FP8 (`quantization_config={"transformer":
  {"method": "fp8"}}` — the text encoder stays BF16). Transformer-only
  still collapsing pins the culprit on the Marlin-quantized DiT, not the
  encoder and not the image-MLP. An encoder-only isolator could not
  initialize (the encoder's bf16→fp8 conversion transient lands on the GPU
  on top of the resident BF16 DiT and exceeds the card).
- **Correction to issue 1**: #1's offline FP8 validation
  (`qwen_image_21_fp8_all_rtx3090.png`, `…_fp8_imgmlp_…`) shows the same
  collapse on reinspection (opaque fraction 0.035 both) — that run
  validated rc=0 and image production, not output structure, so "FP8
  weight-only works on this card" from #1 overstated it. #1's 18.4 GB
  capacity measurement stands; its quality implication does not.
- Upstream relevance: the recipe's H200 FP8 numbers quote PSNR only for
  text-encoder FP8 (32.1 dB T2I); DiT-FP8 output quality appears
  unmeasured upstream, and this probe adds the Ampere/Marlin cell:
  online FP8 DiT weights yield near-noise latents for Qwen-Image-2.1.
  Until that is fixed (or validated as Ampere-specific), FP8 cannot serve
  as the capacity unlock on this card — which is why the 3090 verdict
  remains `not-built` despite the fp8 capacity row passing.

### Offline knob runs (3090, `Omni(...)` engine kwargs)

The one knob that reaches the cache — `extras.prefix_kv_cache_dtype` —
was exercised offline (`offline_fp8v.py`, `--enforce-eager
--enable-cpu-offload` + engine kwarg):

| Run | 4-ref seed 2301 | Detail |
| --- | --- | --- |
| offline fp8_v | **OOM** | 20.86 GiB allocated + 2.14 GiB reserved-unallocated; same 490 MiB img-MLP ask |
| offline fp8_v + expandable | **OOM** | 22.22 GiB allocated, 483 MiB reserved-unallocated |

FP8-V halves the V half of a cache that contributes only ~0.3 GiB per
reference — an order of magnitude too small to bridge the gap. The
conditioning KV cannot rescue four references on this card by quantization
or by anything else; only a genuinely leaner resident working set
(quantized DiT weights) would, and on this commit that is blocked by the
init path (see 5080).

### Measured run + PSNR (3090)

`edit-compose-pair` (2 refs, 1248×832), seeds 2301–2304, on
cpuoff-expandable, versus the sealed Diffusers attempt's pair samples
(same references, 20 steps; Diffusers side ran an int8-quantized DiT with
KV cache off, wall 81.16 s):

- Wall: **36.6–36.9 s per image** (2.2× faster than Diffusers nocache;
  server `denoise_step_latency` ≈ 1.83 s × 20 steps).
- Peak: 21,976–23,332 MB server-reported; 24,095 MiB nvidia-smi series
  max.
- PSNR vs Diffusers (RGBA-on-RGBA, #1's comparator): **26.09 / 28.75 /
  26.51 / 28.92 dB; mean 27.57 dB** (n=4). Cross-implementation agreement
  at matched seed/inputs, not parity — expected given the different
  samplers/kernels and the Diffusers side's int8 DiT.

The four-reference PSNR cell the issue nominally asks for is
**capacity-blocked at BF16** (every served flag set, and offline with
fp8_v). Under the fp8 serve row it becomes obtainable but meaningless as
quality: measured seeds 2301–2304 give 7.03–7.21 dB (mean 7.11) against
the sealed 4r baseline because the outputs are the collapsed ~97%
transparent renders of the FP8 quality section — recorded in
`evidence/rtx3090/psnr-fp8-rows.json` as the broken-output datum, not as
a usable comparison. The pair row above (27.57 dB at BF16) remains the
highest-reference-count usable comparison this card can produce.
`edit-compose-four` wall for reference: Diffusers 145.8 s (nocache);
vLLM 44.3 s under fp8 (fast but broken) — no valid BF16 number exists.

## Run matrix — RTX 5080

Environment built fresh per #1's recipe (userspace CPython 3.13.16 venv —
the host only ships 3.14 and has no `/usr/bin/python3.13`; clone pinned to
`8484d85d1`; weights/suite verified, the `q3-4r` attempt copied from the
3090 so both legs compare against the identical baseline).

| Run | Flags | Init result |
| --- | --- | --- |
| default | (none) | OOM in 12 s (15.22 GiB allocated at load) |
| cpuoff | `--enable-cpu-offload` | **OOM in 9 s** — DiT stays GPU-resident under cpu-offload; 14.99 GiB allocated |
| cpuoff-layerwise | `+ --enable-layerwise-offload` | **OOM in 10 s** — layerwise streams per-forward, after load |
| fp8 | `--enable-cpu-offload --quantization fp8` | **OOM in ~10 s** — FP8 path engaged (DeepGEMM init visible) but init still fills the card |
| fp8-expandable | `+ expandable_segments` | **OOM** |
| int8 | `--enable-cpu-offload --quantization int8` | **OOM in 8 s** — 15.55 GiB in use |

Common failure: `torch.OutOfMemoryError` at a 96 MiB allocation in
`vllm_omni/diffusion/models/utils.py:159-168 init_parameters`
(`empty_like` under a device context) — the unquantized parameter
skeleton is materialized on-device at pipeline construction, before
weight loading, quantization, or offload can act. Host RAM (30 GiB + zram)
is a separate ceiling for full weight offload (~32 GiB of BF16 weights)
but is not what kills these runs.

## Propagation gap found (3090 leg)

The recipe-documented deploy-YAML route for
`extras.prefix_kv_cache_dtype` **does not reach the transformer through
`vllm serve` on this commit**: with and without the YAML, serve outputs
are byte-identical, and the drop is verified offline at the exact
serve-time construction (`build_diffusion_config` →
`stage_configs[0].diffusion_config.extras` carries only
`auxiliary_text_encoder`/`default_llama_model_id` defaults; deploy stage
`engine_extras.extras` is parsed correctly by `_stage_engine_overrides`
but lost in the structured-config bridge). The offline `Omni(extras=…)`
engine kwarg is accepted (load proceeds; the knob then behaves as
documented). Shapes tried: flat stage key, `engine_extras.<key>`, and
`engine_extras.extras.<key>`. This is a one-line-ish fix candidate in the
config bridge, independent of any KV-spill contribution.

## Reproducibility

Scripts live in this directory (`serve.sh`, `probe.py`, `psnr.py`,
`offline_fp8v.py`, `deploy-fp8v.yaml`) and were pushed verbatim to
`~/Projects/vllm-omni-val/issue2/` on each host; `fixtures.json` (rendered
brief-prose/1 prompts, reference bindings, dimensions) was generated on
the 3090 from the helper repo (`evaluation/media/image/suite.py` +
`services/media/image_service/renderer.py`) and copied to the 5080.
Server lifecycle ran under transient user units
(`systemd-run --user --unit=issue2-serve`); per-run serve/vram logs and
request records are under `runs/<label>/` on each host; condensed
summaries in `evidence/rtx3090/runs-summary.json` and
`evidence/rtx5080/runs-summary.json`.

## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)

- **#27 (serving-profile candidacy)**: on the 3090 the vLLM-Omni path
  serves 1–2 references at 33–37 s/image with the conditioning cache on —
  2.2× the sealed Diffusers nocache baseline — and per-request cache
  pressure of ~0.3 GiB/reference means concurrent multi-reference sessions
  do not accumulate Diffusers-style 2.8 GiB/reference retention. Candidacy
  holds for ≤2-reference profiles on 24 GB; 4-reference profiles remain
  Diffusers-only: FP8 unlocks the capacity (all four counts fit, 3.3×
  baseline speed) but its output is broken on this commit, so the unlock
  order is upstream FP8-output correctness first, then serving.
- **#164 (topology input)**: the 5080 (16 GB) cannot host this stack at
  all — init-time parameter skeleton exceeds the card before any flag
  acts (`diffusion/models/utils.py:159-168`). Any topology plan pairing
  the 5080 with Qwen-Image-2.1 on vLLM-Omni needs upstream load-path work
  (quantized or CPU-materialized init) first; the B-series torchao-int8
  Diffusers stack remains that card's only viable image backend.
- Upstream contribution seam, if ever pursued: prefill-write/decode-read
  at `qwen_image_21_transformer.py:541-598` or the batch helpers at
  `pipeline_qwen_image_21.py:1218-1325`; but note the cache itself is not
  the binding constraint on either card — the 3090 blocks on the resident
  BF16 working set + img-MLP activation, the 5080 on init. A spill path
  alone would not change either verdict, which answers the issue's
  "does it still serve us before committing to anything upstream-sized":
  **no — not as the next step**; the load/init path comes first.

## Out-of-scope confirmation

No code changes; no upstream PRs; no TP/multi-GPU paths touched; no
helper-system deployment evidence produced. The only artifacts are this
report, the probe scripts, and per-host run records.