admin
67eb7fdb48
Evidence-only change: probe kit (serve wrapper, API client, PSNR
comparator, offline fp8_v driver), per-host run summaries, and the
findings report. No code changes to the stack.
Verdicts: not-built on both cards. On the 3090 the served reference
ceiling is two condition images with the conditioning cache on
(2.2x the Diffusers nocache baseline, pair PSNR mean 27.57 dB);
four references OOM on the resident BF16 working set, not the KV
(~0.3 GiB/reference, per-request). On the 5080 the server cannot
initialize under any flag: init_parameters materializes the
unquantized skeleton on-device before quantization/offload engage.
Also documents that the recipe's deploy-YAML extras route for
prefix_kv_cache_dtype does not reach the served transformer on
8484d85d (offline Omni engine kwarg does).
240 lines
13 KiB
Markdown
240 lines
13 KiB
Markdown
# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB)
|
||
|
||
Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and
|
||
`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`,
|
||
vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||
transformers 5.14.1, diffusers git `122b1e11`), served from the
|
||
digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with
|
||
`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds,
|
||
20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed
|
||
Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile
|
||
`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same
|
||
frozen references-suite-2 images, same rendered brief-prose/1 prompts,
|
||
same seeds 2301–2304, same per-fixture resolutions.
|
||
|
||
## Verdicts
|
||
|
||
**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the
|
||
conditioning KV exists anywhere in the pipeline (verified by code read and
|
||
by exhausting every reachable knob). The served reference ceiling is two
|
||
condition images at `--enable-cpu-offload` — with the conditioning cache
|
||
ON, at 2.2× the Diffusers nocache baseline's speed. Four references OOM
|
||
inside the DiT forward under every flag combination, and the binding
|
||
constraint is the resident BF16 working set plus a fixed ~490 MiB image-MLP
|
||
activation, not the conditioning KV: the per-request cache contributes only
|
||
roughly 0.25–0.3 GiB per reference (see "KV size" below).
|
||
|
||
**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.**
|
||
Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8,
|
||
fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB
|
||
allocation with the card full. The failure is in
|
||
`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the
|
||
unquantized parameter skeleton is allocated on-device at construction,
|
||
before any quantization or offload path engages, so no runtime flag can
|
||
shrink the load-time peak. The reference path is unreachable on this card
|
||
with this stack; its one-reference Diffusers ceiling (B2g) relied on a
|
||
torchao-int8 + group-offload load path that never materializes the full
|
||
BF16 model on the GPU. The conditioning-KV question is moot on this card.
|
||
|
||
## Where the cache lives (issue scope item 2)
|
||
|
||
- Created per request at
|
||
`vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207`
|
||
(`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG
|
||
branch `cond`/`uncond`), valid because `causal_condition=True` makes
|
||
text + condition-image K/V timestep-independent.
|
||
- Written at prefill (first denoise step) and read at decode in
|
||
`qwen_image_21/.../qwen_image_21_transformer.py:538-598`
|
||
(`branch_cache["key"] = key[:, :cache_write_len].clone()` and the
|
||
decode-side `torch.cat` prepend).
|
||
- Batched mixed-phase assembly/scatter at
|
||
`pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and
|
||
`:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in
|
||
`qwen_image_21/decode_graph.py` (inactive here — eager).
|
||
- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's
|
||
paged KVCacheManager — the model deliberately runs `DENSE_LEGACY`
|
||
(`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of
|
||
`PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via
|
||
`extras.prefix_kv_cache_dtype`.
|
||
- **Lifetime**: per-request only; freed by refcount when the runner drops
|
||
the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`).
|
||
No cross-request retention, no LRU, no dedup — unlike Diffusers' helper
|
||
cache, nothing persists between requests.
|
||
- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` /
|
||
`fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the
|
||
propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode`
|
||
(static-buffer copy), `cuda_graph_max_decode_graphs`.
|
||
`--swap-space` and `--cpu-offload-gb` **do not exist** in this fork;
|
||
`--gpu-memory-utilization` sizes only the unrelated paged-scheduler
|
||
path. The weight-offload flags (`--enable-cpu-offload`,
|
||
`--enable-layerwise-offload`) never move KV tensors.
|
||
- **Spill path**: none. A host-DRAM contribution would go at the prefill
|
||
write / decode read (`qwen_image_21_transformer.py:541-598` — allocate
|
||
on pinned CPU, prefetch before the decode `torch.cat`) or at the batch
|
||
boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the
|
||
request-owns-independent-storage invariant that the decode-graph weakref
|
||
identity check at `decode_graph.py:350-363` depends on.
|
||
|
||
## KV size (empirical)
|
||
|
||
Between one and two references the per-request peak moves by only
|
||
~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+
|
||
expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image —
|
||
roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the
|
||
helper-system measured Diffusers retaining (multi-model-helper-server
|
||
#160). The cache is per-request and lean; it is not what blocks four
|
||
references.
|
||
|
||
## Run matrix — RTX 3090
|
||
|
||
All runs: `vllm serve <snapshot> --omni --port 8091 --enforce-eager
|
||
--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via
|
||
`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under
|
||
`~/Projects/vllm-omni-val/issue2/runs/<label>/` on the host.
|
||
|
||
| Run | Flags | Init | 1-ref | 2-ref | 4-ref | nvidia-smi peak |
|
||
| --- | --- | --- | --- | --- | --- | --- |
|
||
| default | (none) | **OOM in 14 s** (23.51 GiB in use at load) | — | — | — | 23,939 MiB |
|
||
| cpuoff | `--enable-cpu-offload` | ok | OK 34.2 s | OK 36.7 s | **OOM** in request | 24,015 MiB |
|
||
| cpuoff-layerwise | `+ --enable-layerwise-offload` | ok | **OOM** | **OOM** | **OOM** | 24,137 MiB |
|
||
| cpuoff-fp8v (yaml) | `+ --deploy-config deploy-fp8v.yaml` | ok | OK 34.3 s | OK 36.8 s | **OOM** | 24,015 MiB |
|
||
| cpuoff-expandable | `+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ok | OK 33.8 s | OK 36.7 s | **OOM** | 24,095 MiB |
|
||
|
||
- 4-ref OOM signature (identical ask across configs): `Tried to allocate
|
||
490.00 MiB` inside the image-branch MLP forward
|
||
(`qwen_image_21_transformer.py:293`, via `predict_noise` — prefill
|
||
encoding has already succeeded). At cpuoff: 21.02 GiB allocated + 1.84
|
||
GiB reserved-unallocated; at cpuoff-expandable: 22.29 GiB allocated,
|
||
505 MiB reserved-unallocated — fragmentation removed, still short:
|
||
genuine capacity gap.
|
||
- Layerwise offload is strictly counterproductive here: even one reference
|
||
OOMs (21.76 GiB allocated during the request).
|
||
- Determinism: seed-2301 outputs are byte-identical (sha256
|
||
`45fac0b8…` pair, `3c4eb2ef…` single) across cpuoff, cpuoff-fp8v,
|
||
cpuoff-expandable — the frozen-seed instrument is sound.
|
||
|
||
### Offline knob runs (3090, `Omni(...)` engine kwargs)
|
||
|
||
The one knob that reaches the cache — `extras.prefix_kv_cache_dtype` —
|
||
was exercised offline (`offline_fp8v.py`, `--enforce-eager
|
||
--enable-cpu-offload` + engine kwarg):
|
||
|
||
| Run | 4-ref seed 2301 | Detail |
|
||
| --- | --- | --- |
|
||
| offline fp8_v | **OOM** | 20.86 GiB allocated + 2.14 GiB reserved-unallocated; same 490 MiB img-MLP ask |
|
||
| offline fp8_v + expandable | **OOM** | 22.22 GiB allocated, 483 MiB reserved-unallocated |
|
||
|
||
FP8-V halves the V half of a cache that contributes only ~0.3 GiB per
|
||
reference — an order of magnitude too small to bridge the gap. The
|
||
conditioning KV cannot rescue four references on this card by quantization
|
||
or by anything else; only a genuinely leaner resident working set
|
||
(quantized DiT weights) would, and on this commit that is blocked by the
|
||
init path (see 5080).
|
||
|
||
### Measured run + PSNR (3090)
|
||
|
||
`edit-compose-pair` (2 refs, 1248×832), seeds 2301–2304, on
|
||
cpuoff-expandable, versus the sealed Diffusers attempt's pair samples
|
||
(same references, 20 steps; Diffusers side ran an int8-quantized DiT with
|
||
KV cache off, wall 81.16 s):
|
||
|
||
- Wall: **36.6–36.9 s per image** (2.2× faster than Diffusers nocache;
|
||
server `denoise_step_latency` ≈ 1.83 s × 20 steps).
|
||
- Peak: 21,976–23,332 MB server-reported; 24,095 MiB nvidia-smi series
|
||
max.
|
||
- PSNR vs Diffusers (RGBA-on-RGBA, #1's comparator): **26.09 / 28.75 /
|
||
26.51 / 28.92 dB; mean 27.57 dB** (n=4). Cross-implementation agreement
|
||
at matched seed/inputs, not parity — expected given the different
|
||
samplers/kernels and the Diffusers side's int8 DiT.
|
||
|
||
The four-reference PSNR cell the issue nominally asks for is
|
||
**capacity-blocked on every served flag set** (and offline with fp8_v);
|
||
the pair row is the highest-reference-count comparison this card can
|
||
produce. `edit-compose-four` single-ref walls for reference:
|
||
Diffusers 145.8 s (nocache); vLLM would-be number unobtainable.
|
||
|
||
## Run matrix — RTX 5080
|
||
|
||
Environment built fresh per #1's recipe (userspace CPython 3.13.16 venv —
|
||
the host only ships 3.14 and has no `/usr/bin/python3.13`; clone pinned to
|
||
`8484d85d1`; weights/suite verified, the `q3-4r` attempt copied from the
|
||
3090 so both legs compare against the identical baseline).
|
||
|
||
| Run | Flags | Init result |
|
||
| --- | --- | --- |
|
||
| default | (none) | OOM in 12 s (15.22 GiB allocated at load) |
|
||
| cpuoff | `--enable-cpu-offload` | **OOM in 9 s** — DiT stays GPU-resident under cpu-offload; 14.99 GiB allocated |
|
||
| cpuoff-layerwise | `+ --enable-layerwise-offload` | **OOM in 10 s** — layerwise streams per-forward, after load |
|
||
| fp8 | `--enable-cpu-offload --quantization fp8` | **OOM in ~10 s** — FP8 path engaged (DeepGEMM init visible) but init still fills the card |
|
||
| fp8-expandable | `+ expandable_segments` | **OOM** |
|
||
| int8 | `--enable-cpu-offload --quantization int8` | **OOM in 8 s** — 15.55 GiB in use |
|
||
|
||
Common failure: `torch.OutOfMemoryError` at a 96 MiB allocation in
|
||
`vllm_omni/diffusion/models/utils.py:159-168 init_parameters`
|
||
(`empty_like` under a device context) — the unquantized parameter
|
||
skeleton is materialized on-device at pipeline construction, before
|
||
weight loading, quantization, or offload can act. Host RAM (30 GiB + zram)
|
||
is a separate ceiling for full weight offload (~32 GiB of BF16 weights)
|
||
but is not what kills these runs.
|
||
|
||
## Propagation gap found (3090 leg)
|
||
|
||
The recipe-documented deploy-YAML route for
|
||
`extras.prefix_kv_cache_dtype` **does not reach the transformer through
|
||
`vllm serve` on this commit**: with and without the YAML, serve outputs
|
||
are byte-identical, and the drop is verified offline at the exact
|
||
serve-time construction (`build_diffusion_config` →
|
||
`stage_configs[0].diffusion_config.extras` carries only
|
||
`auxiliary_text_encoder`/`default_llama_model_id` defaults; deploy stage
|
||
`engine_extras.extras` is parsed correctly by `_stage_engine_overrides`
|
||
but lost in the structured-config bridge). The offline `Omni(extras=…)`
|
||
engine kwarg is accepted (load proceeds; the knob then behaves as
|
||
documented). Shapes tried: flat stage key, `engine_extras.<key>`, and
|
||
`engine_extras.extras.<key>`. This is a one-line-ish fix candidate in the
|
||
config bridge, independent of any KV-spill contribution.
|
||
|
||
## Reproducibility
|
||
|
||
Scripts live in this directory (`serve.sh`, `probe.py`, `psnr.py`,
|
||
`offline_fp8v.py`, `deploy-fp8v.yaml`) and were pushed verbatim to
|
||
`~/Projects/vllm-omni-val/issue2/` on each host; `fixtures.json` (rendered
|
||
brief-prose/1 prompts, reference bindings, dimensions) was generated on
|
||
the 3090 from the helper repo (`evaluation/media/image/suite.py` +
|
||
`services/media/image_service/renderer.py`) and copied to the 5080.
|
||
Server lifecycle ran under transient user units
|
||
(`systemd-run --user --unit=issue2-serve`); per-run serve/vram logs and
|
||
request records are under `runs/<label>/` on each host; condensed
|
||
summaries in `evidence/rtx3090/runs-summary.json` and
|
||
`evidence/rtx5080/runs-summary.json`.
|
||
|
||
## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)
|
||
|
||
- **#27 (serving-profile candidacy)**: on the 3090 the vLLM-Omni path
|
||
serves 1–2 references at 33–37 s/image with the conditioning cache on —
|
||
2.2× the sealed Diffusers nocache baseline — and per-request cache
|
||
pressure of ~0.3 GiB/reference means concurrent multi-reference sessions
|
||
do not accumulate Diffusers-style 2.8 GiB/reference retention. Candidacy
|
||
holds for ≤2-reference profiles on 24 GB; 4-reference profiles remain
|
||
Diffusers-only until the resident-working-set/init path shrinks
|
||
(quantized-load init) or a KV spill lands.
|
||
- **#164 (topology input)**: the 5080 (16 GB) cannot host this stack at
|
||
all — init-time parameter skeleton exceeds the card before any flag
|
||
acts (`diffusion/models/utils.py:159-168`). Any topology plan pairing
|
||
the 5080 with Qwen-Image-2.1 on vLLM-Omni needs upstream load-path work
|
||
(quantized or CPU-materialized init) first; the B-series torchao-int8
|
||
Diffusers stack remains that card's only viable image backend.
|
||
- Upstream contribution seam, if ever pursued: prefill-write/decode-read
|
||
at `qwen_image_21_transformer.py:541-598` or the batch helpers at
|
||
`pipeline_qwen_image_21.py:1218-1325`; but note the cache itself is not
|
||
the binding constraint on either card — the 3090 blocks on the resident
|
||
BF16 working set + img-MLP activation, the 5080 on init. A spill path
|
||
alone would not change either verdict, which answers the issue's
|
||
"does it still serve us before committing to anything upstream-sized":
|
||
**no — not as the next step**; the load/init path comes first.
|
||
|
||
## Out-of-scope confirmation
|
||
|
||
No code changes; no upstream PRs; no TP/multi-GPU paths touched; no
|
||
helper-system deployment evidence produced. The only artifacts are this
|
||
report, the probe scripts, and per-host run records.
|