11 changed files with 2498 additions and 0 deletions
@@ -0,0 +1,297 @@
+# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB)
+
+Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and
+`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`,
+vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
+transformers 5.14.1, diffusers git `122b1e11`), served from the
+digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with
+`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds,
+20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed
+Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile
+`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same
+frozen references-suite-2 images, same rendered brief-prose/1 prompts,
+same seeds 2301–2304, same per-fixture resolutions.
+
+## Verdicts
+
+**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the
+conditioning KV exists anywhere in the pipeline (verified by code read and
+by exhausting every reachable knob). At BF16 the served reference ceiling
+is two condition images at `--enable-cpu-offload` — with the conditioning
+cache ON, at 2.2× the Diffusers nocache baseline's speed. Four references
+OOM inside the DiT forward under every BF16 flag combination, and the
+binding constraint is the resident BF16 working set plus a fixed ~490 MiB
+image-MLP activation, not the conditioning KV: the per-request cache
+contributes only roughly 0.25–0.3 GiB per reference (see "KV size" below).
+FP8 weight quantization (`--quantization fp8`) does free the ~6 GiB the
+arithmetic calls for — all of one, two, three, and four references then
+fit and run (4-ref: 44.1–44.4 s, server peak 23,522 MB, nvidia-smi series
+max 23,983 MiB) — **but on this commit and card the FP8 path produces
+structurally broken output** (~96.5–97% transparent RGBA, grey noise RGB,
+PSNR 7–9 dB), for the edit path at every reference count and for T2I
+alike (see "FP8 quality collapse"). So the verdict stands: no usable path
+to four references, and none that spills the KV.
+
+**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.**
+Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8,
+fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB
+allocation with the card full. The failure is in
+`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the
+unquantized parameter skeleton is allocated on-device at construction,
+before any quantization or offload path engages, so no runtime flag can
+shrink the load-time peak. The reference path is unreachable on this card
+with this stack; its one-reference Diffusers ceiling (B2g) relied on a
+torchao-int8 + group-offload load path that never materializes the full
+BF16 model on the GPU. The conditioning-KV question is moot on this card.
+
+## Where the cache lives (issue scope item 2)
+
+- Created per request at
+  `vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207`
+  (`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG
+  branch `cond`/`uncond`), valid because `causal_condition=True` makes
+  text + condition-image K/V timestep-independent.
+- Written at prefill (first denoise step) and read at decode in
+  `qwen_image_21/.../qwen_image_21_transformer.py:538-598`
+  (`branch_cache["key"] = key[:, :cache_write_len].clone()` and the
+  decode-side `torch.cat` prepend).
+- Batched mixed-phase assembly/scatter at
+  `pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and
+  `:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in
+  `qwen_image_21/decode_graph.py` (inactive here — eager).
+- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's
+  paged KVCacheManager — the model deliberately runs `DENSE_LEGACY`
+  (`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of
+  `PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via
+  `extras.prefix_kv_cache_dtype`.
+- **Lifetime**: per-request only; freed by refcount when the runner drops
+  the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`).
+  No cross-request retention, no LRU, no dedup — unlike Diffusers' helper
+  cache, nothing persists between requests.
+- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` /
+  `fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the
+  propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode`
+  (static-buffer copy), `cuda_graph_max_decode_graphs`.
+  `--swap-space` and `--cpu-offload-gb` **do not exist** in this fork;
+  `--gpu-memory-utilization` sizes only the unrelated paged-scheduler
+  path. The weight-offload flags (`--enable-cpu-offload`,
+  `--enable-layerwise-offload`) never move KV tensors.
+- **Spill path**: none. A host-DRAM contribution would go at the prefill
+  write / decode read (`qwen_image_21_transformer.py:541-598` — allocate
+  on pinned CPU, prefetch before the decode `torch.cat`) or at the batch
+  boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the
+  request-owns-independent-storage invariant that the decode-graph weakref
+  identity check at `decode_graph.py:350-363` depends on.
+
+## KV size (empirical)
+
+Between one and two references the per-request peak moves by only
+~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+
+expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image —
+roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the
+helper-system measured Diffusers retaining (multi-model-helper-server
+#160). The cache is per-request and lean; it is not what blocks four
+references.
+
+## Run matrix — RTX 3090
+
+All runs: `vllm serve <snapshot> --omni --port 8091 --enforce-eager
+--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via
+`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under
+`~/Projects/vllm-omni-val/issue2/runs/<label>/` on the host.
+
+| Run | Flags | Init | 1-ref | 2-ref | 4-ref | nvidia-smi peak |
+| --- | --- | --- | --- | --- | --- | --- |
+| default | (none) | **OOM in 14 s** (23.51 GiB in use at load) | — | — | — | 23,939 MiB |
+| cpuoff | `--enable-cpu-offload` | ok | OK 34.2 s | OK 36.7 s | **OOM** in request | 24,015 MiB |
+| cpuoff-layerwise | `+ --enable-layerwise-offload` | ok | **OOM** | **OOM** | **OOM** | 24,137 MiB |
+| cpuoff-fp8v (yaml) | `+ --deploy-config deploy-fp8v.yaml` | ok | OK 34.3 s | OK 36.8 s | **OOM** | 24,015 MiB |
+| cpuoff-expandable | `+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ok | OK 33.8 s | OK 36.7 s | **OOM** | 24,095 MiB |
+| fp8 | `--enable-cpu-offload --quantization fp8` | ok | OK 29.9 s | OK 34.1 s (+3-ref OK 38.9 s) | OK 44.3 s — **output broken** | 23,983 MiB |
+
+The `fp8` row (review follow-up; #1 validated this flag's capacity
+offline) also covered the previously untested three-reference count via
+`edit-compose-three` — derived from `edit-compose-four` minus
+`ref-shop-hesper` and its Subject line, four→three wording, same
+instrument (recorded in `fixtures.json`). Measured under fp8, seeds
+2301–2304: four 44.06–44.35 s (peak 22,918 MB), pair 34.62–35.05 s
+(peak 21,504 MB), three 39.44–39.69 s (peak 23,532 MB) — every request
+succeeds capacity-wise, and Marlin W8A16 is *faster* than BF16
+(4-ref 44.3 s vs Diffusers' 145.8 s = 3.3×). The quality outcome is in
+the next section.
+
+- 4-ref OOM signature (identical ask across configs): `Tried to allocate
+  490.00 MiB` inside the image-branch MLP forward
+  (`qwen_image_21_transformer.py:293`, via `predict_noise` — prefill
+  encoding has already succeeded). At cpuoff: 21.02 GiB allocated + 1.84
+  GiB reserved-unallocated; at cpuoff-expandable: 22.29 GiB allocated,
+  505 MiB reserved-unallocated — fragmentation removed, still short:
+  genuine capacity gap.
+- Layerwise offload is strictly counterproductive here: even one reference
+  OOMs (21.76 GiB allocated during the request).
+- Determinism: seed-2301 outputs are byte-identical (sha256
+  `45fac0b8…` pair, `3c4eb2ef…` single) across cpuoff, cpuoff-fp8v,
+  cpuoff-expandable — the frozen-seed instrument is sound.
+
+### FP8 quality collapse (and a correction to issue 1)
+
+Every FP8-weight output this probe produced is structurally broken, not
+merely noisier:
+
+- Signature (identical everywhere): RGBA output ~96.5–97% transparent
+  (opaque fraction 0.008–0.035, alpha mean ≈115), near-flat grey RGB
+  (mean ≈120–130), PSNR vs the Diffusers baseline 6.6–9.0 dB. The BF16
+  path on the same requests is 100% opaque and 26–29 dB.
+- Scope: the serve path at 1, 2, 3, and 4 references (all seeds measured),
+  the offline path with `fp8` minus `img_mlp`, and the offline path with
+  **transformer-only** FP8 (`quantization_config={"transformer":
+  {"method": "fp8"}}` — the text encoder stays BF16). Transformer-only
+  still collapsing pins the culprit on the Marlin-quantized DiT, not the
+  encoder and not the image-MLP. An encoder-only isolator could not
+  initialize (the encoder's bf16→fp8 conversion transient lands on the GPU
+  on top of the resident BF16 DiT and exceeds the card).
+- **Correction to issue 1**: #1's offline FP8 validation
+  (`qwen_image_21_fp8_all_rtx3090.png`, `…_fp8_imgmlp_…`) shows the same
+  collapse on reinspection (opaque fraction 0.035 both) — that run
+  validated rc=0 and image production, not output structure, so "FP8
+  weight-only works on this card" from #1 overstated it. #1's 18.4 GB
+  capacity measurement stands; its quality implication does not.
+- Upstream relevance: the recipe's H200 FP8 numbers quote PSNR only for
+  text-encoder FP8 (32.1 dB T2I); DiT-FP8 output quality appears
+  unmeasured upstream, and this probe adds the Ampere/Marlin cell:
+  online FP8 DiT weights yield near-noise latents for Qwen-Image-2.1.
+  Until that is fixed (or validated as Ampere-specific), FP8 cannot serve
+  as the capacity unlock on this card — which is why the 3090 verdict
+  remains `not-built` despite the fp8 capacity row passing.
+
+### Offline knob runs (3090, `Omni(...)` engine kwargs)
+
+The one knob that reaches the cache — `extras.prefix_kv_cache_dtype` —
+was exercised offline (`offline_fp8v.py`, `--enforce-eager
+--enable-cpu-offload` + engine kwarg):
+
+| Run | 4-ref seed 2301 | Detail |
+| --- | --- | --- |
+| offline fp8_v | **OOM** | 20.86 GiB allocated + 2.14 GiB reserved-unallocated; same 490 MiB img-MLP ask |
+| offline fp8_v + expandable | **OOM** | 22.22 GiB allocated, 483 MiB reserved-unallocated |
+
+FP8-V halves the V half of a cache that contributes only ~0.3 GiB per
+reference — an order of magnitude too small to bridge the gap. The
+conditioning KV cannot rescue four references on this card by quantization
+or by anything else; only a genuinely leaner resident working set
+(quantized DiT weights) would, and on this commit that is blocked by the
+init path (see 5080).
+
+### Measured run + PSNR (3090)
+
+`edit-compose-pair` (2 refs, 1248×832), seeds 2301–2304, on
+cpuoff-expandable, versus the sealed Diffusers attempt's pair samples
+(same references, 20 steps; Diffusers side ran an int8-quantized DiT with
+KV cache off, wall 81.16 s):
+
+- Wall: **36.6–36.9 s per image** (2.2× faster than Diffusers nocache;
+  server `denoise_step_latency` ≈ 1.83 s × 20 steps).
+- Peak: 21,976–23,332 MB server-reported; 24,095 MiB nvidia-smi series
+  max.
+- PSNR vs Diffusers (RGBA-on-RGBA, #1's comparator): **26.09 / 28.75 /
+  26.51 / 28.92 dB; mean 27.57 dB** (n=4). Cross-implementation agreement
+  at matched seed/inputs, not parity — expected given the different
+  samplers/kernels and the Diffusers side's int8 DiT.
+
+The four-reference PSNR cell the issue nominally asks for is
+**capacity-blocked at BF16** (every served flag set, and offline with
+fp8_v). Under the fp8 serve row it becomes obtainable but meaningless as
+quality: measured seeds 2301–2304 give 7.03–7.21 dB (mean 7.11) against
+the sealed 4r baseline because the outputs are the collapsed ~97%
+transparent renders of the FP8 quality section — recorded in
+`evidence/rtx3090/psnr-fp8-rows.json` as the broken-output datum, not as
+a usable comparison. The pair row above (27.57 dB at BF16) remains the
+highest-reference-count usable comparison this card can produce.
+`edit-compose-four` wall for reference: Diffusers 145.8 s (nocache);
+vLLM 44.3 s under fp8 (fast but broken) — no valid BF16 number exists.
+
+## Run matrix — RTX 5080
+
+Environment built fresh per #1's recipe (userspace CPython 3.13.16 venv —
+the host only ships 3.14 and has no `/usr/bin/python3.13`; clone pinned to
+`8484d85d1`; weights/suite verified, the `q3-4r` attempt copied from the
+3090 so both legs compare against the identical baseline).
+
+| Run | Flags | Init result |
+| --- | --- | --- |
+| default | (none) | OOM in 12 s (15.22 GiB allocated at load) |
+| cpuoff | `--enable-cpu-offload` | **OOM in 9 s** — DiT stays GPU-resident under cpu-offload; 14.99 GiB allocated |
+| cpuoff-layerwise | `+ --enable-layerwise-offload` | **OOM in 10 s** — layerwise streams per-forward, after load |
+| fp8 | `--enable-cpu-offload --quantization fp8` | **OOM in ~10 s** — FP8 path engaged (DeepGEMM init visible) but init still fills the card |
+| fp8-expandable | `+ expandable_segments` | **OOM** |
+| int8 | `--enable-cpu-offload --quantization int8` | **OOM in 8 s** — 15.55 GiB in use |
+
+Common failure: `torch.OutOfMemoryError` at a 96 MiB allocation in
+`vllm_omni/diffusion/models/utils.py:159-168 init_parameters`
+(`empty_like` under a device context) — the unquantized parameter
+skeleton is materialized on-device at pipeline construction, before
+weight loading, quantization, or offload can act. Host RAM (30 GiB + zram)
+is a separate ceiling for full weight offload (~32 GiB of BF16 weights)
+but is not what kills these runs.
+
+## Propagation gap found (3090 leg)
+
+The recipe-documented deploy-YAML route for
+`extras.prefix_kv_cache_dtype` **does not reach the transformer through
+`vllm serve` on this commit**: with and without the YAML, serve outputs
+are byte-identical, and the drop is verified offline at the exact
+serve-time construction (`build_diffusion_config` →
+`stage_configs[0].diffusion_config.extras` carries only
+`auxiliary_text_encoder`/`default_llama_model_id` defaults; deploy stage
+`engine_extras.extras` is parsed correctly by `_stage_engine_overrides`
+but lost in the structured-config bridge). The offline `Omni(extras=…)`
+engine kwarg is accepted (load proceeds; the knob then behaves as
+documented). Shapes tried: flat stage key, `engine_extras.<key>`, and
+`engine_extras.extras.<key>`. This is a one-line-ish fix candidate in the
+config bridge, independent of any KV-spill contribution.
+
+## Reproducibility
+
+Scripts live in this directory (`serve.sh`, `probe.py`, `psnr.py`,
+`offline_fp8v.py`, `deploy-fp8v.yaml`) and were pushed verbatim to
+`~/Projects/vllm-omni-val/issue2/` on each host; `fixtures.json` (rendered
+brief-prose/1 prompts, reference bindings, dimensions) was generated on
+the 3090 from the helper repo (`evaluation/media/image/suite.py` +
+`services/media/image_service/renderer.py`) and copied to the 5080.
+Server lifecycle ran under transient user units
+(`systemd-run --user --unit=issue2-serve`); per-run serve/vram logs and
+request records are under `runs/<label>/` on each host; condensed
+summaries in `evidence/rtx3090/runs-summary.json` and
+`evidence/rtx5080/runs-summary.json`.
+
+## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)
+
+- **#27 (serving-profile candidacy)**: on the 3090 the vLLM-Omni path
+  serves 1–2 references at 33–37 s/image with the conditioning cache on —
+  2.2× the sealed Diffusers nocache baseline — and per-request cache
+  pressure of ~0.3 GiB/reference means concurrent multi-reference sessions
+  do not accumulate Diffusers-style 2.8 GiB/reference retention. Candidacy
+  holds for ≤2-reference profiles on 24 GB; 4-reference profiles remain
+  Diffusers-only: FP8 unlocks the capacity (all four counts fit, 3.3×
+  baseline speed) but its output is broken on this commit, so the unlock
+  order is upstream FP8-output correctness first, then serving.
+- **#164 (topology input)**: the 5080 (16 GB) cannot host this stack at
+  all — init-time parameter skeleton exceeds the card before any flag
+  acts (`diffusion/models/utils.py:159-168`). Any topology plan pairing
+  the 5080 with Qwen-Image-2.1 on vLLM-Omni needs upstream load-path work
+  (quantized or CPU-materialized init) first; the B-series torchao-int8
+  Diffusers stack remains that card's only viable image backend.
+- Upstream contribution seam, if ever pursued: prefill-write/decode-read
+  at `qwen_image_21_transformer.py:541-598` or the batch helpers at
+  `pipeline_qwen_image_21.py:1218-1325`; but note the cache itself is not
+  the binding constraint on either card — the 3090 blocks on the resident
+  BF16 working set + img-MLP activation, the 5080 on init. A spill path
+  alone would not change either verdict, which answers the issue's
+  "does it still serve us before committing to anything upstream-sized":
+  **no — not as the next step**; the load/init path comes first.
+
+## Out-of-scope confirmation
+
+No code changes; no upstream PRs; no TP/multi-GPU paths touched; no
+helper-system deployment evidence produced. The only artifacts are this
+report, the probe scripts, and per-host run records.
@@ -0,0 +1,7 @@
+# Issue 2 probe: Qwen-Image-2.1 with FP8 V-only prefix-KV storage.
+# Merged over the pipeline's declared single-stage topology; every other
+# field falls back to StageDeployConfig defaults.
+stages:
+  - stage_id: 0
+    engine_extras:
+      prefix_kv_cache_dtype: "fp8_v"
@@ -0,0 +1,30 @@
+{
+ "cpuoff-expandable": {
+  "rows": [
+   {
+    "seed": 2301,
+    "psnr_db": 26.09,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2302,
+    "psnr_db": 28.75,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2303,
+    "psnr_db": 26.51,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2304,
+    "psnr_db": 28.92,
+    "modes": "RGBA/RGBA"
+   }
+  ],
+  "mean_db": 27.57,
+  "min_db": 26.09,
+  "max_db": 28.92,
+  "n": 4
+ }
+}
@@ -0,0 +1,112 @@
+{
+ "edit-compose-four": {
+  "cpuoff-fp8": {
+   "rows": [
+    {
+     "seed": 2301,
+     "psnr_db": 7.03,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2302,
+     "psnr_db": 7.11,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2303,
+     "psnr_db": 7.1,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2304,
+     "psnr_db": 7.21,
+     "modes": "RGBA/RGBA"
+    }
+   ],
+   "mean_db": 7.11,
+   "min_db": 7.03,
+   "max_db": 7.21,
+   "n": 4
+  },
+  "offline-fp8-imgmlp": {
+   "rows": [
+    {
+     "seed": 2301,
+     "psnr_db": 7.27,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2302,
+     "error": "missing"
+    },
+    {
+     "seed": 2303,
+     "error": "missing"
+    },
+    {
+     "seed": 2304,
+     "error": "missing"
+    }
+   ],
+   "mean_db": 7.27,
+   "min_db": 7.27,
+   "max_db": 7.27,
+   "n": 1
+  },
+  "offline-fp8-dit": {
+   "rows": [
+    {
+     "seed": 2301,
+     "psnr_db": 6.69,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2302,
+     "error": "missing"
+    },
+    {
+     "seed": 2303,
+     "error": "missing"
+    },
+    {
+     "seed": 2304,
+     "error": "missing"
+    }
+   ],
+   "mean_db": 6.69,
+   "min_db": 6.69,
+   "max_db": 6.69,
+   "n": 1
+  }
+ },
+ "edit-compose-pair": {
+  "cpuoff-fp8": {
+   "rows": [
+    {
+     "seed": 2301,
+     "psnr_db": 6.57,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2302,
+     "psnr_db": 7.88,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2303,
+     "psnr_db": 9.02,
+     "modes": "RGBA/RGBA"
+    },
+    {
+     "seed": 2304,
+     "psnr_db": 8.95,
+     "modes": "RGBA/RGBA"
+    }
+   ],
+   "mean_db": 8.11,
+   "min_db": 6.57,
+   "max_db": 9.02,
+   "n": 4
+  }
+ }
+}
@@ -0,0 +1,40 @@
+{
+ "runs": {
+  "cpuoff": {
+   "flags": "--enable-cpu-offload",
+   "serve_rc": 1,
+   "serve_wall_s": 9,
+   "vram_peak_mib": 15856
+  },
+  "cpuoff-layerwise": {
+   "flags": "--enable-cpu-offload --enable-layerwise-offload",
+   "serve_rc": 1,
+   "serve_wall_s": 10,
+   "vram_peak_mib": 15856
+  },
+  "default": {
+   "flags": "flags:",
+   "serve_rc": 1,
+   "serve_wall_s": 12,
+   "vram_peak_mib": 15926
+  },
+  "fp8": {
+   "flags": "--enable-cpu-offload --quantization fp8",
+   "serve_rc": 1,
+   "serve_wall_s": 13,
+   "vram_peak_mib": 15930
+  },
+  "fp8-expandable": {
+   "flags": "--enable-cpu-offload --quantization fp8",
+   "serve_rc": 1,
+   "serve_wall_s": 9,
+   "vram_peak_mib": 15876
+  },
+  "int8": {
+   "flags": "--enable-cpu-offload --quantization int8",
+   "serve_rc": 1,
+   "serve_wall_s": 8,
+   "vram_peak_mib": 15930
+  }
+ }
+}
@@ -0,0 +1,49 @@
+{
+  "edit-expression-ysolde": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde"
+    ],
+    "width": 864,
+    "height": 1152,
+    "prompt": "Change only her expression and the chart: she now laughs openly and lowers the chart; keep her face, hair, clothing, the room, the framing, and the lighting as in the reference. Result: Ysolde laughs at something off-frame in the map room. Emotional tone: sudden delight.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Laughing openly, the chart lowered to her chest.\nSetting: The map room of a harbor guildhall, charts pinned to timber walls, rain on the window.\nComposition: close-up; head and shoulders, three-quarter view, face in sharp focus; lighting: cool grey window light from the left with a warm lamp glow from below\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
+    "negative_prompt": null
+  },
+  "edit-compose-pair": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde",
+      "ref-cutout-tamsin"
+    ],
+    "width": 1248,
+    "height": 832,
+    "prompt": "Place the woman from the first reference and the figure from the second reference together; keep both identities exactly. Result: Ysolde and Tamsin share one unrolled map at a harbor wall. Emotional tone: wary cooperation.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Holding one end of an unrolled map, explaining a route.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Holding the other end of the map, listening sceptically.\nSetting: A sea wall above the harbor at midday, fishing boats below.\nComposition: medium shot; both figures side by side, the map between them; lighting: bright midday sun with short shadows\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
+    "negative_prompt": null
+  },
+  "edit-compose-four": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde",
+      "ref-storm-corvin",
+      "ref-cutout-tamsin",
+      "ref-shop-hesper"
+    ],
+    "width": 1344,
+    "height": 768,
+    "prompt": "Bring the four characters from the references together in one scene; keep every identity exactly. Result: All four gather around the apothecary counter. Emotional tone: conspiratorial calm.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Unfolding a chart on the counter.\nSubject: Corvin Ashdown, an adult man in his mid-fifties, with pale, heavily freckled skin, short receding red hair and a grey-streaked red beard, and pale blue eyes; wearing a dark green harbormaster's greatcoat with a silver chain of office; a black leather patch over the left eye. Leaning on the doorframe.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Sniffing a stoppered bottle.\nSubject: Hesper Quill, an adult woman in her early seventies, with light brown, deeply weathered skin, long white hair in one thick plait, and grey-green eyes; wearing a lavender dress under a stained canvas apothecary's apron; round wire-rimmed spectacles and a silver ring on every finger of the right hand. Behind the counter, weighing herbs on a brass scale.\nSetting: Inside the apothecary: shelves of bottled remedies, drying herbs hanging from the beams, a long counter.\nComposition: wide shot; from behind the counter's end, all four faces visible; lighting: warm lamplight with green light filtering through bottle glass\nStyle: Anime illustration: clean line art, cel shading, flat vivid colour, expressive faces.",
+    "negative_prompt": null
+  },
+  "edit-compose-three": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde",
+      "ref-storm-corvin",
+      "ref-cutout-tamsin"
+    ],
+    "width": 1344,
+    "height": 768,
+    "prompt": "Bring the three characters from the references together in one scene; keep every identity exactly. Result: All three gather around the apothecary counter. Emotional tone: conspiratorial calm.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Unfolding a chart on the counter.\nSubject: Corvin Ashdown, an adult man in his mid-fifties, with pale, heavily freckled skin, short receding red hair and a grey-streaked red beard, and pale blue eyes; wearing a dark green harbormaster's greatcoat with a silver chain of office; a black leather patch over the left eye. Leaning on the doorframe.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Sniffing a stoppered bottle.\nSetting: Inside the apothecary: shelves of bottled remedies, drying herbs hanging from the beams, a long counter.\nComposition: wide shot; from behind the counter's end, all four faces visible; lighting: warm lamplight with green light filtering through bottle glass\nStyle: Anime illustration: clean line art, cel shading, flat vivid colour, expressive faces.",
+    "negative_prompt": null,
+    "derived_from": "edit-compose-four minus ref-shop-hesper and its Subject line; four->three wording"
+  }
+}
@@ -0,0 +1,146 @@
+#!/usr/bin/env python
+"""Issue 2 probe: offline Qwen-Image-2.1 edit driver with prefix-KV extras.
+
+The served path cannot reach `extras.prefix_kv_cache_dtype` on commit
+8484d85d (the structured-config bridge drops deploy-YAML extras), so this
+driver exercises the knob through the offline `Omni(...)` engine kwargs —
+the other route the recipe names. It mirrors image_edit.py's request
+construction: same prompt builder, same sampling fields, one request at a
+time.
+
+Usage: offline_fp8v.py --out RUN_DIR [--kv-dtype fp8_v] [--fixture FID]
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import pathlib
+import time
+
+import torch
+from PIL import Image
+
+from vllm_omni.entrypoints.omni import Omni
+from vllm_omni.inputs.data import OmniDiffusionSamplingParams
+from vllm_omni.model_extras import (
+    build_image_to_image_prompt,
+    get_model_class_name,
+)
+from vllm_omni.platforms import current_omni_platform
+
+W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
+SUITE = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
+)
+SNAP = (
+    pathlib.Path.home()
+    / ".local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758"
+)
+STEPS = 20
+CFG = 1.0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", required=True, type=pathlib.Path)
+    parser.add_argument("--kv-dtype", default="fp8_v")
+    parser.add_argument(
+        "--quant",
+        default="none",
+        choices=["none", "fp8", "fp8-imgmlp", "fp8-dit", "fp8-encoder"],
+        help="weight quantization: none, fp8 (all), fp8 minus img_mlp, transformer-only, or text-encoder-only",
+    )
+    parser.add_argument("--fixture", default="edit-compose-four")
+    parser.add_argument("--seeds", type=lambda s: tuple(int(x) for x in s.split(",")), default=(2301,))
+    parser.add_argument("--no-extras", action="store_true", help="run without the extras knob (control)")
+    args = parser.parse_args()
+
+    fixture = json.loads((W / "fixtures.json").read_text())[args.fixture]
+    refs = [Image.open(SUITE / f"{r}.png").convert("RGB") for r in fixture["references"]]
+
+    omni_kwargs = dict(
+        model=str(SNAP),
+        enforce_eager=True,
+        enable_cpu_offload=True,
+        init_timeout=1200,
+    )
+    if args.quant == "fp8":
+        omni_kwargs["quantization"] = "fp8"
+    elif args.quant == "fp8-imgmlp":
+        omni_kwargs["quantization_config"] = {
+            "method": "fp8",
+            "ignored_layers": ["img_mlp"],
+        }
+    elif args.quant == "fp8-dit":
+        omni_kwargs["quantization_config"] = {"transformer": {"method": "fp8"}}
+    elif args.quant == "fp8-encoder":
+        omni_kwargs["quantization_config"] = {"text_encoder": {"method": "fp8"}}
+    if not args.no_extras:
+        omni_kwargs["extras"] = {"prefix_kv_cache_dtype": args.kv_dtype}
+    t0 = time.time()
+    omni = Omni(**omni_kwargs)
+    print(
+        f"load_s={time.time() - t0:.1f} quant={args.quant} extras={omni_kwargs.get('extras')}",
+        flush=True,
+    )
+    model_class_name = get_model_class_name(omni)
+
+    args.out.mkdir(parents=True, exist_ok=True)
+    records = []
+    for seed in args.seeds:
+        generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(seed)
+        prompt_dict = build_image_to_image_prompt(
+            model_class_name=model_class_name,
+            prompt=fixture["prompt"],
+            negative_prompt=None,
+            input_image=refs[0] if len(refs) == 1 else refs,
+            height=fixture["height"],
+            width=fixture["width"],
+        )
+        params = OmniDiffusionSamplingParams(
+            generator=generator,
+            seed=seed,
+            true_cfg_scale=CFG,
+            num_inference_steps=STEPS,
+            height=fixture["height"],
+            width=fixture["width"],
+        )
+        torch.cuda.reset_peak_memory_stats()
+        t1 = time.time()
+        rec = {"fixture_id": args.fixture, "seed": seed, "n_references": len(refs), "kv_dtype": args.kv_dtype, "quant": args.quant}
+        try:
+            outputs = omni.generate(prompt_dict, sampling_params_list=[params])
+            wall = time.time() - t1
+            images = None
+            for output in outputs:
+                images = getattr(output, "images", None)
+                if images:
+                    break
+            if not images:
+                raise RuntimeError("no images in output")
+            out_path = args.out / "measured" / f"{args.fixture}.s{seed}.png"
+            out_path.parent.mkdir(parents=True, exist_ok=True)
+            images[0].save(out_path)
+            png = out_path.read_bytes()
+            rec.update(
+                wall_s=round(wall, 2),
+                peak_gpu_mb=round(torch.cuda.max_memory_allocated() / 2**20, 1),
+                bytes=len(png),
+                sha256=hashlib.sha256(png).hexdigest(),
+            )
+        except Exception as exc:  # noqa: BLE001 - recorded as the probe result
+            rec.update(wall_s=round(time.time() - t1, 2), error=f"{type(exc).__name__}: {exc}"[:2000])
+        records.append(rec)
+        print(json.dumps(rec), flush=True)
+
+    (args.out / f"offline-{args.fixture}.json").write_text(json.dumps(records, indent=1) + "\n")
+    ok = all("sha256" in r for r in records)
+    print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,164 @@
+#!/usr/bin/env python
+"""Issue 2 probe client: reference-conditioned requests through the API.
+
+Functional pass: one-, two-, and four-reference fixtures at seed 2301.
+Measured pass: the four-reference fixture at seeds 2301-2304.
+
+Every request mirrors the sealed Diffusers baseline attempt
+2026-10-07T170729Z-q3-4r (same frozen references, same rendered prompt,
+20 steps, true_cfg_scale 1.0, fixture resolution, no negative prompt),
+submitted one at a time through POST /v1/images/edits.
+
+Usage: probe.py --out RUN_DIR [--functional] [--measured] [--host H:P]
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import hashlib
+import json
+import pathlib
+import time
+
+import requests
+
+W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
+SUITE = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
+)
+STEPS = 20
+CFG = 1.0
+FUNCTIONAL_FIXTURES = (
+    "edit-expression-ysolde",  # 1 reference
+    "edit-compose-pair",        # 2 references
+    "edit-compose-three",       # 3 references (derived; see fixtures.json)
+    "edit-compose-four",        # 4 references
+)
+MEASURED_FIXTURE = "edit-compose-four"
+MEASURED_SEEDS = (2301, 2302, 2303, 2304)
+
+
+def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
+    files = []
+    handles = []
+    for ref in fixture["references"]:
+        path = SUITE / f"{ref}.png"
+        handle = open(path, "rb")
+        handles.append(handle)
+        files.append(("image[]", (path.name, handle, "image/png")))
+    data = {
+        "model": "Qwen/Qwen-Image-2.1",
+        "prompt": fixture["prompt"],
+        "seed": str(seed),
+        "num_inference_steps": str(STEPS),
+        "true_cfg_scale": str(CFG),
+        "size": f"{fixture['width']}x{fixture['height']}",
+        "n": "1",
+        "response_format": "b64_json",
+        "output_format": "png",
+        "return_stage_metrics": "true",
+    }
+    rec: dict = {
+        "fixture_id": None,
+        "seed": seed,
+        "n_references": len(fixture["references"]),
+        "steps": STEPS,
+        "true_cfg_scale": CFG,
+        "size": f"{fixture['width']}x{fixture['height']}",
+    }
+    t0 = time.time()
+    try:
+        response = requests.post(
+            f"http://{host}/v1/images/edits", data=data, files=files, timeout=1200
+        )
+        rec["wall_s"] = round(time.time() - t0, 2)
+        rec["http"] = response.status_code
+        if response.status_code == 200:
+            body = response.json()
+            png = base64.b64decode(body["data"][0]["b64_json"])
+            rec["bytes"] = len(png)
+            rec["sha256"] = hashlib.sha256(png).hexdigest()
+            rec["usage"] = body.get("usage")
+            for key in ("stage_metrics", "metrics", "timings"):
+                if body.get(key):
+                    rec[key] = body[key]
+            return rec, png
+        rec["error"] = response.text[:2000]
+        return rec, None
+    except Exception as exc:  # noqa: BLE001 - recorded as the probe result
+        rec["wall_s"] = round(time.time() - t0, 2)
+        rec["error"] = f"{type(exc).__name__}: {exc}"
+        return rec, None
+    finally:
+        for handle in handles:
+            handle.close()
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", required=True, type=pathlib.Path)
+    parser.add_argument("--host", default="127.0.0.1:8091")
+    parser.add_argument("--functional", action="store_true")
+    parser.add_argument("--measured", action="store_true")
+    parser.add_argument(
+        "--fixture",
+        default=MEASURED_FIXTURE,
+        help="fixture id for --measured (default: edit-compose-four)",
+    )
+    parser.add_argument(
+        "--seeds",
+        type=lambda s: tuple(int(x) for x in s.split(",")),
+        default=MEASURED_SEEDS,
+        help="comma-separated seeds for --measured",
+    )
+    args = parser.parse_args()
+
+    fixtures = json.loads((W / "fixtures.json").read_text())
+    args.out.mkdir(parents=True, exist_ok=True)
+
+    plan: list[tuple[str, pathlib.Path, int]] = []
+    if args.functional:
+        for fixture_id in FUNCTIONAL_FIXTURES:
+            plan.append(
+                (fixture_id, args.out / "functional" / f"{fixture_id}.png", 2301)
+            )
+    if args.measured:
+        for seed in args.seeds:
+            plan.append(
+                (
+                    args.fixture,
+                    args.out / "measured" / f"{args.fixture}.s{seed}.png",
+                    seed,
+                )
+            )
+
+    records = []
+    for fixture_id, out_path, seed in plan:
+        rec, png = submit(args.host, fixtures[fixture_id], seed)
+        rec["fixture_id"] = fixture_id
+        records.append(rec)
+        print(json.dumps(rec), flush=True)
+        if png is not None:
+            out_path.parent.mkdir(parents=True, exist_ok=True)
+            out_path.write_bytes(png)
+        elif rec.get("http") == 200:
+            print(f"warning: 200 but no image for {fixture_id} seed {seed}", flush=True)
+
+    suffix_parts = []
+    if args.functional:
+        suffix_parts.append("functional")
+    if args.measured:
+        suffix_parts.append(f"measured-{args.fixture}")
+    suffix = "-".join(suffix_parts)
+    (args.out / f"{suffix}.json").write_text(
+        json.dumps(records, indent=1) + "\n"
+    )
+    ok = all(r.get("http") == 200 and "sha256" in r for r in records)
+    print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+"""Issue 2 probe: PSNR of vLLM outputs against the sealed Diffusers
+four-reference baseline attempt 2026-10-07T170729Z-q3-4r.
+
+Usage: psnr.py --fixture FIXTURE_ID RUN_DIR [RUN_DIR ...]
+
+Compares each RUN_DIR/measured/FIXTURE_ID.sSEED.png against the baseline
+sample with the same fixture and seed. RGBA outputs are composited over
+white, matching issue 1's comparator.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import pathlib
+
+import numpy as np
+from PIL import Image
+
+ATTEMPT = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/attempts/2026-10-07T170729Z-q3-4r"
+)
+
+
+def to_rgb(array: np.ndarray) -> np.ndarray:
+    if array.shape[2] == 4:
+        alpha = array[..., 3:4].astype(np.float64) / 255.0
+        bg = np.full_like(array[..., :3], 255, dtype=np.float64)
+        return np.round(
+            array[..., :3].astype(np.float64) * alpha + bg * (1 - alpha)
+        ).clip(0, 255).astype(np.uint8)
+    return array[..., :3]
+
+
+def psnr(a: np.ndarray, b: np.ndarray) -> float:
+    mse = float(np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2))
+    return float("inf") if mse == 0 else 10 * math.log10(255.0**2 / mse)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--fixture", default="edit-compose-four")
+    parser.add_argument("runs", nargs="+", type=pathlib.Path)
+    args = parser.parse_args()
+    fixture = args.fixture
+
+    samples = json.loads((ATTEMPT / "samples.json").read_text())["samples"]
+    baseline = {
+        s["seed"]: ATTEMPT / s["file"]
+        for s in samples
+        if s["fixture_id"] == fixture and s["status"] == "succeeded"
+    }
+    summary = {}
+    for run_dir in args.runs:
+        rows = []
+        for seed, base_path in sorted(baseline.items()):
+            ours = run_dir / "measured" / f"{fixture}.s{seed}.png"
+            if not ours.exists():
+                rows.append({"seed": seed, "error": "missing"})
+                continue
+            bm, om = Image.open(base_path), Image.open(ours)
+            ba, oa = np.asarray(bm), np.asarray(om)
+            if ba.shape[:2] != oa.shape[:2]:
+                rows.append(
+                    {"seed": seed, "error": f"size {oa.shape[:2]} vs {ba.shape[:2]}"}
+                )
+                continue
+            use_rgba = bm.mode == "RGBA" and om.mode == "RGBA"
+            a, b = (ba, oa) if use_rgba else (to_rgb(ba), to_rgb(oa))
+            rows.append(
+                {
+                    "seed": seed,
+                    "psnr_db": round(psnr(a, b), 2),
+                    "modes": f"{bm.mode}/{om.mode}",
+                }
+            )
+        vals = [r["psnr_db"] for r in rows if "psnr_db" in r]
+        if vals:
+            summary[run_dir.name] = {
+                "rows": rows,
+                "mean_db": round(sum(vals) / len(vals), 2),
+                "min_db": min(vals),
+                "max_db": max(vals),
+                "n": len(vals),
+            }
+        else:
+            summary[run_dir.name] = {"rows": rows}
+    print(json.dumps(summary, indent=1))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Issue 2 probe: serve Qwen-Image-2.1 on the RTX 3090 with one labeled
+# flag set. VRAM is polled once per second for the whole server lifetime,
+# matching issue 1's instrument (nvidia-smi csv series).
+#
+# Usage: serve.sh LABEL [extra vllm flags...]
+set -uo pipefail
+
+BASE=$HOME/Projects/vllm-omni-val
+W=$BASE/issue2
+LABEL=$1; shift
+RUN=$W/runs/$LABEL
+mkdir -p "$RUN"
+
+SNAP=$HOME/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758
+export HF_HOME=$HOME/.local/state/multi-model-helper-server/state/image-service/weights
+export HF_HUB_OFFLINE=1
+
+( while sleep 1; do nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader; done ) > "$RUN/vram.log" 2>&1 &
+POLL=$!
+T0=$(date +%s)
+echo "flags: $*" > "$RUN/flags.txt"
+
+$BASE/venv/bin/vllm serve "$SNAP" --omni --port 8091 \
+  --served-model-name Qwen/Qwen-Image-2.1 \
+  --enforce-eager "$@" > "$RUN/serve.log" 2>&1
+RC=$?
+kill $POLL 2>/dev/null
+echo "serve_rc=$RC wall=$(( $(date +%s) - T0 ))s" >> "$RUN/serve.log"