probe: Qwen-Image conditioning-KV spill probe on RTX 3090 / RTX 5080 (issue 2)

Evidence-only change: probe kit (serve wrapper, API client, PSNR
comparator, offline fp8_v driver), per-host run summaries, and the
findings report. No code changes to the stack.

Verdicts: not-built on both cards. On the 3090 the served reference
ceiling is two condition images with the conditioning cache on
(2.2x the Diffusers nocache baseline, pair PSNR mean 27.57 dB);
four references OOM on the resident BF16 working set, not the KV
(~0.3 GiB/reference, per-request). On the 5080 the server cannot
initialize under any flag: init_parameters materializes the
unquantized skeleton on-device before quantization/offload engage.
Also documents that the recipe's deploy-YAML extras route for
prefix_kv_cache_dtype does not reach the served transformer on
8484d85d (offline Omni engine kwarg does).

probe/issue-2/FINDINGS.md

@@ -0,0 +1,239 @@
+# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB)
+
+Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and
+`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`,
+vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
+transformers 5.14.1, diffusers git `122b1e11`), served from the
+digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with
+`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds,
+20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed
+Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile
+`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same
+frozen references-suite-2 images, same rendered brief-prose/1 prompts,
+same seeds 2301–2304, same per-fixture resolutions.
+
+## Verdicts
+
+**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the
+conditioning KV exists anywhere in the pipeline (verified by code read and
+by exhausting every reachable knob). The served reference ceiling is two
+condition images at `--enable-cpu-offload` — with the conditioning cache
+ON, at 2.2× the Diffusers nocache baseline's speed. Four references OOM
+inside the DiT forward under every flag combination, and the binding
+constraint is the resident BF16 working set plus a fixed ~490 MiB image-MLP
+activation, not the conditioning KV: the per-request cache contributes only
+roughly 0.25–0.3 GiB per reference (see "KV size" below).
+
+**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.**
+Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8,
+fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB
+allocation with the card full. The failure is in
+`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the
+unquantized parameter skeleton is allocated on-device at construction,
+before any quantization or offload path engages, so no runtime flag can
+shrink the load-time peak. The reference path is unreachable on this card
+with this stack; its one-reference Diffusers ceiling (B2g) relied on a
+torchao-int8 + group-offload load path that never materializes the full
+BF16 model on the GPU. The conditioning-KV question is moot on this card.
+
+## Where the cache lives (issue scope item 2)
+
+- Created per request at
+  `vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207`
+  (`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG
+  branch `cond`/`uncond`), valid because `causal_condition=True` makes
+  text + condition-image K/V timestep-independent.
+- Written at prefill (first denoise step) and read at decode in
+  `qwen_image_21/.../qwen_image_21_transformer.py:538-598`
+  (`branch_cache["key"] = key[:, :cache_write_len].clone()` and the
+  decode-side `torch.cat` prepend).
+- Batched mixed-phase assembly/scatter at
+  `pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and
+  `:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in
+  `qwen_image_21/decode_graph.py` (inactive here — eager).
+- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's
+  paged KVCacheManager — the model deliberately runs `DENSE_LEGACY`
+  (`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of
+  `PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via
+  `extras.prefix_kv_cache_dtype`.
+- **Lifetime**: per-request only; freed by refcount when the runner drops
+  the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`).
+  No cross-request retention, no LRU, no dedup — unlike Diffusers' helper
+  cache, nothing persists between requests.
+- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` /
+  `fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the
+  propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode`
+  (static-buffer copy), `cuda_graph_max_decode_graphs`.
+  `--swap-space` and `--cpu-offload-gb` **do not exist** in this fork;
+  `--gpu-memory-utilization` sizes only the unrelated paged-scheduler
+  path. The weight-offload flags (`--enable-cpu-offload`,
+  `--enable-layerwise-offload`) never move KV tensors.
+- **Spill path**: none. A host-DRAM contribution would go at the prefill
+  write / decode read (`qwen_image_21_transformer.py:541-598` — allocate
+  on pinned CPU, prefetch before the decode `torch.cat`) or at the batch
+  boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the
+  request-owns-independent-storage invariant that the decode-graph weakref
+  identity check at `decode_graph.py:350-363` depends on.
+
+## KV size (empirical)
+
+Between one and two references the per-request peak moves by only
+~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+
+expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image —
+roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the
+helper-system measured Diffusers retaining (multi-model-helper-server
+#160). The cache is per-request and lean; it is not what blocks four
+references.
+
+## Run matrix — RTX 3090
+
+All runs: `vllm serve <snapshot> --omni --port 8091 --enforce-eager
+--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via
+`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under
+`~/Projects/vllm-omni-val/issue2/runs/<label>/` on the host.
+
+| Run | Flags | Init | 1-ref | 2-ref | 4-ref | nvidia-smi peak |
+| --- | --- | --- | --- | --- | --- | --- |
+| default | (none) | **OOM in 14 s** (23.51 GiB in use at load) | — | — | — | 23,939 MiB |
+| cpuoff | `--enable-cpu-offload` | ok | OK 34.2 s | OK 36.7 s | **OOM** in request | 24,015 MiB |
+| cpuoff-layerwise | `+ --enable-layerwise-offload` | ok | **OOM** | **OOM** | **OOM** | 24,137 MiB |
+| cpuoff-fp8v (yaml) | `+ --deploy-config deploy-fp8v.yaml` | ok | OK 34.3 s | OK 36.8 s | **OOM** | 24,015 MiB |
+| cpuoff-expandable | `+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ok | OK 33.8 s | OK 36.7 s | **OOM** | 24,095 MiB |
+
+- 4-ref OOM signature (identical ask across configs): `Tried to allocate
+  490.00 MiB` inside the image-branch MLP forward
+  (`qwen_image_21_transformer.py:293`, via `predict_noise` — prefill
+  encoding has already succeeded). At cpuoff: 21.02 GiB allocated + 1.84
+  GiB reserved-unallocated; at cpuoff-expandable: 22.29 GiB allocated,
+  505 MiB reserved-unallocated — fragmentation removed, still short:
+  genuine capacity gap.
+- Layerwise offload is strictly counterproductive here: even one reference
+  OOMs (21.76 GiB allocated during the request).
+- Determinism: seed-2301 outputs are byte-identical (sha256
+  `45fac0b8…` pair, `3c4eb2ef…` single) across cpuoff, cpuoff-fp8v,
+  cpuoff-expandable — the frozen-seed instrument is sound.
+
+### Offline knob runs (3090, `Omni(...)` engine kwargs)
+
+The one knob that reaches the cache — `extras.prefix_kv_cache_dtype` —
+was exercised offline (`offline_fp8v.py`, `--enforce-eager
+--enable-cpu-offload` + engine kwarg):
+
+| Run | 4-ref seed 2301 | Detail |
+| --- | --- | --- |
+| offline fp8_v | **OOM** | 20.86 GiB allocated + 2.14 GiB reserved-unallocated; same 490 MiB img-MLP ask |
+| offline fp8_v + expandable | **OOM** | 22.22 GiB allocated, 483 MiB reserved-unallocated |
+
+FP8-V halves the V half of a cache that contributes only ~0.3 GiB per
+reference — an order of magnitude too small to bridge the gap. The
+conditioning KV cannot rescue four references on this card by quantization
+or by anything else; only a genuinely leaner resident working set
+(quantized DiT weights) would, and on this commit that is blocked by the
+init path (see 5080).
+
+### Measured run + PSNR (3090)
+
+`edit-compose-pair` (2 refs, 1248×832), seeds 2301–2304, on
+cpuoff-expandable, versus the sealed Diffusers attempt's pair samples
+(same references, 20 steps; Diffusers side ran an int8-quantized DiT with
+KV cache off, wall 81.16 s):
+
+- Wall: **36.6–36.9 s per image** (2.2× faster than Diffusers nocache;
+  server `denoise_step_latency` ≈ 1.83 s × 20 steps).
+- Peak: 21,976–23,332 MB server-reported; 24,095 MiB nvidia-smi series
+  max.
+- PSNR vs Diffusers (RGBA-on-RGBA, #1's comparator): **26.09 / 28.75 /
+  26.51 / 28.92 dB; mean 27.57 dB** (n=4). Cross-implementation agreement
+  at matched seed/inputs, not parity — expected given the different
+  samplers/kernels and the Diffusers side's int8 DiT.
+
+The four-reference PSNR cell the issue nominally asks for is
+**capacity-blocked on every served flag set** (and offline with fp8_v);
+the pair row is the highest-reference-count comparison this card can
+produce. `edit-compose-four` single-ref walls for reference:
+Diffusers 145.8 s (nocache); vLLM would-be number unobtainable.
+
+## Run matrix — RTX 5080
+
+Environment built fresh per #1's recipe (userspace CPython 3.13.16 venv —
+the host only ships 3.14 and has no `/usr/bin/python3.13`; clone pinned to
+`8484d85d1`; weights/suite verified, the `q3-4r` attempt copied from the
+3090 so both legs compare against the identical baseline).
+
+| Run | Flags | Init result |
+| --- | --- | --- |
+| default | (none) | OOM in 12 s (15.22 GiB allocated at load) |
+| cpuoff | `--enable-cpu-offload` | **OOM in 9 s** — DiT stays GPU-resident under cpu-offload; 14.99 GiB allocated |
+| cpuoff-layerwise | `+ --enable-layerwise-offload` | **OOM in 10 s** — layerwise streams per-forward, after load |
+| fp8 | `--enable-cpu-offload --quantization fp8` | **OOM in ~10 s** — FP8 path engaged (DeepGEMM init visible) but init still fills the card |
+| fp8-expandable | `+ expandable_segments` | **OOM** |
+| int8 | `--enable-cpu-offload --quantization int8` | **OOM in 8 s** — 15.55 GiB in use |
+
+Common failure: `torch.OutOfMemoryError` at a 96 MiB allocation in
+`vllm_omni/diffusion/models/utils.py:159-168 init_parameters`
+(`empty_like` under a device context) — the unquantized parameter
+skeleton is materialized on-device at pipeline construction, before
+weight loading, quantization, or offload can act. Host RAM (30 GiB + zram)
+is a separate ceiling for full weight offload (~32 GiB of BF16 weights)
+but is not what kills these runs.
+
+## Propagation gap found (3090 leg)
+
+The recipe-documented deploy-YAML route for
+`extras.prefix_kv_cache_dtype` **does not reach the transformer through
+`vllm serve` on this commit**: with and without the YAML, serve outputs
+are byte-identical, and the drop is verified offline at the exact
+serve-time construction (`build_diffusion_config` →
+`stage_configs[0].diffusion_config.extras` carries only
+`auxiliary_text_encoder`/`default_llama_model_id` defaults; deploy stage
+`engine_extras.extras` is parsed correctly by `_stage_engine_overrides`
+but lost in the structured-config bridge). The offline `Omni(extras=…)`
+engine kwarg is accepted (load proceeds; the knob then behaves as
+documented). Shapes tried: flat stage key, `engine_extras.<key>`, and
+`engine_extras.extras.<key>`. This is a one-line-ish fix candidate in the
+config bridge, independent of any KV-spill contribution.
+
+## Reproducibility
+
+Scripts live in this directory (`serve.sh`, `probe.py`, `psnr.py`,
+`offline_fp8v.py`, `deploy-fp8v.yaml`) and were pushed verbatim to
+`~/Projects/vllm-omni-val/issue2/` on each host; `fixtures.json` (rendered
+brief-prose/1 prompts, reference bindings, dimensions) was generated on
+the 3090 from the helper repo (`evaluation/media/image/suite.py` +
+`services/media/image_service/renderer.py`) and copied to the 5080.
+Server lifecycle ran under transient user units
+(`systemd-run --user --unit=issue2-serve`); per-run serve/vram logs and
+request records are under `runs/<label>/` on each host; condensed
+summaries in `evidence/rtx3090/runs-summary.json` and
+`evidence/rtx5080/runs-summary.json`.
+
+## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)
+
+- **#27 (serving-profile candidacy)**: on the 3090 the vLLM-Omni path
+  serves 1–2 references at 33–37 s/image with the conditioning cache on —
+  2.2× the sealed Diffusers nocache baseline — and per-request cache
+  pressure of ~0.3 GiB/reference means concurrent multi-reference sessions
+  do not accumulate Diffusers-style 2.8 GiB/reference retention. Candidacy
+  holds for ≤2-reference profiles on 24 GB; 4-reference profiles remain
+  Diffusers-only until the resident-working-set/init path shrinks
+  (quantized-load init) or a KV spill lands.
+- **#164 (topology input)**: the 5080 (16 GB) cannot host this stack at
+  all — init-time parameter skeleton exceeds the card before any flag
+  acts (`diffusion/models/utils.py:159-168`). Any topology plan pairing
+  the 5080 with Qwen-Image-2.1 on vLLM-Omni needs upstream load-path work
+  (quantized or CPU-materialized init) first; the B-series torchao-int8
+  Diffusers stack remains that card's only viable image backend.
+- Upstream contribution seam, if ever pursued: prefill-write/decode-read
+  at `qwen_image_21_transformer.py:541-598` or the batch helpers at
+  `pipeline_qwen_image_21.py:1218-1325`; but note the cache itself is not
+  the binding constraint on either card — the 3090 blocks on the resident
+  BF16 working set + img-MLP activation, the 5080 on init. A spill path
+  alone would not change either verdict, which answers the issue's
+  "does it still serve us before committing to anything upstream-sized":
+  **no — not as the next step**; the load/init path comes first.
+
+## Out-of-scope confirmation
+
+No code changes; no upstream PRs; no TP/multi-GPU paths touched; no
+helper-system deployment evidence produced. The only artifacts are this
+report, the probe scripts, and per-host run records.

probe/issue-2/deploy-fp8v.yaml

@@ -0,0 +1,7 @@
+# Issue 2 probe: Qwen-Image-2.1 with FP8 V-only prefix-KV storage.
+# Merged over the pipeline's declared single-stage topology; every other
+# field falls back to StageDeployConfig defaults.
+stages:
+  - stage_id: 0
+    engine_extras:
+      prefix_kv_cache_dtype: "fp8_v"

probe/issue-2/evidence/rtx3090/psnr-edit-compose-pair.json

@@ -0,0 +1,30 @@
+{
+ "cpuoff-expandable": {
+  "rows": [
+   {
+    "seed": 2301,
+    "psnr_db": 26.09,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2302,
+    "psnr_db": 28.75,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2303,
+    "psnr_db": 26.51,
+    "modes": "RGBA/RGBA"
+   },
+   {
+    "seed": 2304,
+    "psnr_db": 28.92,
+    "modes": "RGBA/RGBA"
+   }
+  ],
+  "mean_db": 27.57,
+  "min_db": 26.09,
+  "max_db": 28.92,
+  "n": 4
+ }
+}

probe/issue-2/evidence/rtx3090/runs-summary.json

@@ -0,0 +1,646 @@
+{
+ "runs": {
+  "cpuoff": {
+   "flags": "--enable-cpu-offload",
+   "vram_peak_mib": 24015,
+   "functional.json": [
+    {
+     "fixture_id": "edit-expression-ysolde",
+     "seed": 2301,
+     "n_references": 1,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "864x1152",
+     "wall_s": 34.16,
+     "http": 200,
+     "bytes": 3983827,
+     "sha256": "3c4eb2ef129dbf8afe518b32069cdd9db7811cdedf3ac09557398de193bff972",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 34077.84652709961,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 995328,
+        "denoise_step_latency_ms": 1703.8923263549805,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 34078.10688018799,
+        "image_time_to_first_output_ms": 34077.83102989197,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.23758299903420266,
+       "stage_0_gen_ms": 34077.84652709961
+      },
+      "peak_memory_mb": 23444.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2301,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.68,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "45fac0b8683cac36d4741f71bd1233b66b41b41b2e322c5a870fb6bc303b2ec4",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36615.47255516052,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1830.7736277580261,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36615.6051158905,
+        "image_time_to_first_output_ms": 36615.44394493103,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.12273299944354221,
+       "stage_0_gen_ms": 36615.47255516052
+      },
+      "peak_memory_mb": 23638.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1344x768",
+     "wall_s": 12.24,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 306.38 MiB is free. Including non-PyTorch memory, this process has 23.18 GiB memory in use. Of the allocated memory 21.02 GiB is allocated by PyTorch, and 1.84 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    }
+   ]
+  },
+  "cpuoff-expandable": {
+   "flags": "--enable-cpu-offload",
+   "vram_peak_mib": 24095,
+   "functional.json": [
+    {
+     "fixture_id": "edit-expression-ysolde",
+     "seed": 2301,
+     "n_references": 1,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "864x1152",
+     "wall_s": 33.84,
+     "http": 200,
+     "bytes": 3983827,
+     "sha256": "3c4eb2ef129dbf8afe518b32069cdd9db7811cdedf3ac09557398de193bff972",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 33751.58882141113,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 995328,
+        "denoise_step_latency_ms": 1687.5794410705566,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 33751.80244445801,
+        "image_time_to_first_output_ms": 33751.57427787781,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.19045899898628704,
+       "stage_0_gen_ms": 33751.58882141113
+      },
+      "peak_memory_mb": 21458.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2301,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.68,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "45fac0b8683cac36d4741f71bd1233b66b41b41b2e322c5a870fb6bc303b2ec4",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36616.71161651611,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1830.8355808258057,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36616.929054260254,
+        "image_time_to_first_output_ms": 36616.68133735657,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.1754260010784492,
+       "stage_0_gen_ms": 36616.71161651611
+      },
+      "peak_memory_mb": 21896.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1344x768",
+     "wall_s": 13.39,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 382.38 MiB is free. Including non-PyTorch memory, this process has 23.11 GiB memory in use. Of the allocated memory 22.29 GiB is allocated by PyTorch, and 505.35 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    }
+   ],
+   "measured-edit-compose-pair.json": [
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2301,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.61,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "45fac0b8683cac36d4741f71bd1233b66b41b41b2e322c5a870fb6bc303b2ec4",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36539.32213783264,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1826.966106891632,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36539.42966461182,
+        "image_time_to_first_output_ms": 36539.30997848511,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.09375800073030405,
+       "stage_0_gen_ms": 36539.32213783264
+      },
+      "peak_memory_mb": 23332.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2302,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.71,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "e45a25c5b45ac7b1818843d2f589d164f3f780432472a692defc7569b5c28419",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36643.63503456116,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1832.1817517280579,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36643.73755455017,
+        "image_time_to_first_output_ms": 36643.60785484314,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.10074199963128194,
+       "stage_0_gen_ms": 36643.63503456116
+      },
+      "peak_memory_mb": 21976.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2303,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.84,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "7cf52663ff4691833241766a7c7aa45b373ff54044599887f8bab0e2519c7846",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36772.2589969635,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1838.612949848175,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36771.2607383728,
+        "image_time_to_first_output_ms": 36771.116733551025,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.11694399836414959,
+       "stage_0_gen_ms": 36772.2589969635
+      },
+      "peak_memory_mb": 21996.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2304,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.93,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "54ab259e4669e0b24f900956c37d031d8462501b5e37246b9ce7d2a8d1c8f3fe",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36872.36309051514,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1843.6181545257568,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36871.95873260498,
+        "image_time_to_first_output_ms": 36871.82974815369,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.10266400022374,
+       "stage_0_gen_ms": 36872.36309051514
+      },
+      "peak_memory_mb": 21996.0
+     }
+    }
+   ]
+  },
+  "cpuoff-fp8v": {
+   "flags": "--enable-cpu-offload --deploy-config /home/anodyine/Projects/vllm-omni-val/issue2/deploy-fp8v.yaml",
+   "vram_peak_mib": 24015,
+   "functional.json": [
+    {
+     "fixture_id": "edit-expression-ysolde",
+     "seed": 2301,
+     "n_references": 1,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "864x1152",
+     "wall_s": 34.32,
+     "http": 200,
+     "bytes": 3983827,
+     "sha256": "3c4eb2ef129dbf8afe518b32069cdd9db7811cdedf3ac09557398de193bff972",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 34243.89171600342,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 995328,
+        "denoise_step_latency_ms": 1712.194585800171,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 34244.08268928528,
+        "image_time_to_first_output_ms": 34243.87454986572,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.16898400099307764,
+       "stage_0_gen_ms": 34243.89171600342
+      },
+      "peak_memory_mb": 23444.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2301,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 36.78,
+     "http": 200,
+     "bytes": 4155590,
+     "sha256": "45fac0b8683cac36d4741f71bd1233b66b41b41b2e322c5a870fb6bc303b2ec4",
+     "usage": null,
+     "metrics": {
+      "stage_metrics": {
+       "0": {
+        "stage_id": 0,
+        "final_output_type": "image",
+        "finish_reason": null,
+        "num_tokens_in": 0,
+        "num_tokens_out": 0,
+        "stage_gen_time_ms": 36713.937759399414,
+        "postprocess_time_ms": 0.0,
+        "audio_frames": 0,
+        "audio_sample_rate": 0,
+        "audio_duration_s": 0.0,
+        "image_pixels": 1038336,
+        "denoise_step_latency_ms": 1835.6968879699707,
+        "output_unit_type": "image",
+        "output_unit_count": 1,
+        "serving_time_to_first_output_ms": 36714.062213897705,
+        "image_time_to_first_output_ms": 36713.90199661255,
+        "time_per_output_unit_ms": 0.0,
+        "inter_output_latency_ms": 0.0,
+        "inter_output_latencies_ms": [],
+        "vllm_ttft_ms": 0.0,
+        "vllm_tpot_ms": 0.0,
+        "vllm_itl_ms": 0.0,
+        "vllm_itls_ms": [],
+        "stage_name": "diffusion"
+       }
+      },
+      "stage_id": 0,
+      "final_output_type": "image",
+      "num_tokens_in": 0,
+      "num_tokens_out": 0,
+      "stage_durations": {
+       "queue_wait_ms": 0.12299100126256235,
+       "stage_0_gen_ms": 36713.937759399414
+      },
+      "peak_memory_mb": 23638.0
+     }
+    },
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1344x768",
+     "wall_s": 12.25,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 306.38 MiB is free. Including non-PyTorch memory, this process has 23.18 GiB memory in use. Of the allocated memory 21.02 GiB is allocated by PyTorch, and 1.84 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    }
+   ]
+  },
+  "cpuoff-layerwise": {
+   "flags": "--enable-cpu-offload --enable-layerwise-offload",
+   "vram_peak_mib": 24137,
+   "functional.json": [
+    {
+     "fixture_id": "edit-expression-ysolde",
+     "seed": 2301,
+     "n_references": 1,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "864x1152",
+     "wall_s": 26.18,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 1.07 GiB. GPU 0 has a total capacity of 23.61 GiB of which 602.38 MiB is free. Including non-PyTorch memory, this process has 22.94 GiB memory in use. Of the allocated memory 21.76 GiB is allocated by PyTorch, and 852.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    },
+    {
+     "fixture_id": "edit-compose-pair",
+     "seed": 2301,
+     "n_references": 2,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1248x832",
+     "wall_s": 3.32,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 296.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 40.38 MiB is free. Including non-PyTorch memory, this process has 23.49 GiB memory in use. Of the allocated memory 21.88 GiB is allocated by PyTorch, and 1.26 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    },
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "steps": 20,
+     "true_cfg_scale": 1.0,
+     "size": "1344x768",
+     "wall_s": 3.1,
+     "http": 500,
+     "error": "{\"error\":{\"message\":\"Image edit failed: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 290.38 MiB is free. Including non-PyTorch memory, this process has 23.24 GiB memory in use. Of the allocated memory 21.87 GiB is allocated by PyTorch, and 1.03 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org<path>#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)\",\"type\":\"Internal Server Error\",\"param\":null,\"code\":500}}"
+    }
+   ]
+  },
+  "default": {
+   "flags": "flags:",
+   "serve_rc": 1,
+   "serve_wall_s": 14,
+   "vram_peak_mib": 23939
+  },
+  "offline-fp8v": {
+   "offline-edit-compose-four.json": [
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "kv_dtype": "fp8_v",
+     "wall_s": 15.91,
+     "error": "RuntimeError: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 162.38 MiB is free. Including non-PyTorch memory, this process has 23.32 GiB memory in use. Of the allocated memory 20.86 GiB is allocated by PyTorch, and 2.14 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)"
+    }
+   ]
+  },
+  "offline-fp8v-expandable": {
+   "offline-edit-compose-four.json": [
+    {
+     "fixture_id": "edit-compose-four",
+     "seed": 2301,
+     "n_references": 4,
+     "kv_dtype": "fp8_v",
+     "wall_s": 17.83,
+     "error": "RuntimeError: CUDA out of memory. Tried to allocate 490.00 MiB. GPU 0 has a total capacity of 23.61 GiB of which 482.38 MiB is free. Including non-PyTorch memory, this process has 23.01 GiB memory in use. Of the allocated memory 22.22 GiB is allocated by PyTorch, and 483.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)"
+    }
+   ]
+  }
+ }
+}

probe/issue-2/evidence/rtx5080/runs-summary.json

@@ -0,0 +1,40 @@
+{
+ "runs": {
+  "cpuoff": {
+   "flags": "--enable-cpu-offload",
+   "serve_rc": 1,
+   "serve_wall_s": 9,
+   "vram_peak_mib": 15856
+  },
+  "cpuoff-layerwise": {
+   "flags": "--enable-cpu-offload --enable-layerwise-offload",
+   "serve_rc": 1,
+   "serve_wall_s": 10,
+   "vram_peak_mib": 15856
+  },
+  "default": {
+   "flags": "flags:",
+   "serve_rc": 1,
+   "serve_wall_s": 12,
+   "vram_peak_mib": 15926
+  },
+  "fp8": {
+   "flags": "--enable-cpu-offload --quantization fp8",
+   "serve_rc": 1,
+   "serve_wall_s": 13,
+   "vram_peak_mib": 15930
+  },
+  "fp8-expandable": {
+   "flags": "--enable-cpu-offload --quantization fp8",
+   "serve_rc": 1,
+   "serve_wall_s": 9,
+   "vram_peak_mib": 15876
+  },
+  "int8": {
+   "flags": "--enable-cpu-offload --quantization int8",
+   "serve_rc": 1,
+   "serve_wall_s": 8,
+   "vram_peak_mib": 15930
+  }
+ }
+}

probe/issue-2/fixtures.json

@@ -0,0 +1,36 @@
+{
+  "edit-expression-ysolde": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde"
+    ],
+    "width": 864,
+    "height": 1152,
+    "prompt": "Change only her expression and the chart: she now laughs openly and lowers the chart; keep her face, hair, clothing, the room, the framing, and the lighting as in the reference. Result: Ysolde laughs at something off-frame in the map room. Emotional tone: sudden delight.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Laughing openly, the chart lowered to her chest.\nSetting: The map room of a harbor guildhall, charts pinned to timber walls, rain on the window.\nComposition: close-up; head and shoulders, three-quarter view, face in sharp focus; lighting: cool grey window light from the left with a warm lamp glow from below\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
+    "negative_prompt": null
+  },
+  "edit-compose-pair": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde",
+      "ref-cutout-tamsin"
+    ],
+    "width": 1248,
+    "height": 832,
+    "prompt": "Place the woman from the first reference and the figure from the second reference together; keep both identities exactly. Result: Ysolde and Tamsin share one unrolled map at a harbor wall. Emotional tone: wary cooperation.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Holding one end of an unrolled map, explaining a route.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Holding the other end of the map, listening sceptically.\nSetting: A sea wall above the harbor at midday, fishing boats below.\nComposition: medium shot; both figures side by side, the map between them; lighting: bright midday sun with short shadows\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
+    "negative_prompt": null
+  },
+  "edit-compose-four": {
+    "mode": "edit",
+    "references": [
+      "ref-portrait-ysolde",
+      "ref-storm-corvin",
+      "ref-cutout-tamsin",
+      "ref-shop-hesper"
+    ],
+    "width": 1344,
+    "height": 768,
+    "prompt": "Bring the four characters from the references together in one scene; keep every identity exactly. Result: All four gather around the apothecary counter. Emotional tone: conspiratorial calm.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Unfolding a chart on the counter.\nSubject: Corvin Ashdown, an adult man in his mid-fifties, with pale, heavily freckled skin, short receding red hair and a grey-streaked red beard, and pale blue eyes; wearing a dark green harbormaster's greatcoat with a silver chain of office; a black leather patch over the left eye. Leaning on the doorframe.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Sniffing a stoppered bottle.\nSubject: Hesper Quill, an adult woman in her early seventies, with light brown, deeply weathered skin, long white hair in one thick plait, and grey-green eyes; wearing a lavender dress under a stained canvas apothecary's apron; round wire-rimmed spectacles and a silver ring on every finger of the right hand. Behind the counter, weighing herbs on a brass scale.\nSetting: Inside the apothecary: shelves of bottled remedies, drying herbs hanging from the beams, a long counter.\nComposition: wide shot; from behind the counter's end, all four faces visible; lighting: warm lamplight with green light filtering through bottle glass\nStyle: Anime illustration: clean line art, cel shading, flat vivid colour, expressive faces.",
+    "negative_prompt": null
+  }
+}

probe/issue-2/offline_fp8v.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python
+"""Issue 2 probe: offline Qwen-Image-2.1 edit driver with prefix-KV extras.
+
+The served path cannot reach `extras.prefix_kv_cache_dtype` on commit
+8484d85d (the structured-config bridge drops deploy-YAML extras), so this
+driver exercises the knob through the offline `Omni(...)` engine kwargs —
+the other route the recipe names. It mirrors image_edit.py's request
+construction: same prompt builder, same sampling fields, one request at a
+time.
+
+Usage: offline_fp8v.py --out RUN_DIR [--kv-dtype fp8_v] [--fixture FID]
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import pathlib
+import time
+
+import torch
+from PIL import Image
+
+from vllm_omni.entrypoints.omni import Omni
+from vllm_omni.inputs.data import OmniDiffusionSamplingParams
+from vllm_omni.model_extras import (
+    build_image_to_image_prompt,
+    get_model_class_name,
+)
+from vllm_omni.platforms import current_omni_platform
+
+W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
+SUITE = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
+)
+SNAP = (
+    pathlib.Path.home()
+    / ".local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758"
+)
+STEPS = 20
+CFG = 1.0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", required=True, type=pathlib.Path)
+    parser.add_argument("--kv-dtype", default="fp8_v")
+    parser.add_argument("--fixture", default="edit-compose-four")
+    parser.add_argument("--seeds", type=lambda s: tuple(int(x) for x in s.split(",")), default=(2301,))
+    parser.add_argument("--no-extras", action="store_true", help="run without the extras knob (control)")
+    args = parser.parse_args()
+
+    fixture = json.loads((W / "fixtures.json").read_text())[args.fixture]
+    refs = [Image.open(SUITE / f"{r}.png").convert("RGB") for r in fixture["references"]]
+
+    omni_kwargs = dict(
+        model=str(SNAP),
+        enforce_eager=True,
+        enable_cpu_offload=True,
+        init_timeout=1200,
+    )
+    if not args.no_extras:
+        omni_kwargs["extras"] = {"prefix_kv_cache_dtype": args.kv_dtype}
+    t0 = time.time()
+    omni = Omni(**omni_kwargs)
+    print(f"load_s={time.time() - t0:.1f} extras={omni_kwargs.get('extras')}", flush=True)
+    model_class_name = get_model_class_name(omni)
+
+    args.out.mkdir(parents=True, exist_ok=True)
+    records = []
+    for seed in args.seeds:
+        generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(seed)
+        prompt_dict = build_image_to_image_prompt(
+            model_class_name=model_class_name,
+            prompt=fixture["prompt"],
+            negative_prompt=None,
+            input_image=refs[0] if len(refs) == 1 else refs,
+            height=fixture["height"],
+            width=fixture["width"],
+        )
+        params = OmniDiffusionSamplingParams(
+            generator=generator,
+            seed=seed,
+            true_cfg_scale=CFG,
+            num_inference_steps=STEPS,
+            height=fixture["height"],
+            width=fixture["width"],
+        )
+        torch.cuda.reset_peak_memory_stats()
+        t1 = time.time()
+        rec = {"fixture_id": args.fixture, "seed": seed, "n_references": len(refs), "kv_dtype": args.kv_dtype}
+        try:
+            outputs = omni.generate(prompt_dict, sampling_params_list=[params])
+            wall = time.time() - t1
+            images = None
+            for output in outputs:
+                images = getattr(output, "images", None)
+                if images:
+                    break
+            if not images:
+                raise RuntimeError("no images in output")
+            out_path = args.out / "measured" / f"{args.fixture}.s{seed}.png"
+            out_path.parent.mkdir(parents=True, exist_ok=True)
+            images[0].save(out_path)
+            png = out_path.read_bytes()
+            rec.update(
+                wall_s=round(wall, 2),
+                peak_gpu_mb=round(torch.cuda.max_memory_allocated() / 2**20, 1),
+                bytes=len(png),
+                sha256=hashlib.sha256(png).hexdigest(),
+            )
+        except Exception as exc:  # noqa: BLE001 - recorded as the probe result
+            rec.update(wall_s=round(time.time() - t1, 2), error=f"{type(exc).__name__}: {exc}"[:2000])
+        records.append(rec)
+        print(json.dumps(rec), flush=True)
+
+    (args.out / f"offline-{args.fixture}.json").write_text(json.dumps(records, indent=1) + "\n")
+    ok = all("sha256" in r for r in records)
+    print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

probe/issue-2/probe.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python
+"""Issue 2 probe client: reference-conditioned requests through the API.
+
+Functional pass: one-, two-, and four-reference fixtures at seed 2301.
+Measured pass: the four-reference fixture at seeds 2301-2304.
+
+Every request mirrors the sealed Diffusers baseline attempt
+2026-10-07T170729Z-q3-4r (same frozen references, same rendered prompt,
+20 steps, true_cfg_scale 1.0, fixture resolution, no negative prompt),
+submitted one at a time through POST /v1/images/edits.
+
+Usage: probe.py --out RUN_DIR [--functional] [--measured] [--host H:P]
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import hashlib
+import json
+import pathlib
+import time
+
+import requests
+
+W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
+SUITE = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
+)
+STEPS = 20
+CFG = 1.0
+FUNCTIONAL_FIXTURES = (
+    "edit-expression-ysolde",  # 1 reference
+    "edit-compose-pair",        # 2 references
+    "edit-compose-four",        # 4 references
+)
+MEASURED_FIXTURE = "edit-compose-four"
+MEASURED_SEEDS = (2301, 2302, 2303, 2304)
+
+
+def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
+    files = []
+    handles = []
+    for ref in fixture["references"]:
+        path = SUITE / f"{ref}.png"
+        handle = open(path, "rb")
+        handles.append(handle)
+        files.append(("image[]", (path.name, handle, "image/png")))
+    data = {
+        "model": "Qwen/Qwen-Image-2.1",
+        "prompt": fixture["prompt"],
+        "seed": str(seed),
+        "num_inference_steps": str(STEPS),
+        "true_cfg_scale": str(CFG),
+        "size": f"{fixture['width']}x{fixture['height']}",
+        "n": "1",
+        "response_format": "b64_json",
+        "output_format": "png",
+        "return_stage_metrics": "true",
+    }
+    rec: dict = {
+        "fixture_id": None,
+        "seed": seed,
+        "n_references": len(fixture["references"]),
+        "steps": STEPS,
+        "true_cfg_scale": CFG,
+        "size": f"{fixture['width']}x{fixture['height']}",
+    }
+    t0 = time.time()
+    try:
+        response = requests.post(
+            f"http://{host}/v1/images/edits", data=data, files=files, timeout=1200
+        )
+        rec["wall_s"] = round(time.time() - t0, 2)
+        rec["http"] = response.status_code
+        if response.status_code == 200:
+            body = response.json()
+            png = base64.b64decode(body["data"][0]["b64_json"])
+            rec["bytes"] = len(png)
+            rec["sha256"] = hashlib.sha256(png).hexdigest()
+            rec["usage"] = body.get("usage")
+            for key in ("stage_metrics", "metrics", "timings"):
+                if body.get(key):
+                    rec[key] = body[key]
+            return rec, png
+        rec["error"] = response.text[:2000]
+        return rec, None
+    except Exception as exc:  # noqa: BLE001 - recorded as the probe result
+        rec["wall_s"] = round(time.time() - t0, 2)
+        rec["error"] = f"{type(exc).__name__}: {exc}"
+        return rec, None
+    finally:
+        for handle in handles:
+            handle.close()
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", required=True, type=pathlib.Path)
+    parser.add_argument("--host", default="127.0.0.1:8091")
+    parser.add_argument("--functional", action="store_true")
+    parser.add_argument("--measured", action="store_true")
+    parser.add_argument(
+        "--fixture",
+        default=MEASURED_FIXTURE,
+        help="fixture id for --measured (default: edit-compose-four)",
+    )
+    parser.add_argument(
+        "--seeds",
+        type=lambda s: tuple(int(x) for x in s.split(",")),
+        default=MEASURED_SEEDS,
+        help="comma-separated seeds for --measured",
+    )
+    args = parser.parse_args()
+
+    fixtures = json.loads((W / "fixtures.json").read_text())
+    args.out.mkdir(parents=True, exist_ok=True)
+
+    plan: list[tuple[str, pathlib.Path, int]] = []
+    if args.functional:
+        for fixture_id in FUNCTIONAL_FIXTURES:
+            plan.append(
+                (fixture_id, args.out / "functional" / f"{fixture_id}.png", 2301)
+            )
+    if args.measured:
+        for seed in args.seeds:
+            plan.append(
+                (
+                    args.fixture,
+                    args.out / "measured" / f"{args.fixture}.s{seed}.png",
+                    seed,
+                )
+            )
+
+    records = []
+    for fixture_id, out_path, seed in plan:
+        rec, png = submit(args.host, fixtures[fixture_id], seed)
+        rec["fixture_id"] = fixture_id
+        records.append(rec)
+        print(json.dumps(rec), flush=True)
+        if png is not None:
+            out_path.parent.mkdir(parents=True, exist_ok=True)
+            out_path.write_bytes(png)
+        elif rec.get("http") == 200:
+            print(f"warning: 200 but no image for {fixture_id} seed {seed}", flush=True)
+
+    suffix_parts = []
+    if args.functional:
+        suffix_parts.append("functional")
+    if args.measured:
+        suffix_parts.append(f"measured-{args.fixture}")
+    suffix = "-".join(suffix_parts)
+    (args.out / f"{suffix}.json").write_text(
+        json.dumps(records, indent=1) + "\n"
+    )
+    ok = all(r.get("http") == 200 and "sha256" in r for r in records)
+    print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

probe/issue-2/psnr.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+"""Issue 2 probe: PSNR of vLLM outputs against the sealed Diffusers
+four-reference baseline attempt 2026-10-07T170729Z-q3-4r.
+
+Usage: psnr.py --fixture FIXTURE_ID RUN_DIR [RUN_DIR ...]
+
+Compares each RUN_DIR/measured/FIXTURE_ID.sSEED.png against the baseline
+sample with the same fixture and seed. RGBA outputs are composited over
+white, matching issue 1's comparator.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import pathlib
+
+import numpy as np
+from PIL import Image
+
+ATTEMPT = (
+    pathlib.Path.home()
+    / ".local/share/multi-model-helper-server/data/evaluation/media/image/attempts/2026-10-07T170729Z-q3-4r"
+)
+
+
+def to_rgb(array: np.ndarray) -> np.ndarray:
+    if array.shape[2] == 4:
+        alpha = array[..., 3:4].astype(np.float64) / 255.0
+        bg = np.full_like(array[..., :3], 255, dtype=np.float64)
+        return np.round(
+            array[..., :3].astype(np.float64) * alpha + bg * (1 - alpha)
+        ).clip(0, 255).astype(np.uint8)
+    return array[..., :3]
+
+
+def psnr(a: np.ndarray, b: np.ndarray) -> float:
+    mse = float(np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2))
+    return float("inf") if mse == 0 else 10 * math.log10(255.0**2 / mse)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--fixture", default="edit-compose-four")
+    parser.add_argument("runs", nargs="+", type=pathlib.Path)
+    args = parser.parse_args()
+    fixture = args.fixture
+
+    samples = json.loads((ATTEMPT / "samples.json").read_text())["samples"]
+    baseline = {
+        s["seed"]: ATTEMPT / s["file"]
+        for s in samples
+        if s["fixture_id"] == fixture and s["status"] == "succeeded"
+    }
+    summary = {}
+    for run_dir in args.runs:
+        rows = []
+        for seed, base_path in sorted(baseline.items()):
+            ours = run_dir / "measured" / f"{fixture}.s{seed}.png"
+            if not ours.exists():
+                rows.append({"seed": seed, "error": "missing"})
+                continue
+            bm, om = Image.open(base_path), Image.open(ours)
+            ba, oa = np.asarray(bm), np.asarray(om)
+            if ba.shape[:2] != oa.shape[:2]:
+                rows.append(
+                    {"seed": seed, "error": f"size {oa.shape[:2]} vs {ba.shape[:2]}"}
+                )
+                continue
+            use_rgba = bm.mode == "RGBA" and om.mode == "RGBA"
+            a, b = (ba, oa) if use_rgba else (to_rgb(ba), to_rgb(oa))
+            rows.append(
+                {
+                    "seed": seed,
+                    "psnr_db": round(psnr(a, b), 2),
+                    "modes": f"{bm.mode}/{om.mode}",
+                }
+            )
+        vals = [r["psnr_db"] for r in rows if "psnr_db" in r]
+        if vals:
+            summary[run_dir.name] = {
+                "rows": rows,
+                "mean_db": round(sum(vals) / len(vals), 2),
+                "min_db": min(vals),
+                "max_db": max(vals),
+                "n": len(vals),
+            }
+        else:
+            summary[run_dir.name] = {"rows": rows}
+    print(json.dumps(summary, indent=1))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

probe/issue-2/serve.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Issue 2 probe: serve Qwen-Image-2.1 on the RTX 3090 with one labeled
+# flag set. VRAM is polled once per second for the whole server lifetime,
+# matching issue 1's instrument (nvidia-smi csv series).
+#
+# Usage: serve.sh LABEL [extra vllm flags...]
+set -uo pipefail
+
+BASE=$HOME/Projects/vllm-omni-val
+W=$BASE/issue2
+LABEL=$1; shift
+RUN=$W/runs/$LABEL
+mkdir -p "$RUN"
+
+SNAP=$HOME/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758
+export HF_HOME=$HOME/.local/state/multi-model-helper-server/state/image-service/weights
+export HF_HUB_OFFLINE=1
+
+( while sleep 1; do nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader; done ) > "$RUN/vram.log" 2>&1 &
+POLL=$!
+T0=$(date +%s)
+echo "flags: $*" > "$RUN/flags.txt"
+
+$BASE/venv/bin/vllm serve "$SNAP" --omni --port 8091 \
+  --served-model-name Qwen/Qwen-Image-2.1 \
+  --enforce-eager "$@" > "$RUN/serve.log" 2>&1
+RC=$?
+kill $POLL 2>/dev/null
+echo "serve_rc=$RC wall=$(( $(date +%s) - T0 ))s" >> "$RUN/serve.log"