diff --git a/probe/issue-2/FINDINGS.md b/probe/issue-2/FINDINGS.md new file mode 100644 index 000000000..5940da2bd --- /dev/null +++ b/probe/issue-2/FINDINGS.md @@ -0,0 +1,297 @@ +# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB) + +Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and +`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`, +vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130, +transformers 5.14.1, diffusers git `122b1e11`), served from the +digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with +`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds, +20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed +Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile +`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same +frozen references-suite-2 images, same rendered brief-prose/1 prompts, +same seeds 2301–2304, same per-fixture resolutions. + +## Verdicts + +**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the +conditioning KV exists anywhere in the pipeline (verified by code read and +by exhausting every reachable knob). At BF16 the served reference ceiling +is two condition images at `--enable-cpu-offload` — with the conditioning +cache ON, at 2.2× the Diffusers nocache baseline's speed. Four references +OOM inside the DiT forward under every BF16 flag combination, and the +binding constraint is the resident BF16 working set plus a fixed ~490 MiB +image-MLP activation, not the conditioning KV: the per-request cache +contributes only roughly 0.25–0.3 GiB per reference (see "KV size" below). +FP8 weight quantization (`--quantization fp8`) does free the ~6 GiB the +arithmetic calls for — all of one, two, three, and four references then +fit and run (4-ref: 44.1–44.4 s, server peak 23,522 MB, nvidia-smi series +max 23,983 MiB) — **but on this commit and card the FP8 path produces +structurally broken output** (~96.5–97% transparent RGBA, grey noise RGB, +PSNR 7–9 dB), for the edit path at every reference count and for T2I +alike (see "FP8 quality collapse"). So the verdict stands: no usable path +to four references, and none that spills the KV. + +**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.** +Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8, +fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB +allocation with the card full. The failure is in +`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the +unquantized parameter skeleton is allocated on-device at construction, +before any quantization or offload path engages, so no runtime flag can +shrink the load-time peak. The reference path is unreachable on this card +with this stack; its one-reference Diffusers ceiling (B2g) relied on a +torchao-int8 + group-offload load path that never materializes the full +BF16 model on the GPU. The conditioning-KV question is moot on this card. + +## Where the cache lives (issue scope item 2) + +- Created per request at + `vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207` + (`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG + branch `cond`/`uncond`), valid because `causal_condition=True` makes + text + condition-image K/V timestep-independent. +- Written at prefill (first denoise step) and read at decode in + `qwen_image_21/.../qwen_image_21_transformer.py:538-598` + (`branch_cache["key"] = key[:, :cache_write_len].clone()` and the + decode-side `torch.cat` prepend). +- Batched mixed-phase assembly/scatter at + `pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and + `:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in + `qwen_image_21/decode_graph.py` (inactive here — eager). +- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's + paged KVCacheManager — the model deliberately runs `DENSE_LEGACY` + (`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of + `PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via + `extras.prefix_kv_cache_dtype`. +- **Lifetime**: per-request only; freed by refcount when the runner drops + the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`). + No cross-request retention, no LRU, no dedup — unlike Diffusers' helper + cache, nothing persists between requests. +- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` / + `fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the + propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode` + (static-buffer copy), `cuda_graph_max_decode_graphs`. + `--swap-space` and `--cpu-offload-gb` **do not exist** in this fork; + `--gpu-memory-utilization` sizes only the unrelated paged-scheduler + path. The weight-offload flags (`--enable-cpu-offload`, + `--enable-layerwise-offload`) never move KV tensors. +- **Spill path**: none. A host-DRAM contribution would go at the prefill + write / decode read (`qwen_image_21_transformer.py:541-598` — allocate + on pinned CPU, prefetch before the decode `torch.cat`) or at the batch + boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the + request-owns-independent-storage invariant that the decode-graph weakref + identity check at `decode_graph.py:350-363` depends on. + +## KV size (empirical) + +Between one and two references the per-request peak moves by only +~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+ +expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image — +roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the +helper-system measured Diffusers retaining (multi-model-helper-server +#160). The cache is per-request and lean; it is not what blocks four +references. + +## Run matrix — RTX 3090 + +All runs: `vllm serve --omni --port 8091 --enforce-eager +--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via +`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under +`~/Projects/vllm-omni-val/issue2/runs/