Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
4f5f99d8c6 | ||
|
admin
|
67eb7fdb48 |
No files matched your search
@@ -0,0 +1,297 @@
|
||||
# Issue 2 probe: Qwen-Image conditioning KV on RTX 3090 (24 GB) and RTX 5080 (16 GB)
|
||||
|
||||
Probed 2026-10-07/08 on `cachyos-alexander` (RTX 3090, sm_86) and
|
||||
`cachyos-9900x-o11d` (RTX 5080, sm_120), both at upstream main `8484d85d1`,
|
||||
vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||||
transformers 5.14.1, diffusers git `122b1e11`), served from the
|
||||
digest-pinned snapshot `Qwen/Qwen-Image-2.1@790c9263` with
|
||||
`HF_HUB_OFFLINE=1`. Eager mode, one request at a time, fixed seeds,
|
||||
20 steps, `true_cfg_scale 1.0`, no negative prompt — mirroring the sealed
|
||||
Diffusers attempt `2026-10-07T170729Z-q3-4r` (profile
|
||||
`image-q3-int8-dit-20-steps-nocache-4ref`) fixture-for-fixture: same
|
||||
frozen references-suite-2 images, same rendered brief-prose/1 prompts,
|
||||
same seeds 2301–2304, same per-fixture resolutions.
|
||||
|
||||
## Verdicts
|
||||
|
||||
**RTX 3090 (24 GB): not-built.** No host-DRAM spill path for the
|
||||
conditioning KV exists anywhere in the pipeline (verified by code read and
|
||||
by exhausting every reachable knob). At BF16 the served reference ceiling
|
||||
is two condition images at `--enable-cpu-offload` — with the conditioning
|
||||
cache ON, at 2.2× the Diffusers nocache baseline's speed. Four references
|
||||
OOM inside the DiT forward under every BF16 flag combination, and the
|
||||
binding constraint is the resident BF16 working set plus a fixed ~490 MiB
|
||||
image-MLP activation, not the conditioning KV: the per-request cache
|
||||
contributes only roughly 0.25–0.3 GiB per reference (see "KV size" below).
|
||||
FP8 weight quantization (`--quantization fp8`) does free the ~6 GiB the
|
||||
arithmetic calls for — all of one, two, three, and four references then
|
||||
fit and run (4-ref: 44.1–44.4 s, server peak 23,522 MB, nvidia-smi series
|
||||
max 23,983 MiB) — **but on this commit and card the FP8 path produces
|
||||
structurally broken output** (~96.5–97% transparent RGBA, grey noise RGB,
|
||||
PSNR 7–9 dB), for the edit path at every reference count and for T2I
|
||||
alike (see "FP8 quality collapse"). So the verdict stands: no usable path
|
||||
to four references, and none that spills the KV.
|
||||
|
||||
**RTX 5080 (16 GB): not-built, stronger — the server cannot initialize.**
|
||||
Six init attempts (default, cpu-offload, cpu+layerwise offload, fp8,
|
||||
fp8+expandable-segments, int8) all OOM at the same late-init 96 MiB
|
||||
allocation with the card full. The failure is in
|
||||
`vllm_omni/diffusion/models/utils.py:159-168` (`init_parameters`): the
|
||||
unquantized parameter skeleton is allocated on-device at construction,
|
||||
before any quantization or offload path engages, so no runtime flag can
|
||||
shrink the load-time peak. The reference path is unreachable on this card
|
||||
with this stack; its one-reference Diffusers ceiling (B2g) relied on a
|
||||
torchao-int8 + group-offload load path that never materializes the full
|
||||
BF16 model on the GPU. The conditioning-KV question is moot on this card.
|
||||
|
||||
## Where the cache lives (issue scope item 2)
|
||||
|
||||
- Created per request at
|
||||
`vllm_omni/diffusion/models/qwen_image_21/pipeline_qwen_image_21.py:1204-1207`
|
||||
(`state.extra["kv_cache"]`, one dict per DiT block — 32 — keyed by CFG
|
||||
branch `cond`/`uncond`), valid because `causal_condition=True` makes
|
||||
text + condition-image K/V timestep-independent.
|
||||
- Written at prefill (first denoise step) and read at decode in
|
||||
`qwen_image_21/.../qwen_image_21_transformer.py:538-598`
|
||||
(`branch_cache["key"] = key[:, :cache_write_len].clone()` and the
|
||||
decode-side `torch.cat` prepend).
|
||||
- Batched mixed-phase assembly/scatter at
|
||||
`pipeline_qwen_image_21.py:1218-1265` (`_assemble_kv_cache`) and
|
||||
`:1303-1325` (`_scatter_kv_cache`); decode-graph static copies in
|
||||
`qwen_image_21/decode_graph.py` (inactive here — eager).
|
||||
- **Allocator**: plain `torch.Tensor` `.clone()`s on the GPU. Not vLLM's
|
||||
paged KVCacheManager — the model deliberately runs `DENSE_LEGACY`
|
||||
(`recipes/Qwen/Qwen-Image-2.1.md:318-329` documents the rejection of
|
||||
`PAGED_SCHEDULER`). Optional FP8-E4M3 storage (K+V or V-only) via
|
||||
`extras.prefix_kv_cache_dtype`.
|
||||
- **Lifetime**: per-request only; freed by refcount when the runner drops
|
||||
the request state (`diffusion/worker/diffusion_model_runner.py:1034ff`).
|
||||
No cross-request retention, no LRU, no dedup — unlike Diffusers' helper
|
||||
cache, nothing persists between requests.
|
||||
- **Flags that reach it**: `extras.prefix_kv_cache_dtype` (`fp8` /
|
||||
`fp8_v`; deploy-YAML stage extras or offline `Omni(...)` kwarg — see the
|
||||
propagation gap below), `--enforce-eager` / `--enable-cuda-graph-decode`
|
||||
(static-buffer copy), `cuda_graph_max_decode_graphs`.
|
||||
`--swap-space` and `--cpu-offload-gb` **do not exist** in this fork;
|
||||
`--gpu-memory-utilization` sizes only the unrelated paged-scheduler
|
||||
path. The weight-offload flags (`--enable-cpu-offload`,
|
||||
`--enable-layerwise-offload`) never move KV tensors.
|
||||
- **Spill path**: none. A host-DRAM contribution would go at the prefill
|
||||
write / decode read (`qwen_image_21_transformer.py:541-598` — allocate
|
||||
on pinned CPU, prefetch before the decode `torch.cat`) or at the batch
|
||||
boundary helpers (`pipeline_qwen_image_21.py:1218-1325`), preserving the
|
||||
request-owns-independent-storage invariant that the decode-graph weakref
|
||||
identity check at `decode_graph.py:350-363` depends on.
|
||||
|
||||
## KV size (empirical)
|
||||
|
||||
Between one and two references the per-request peak moves by only
|
||||
~200–450 MiB (cpuoff: 23,444→23,638 MB server-reported; cpuoff+
|
||||
expandable: 21,458→21,896 MB), i.e. ≈0.25–0.3 GiB per condition image —
|
||||
roughly 64 KiB/prefix-token — versus the 2.8 GiB per reference the
|
||||
helper-system measured Diffusers retaining (multi-model-helper-server
|
||||
#160). The cache is per-request and lean; it is not what blocks four
|
||||
references.
|
||||
|
||||
## Run matrix — RTX 3090
|
||||
|
||||
All runs: `vllm serve <snapshot> --omni --port 8091 --enforce-eager
|
||||
--served-model-name Qwen/Qwen-Image-2.1 [+ flags]`, readiness via
|
||||
`/health`, VRAM via 1 Hz `nvidia-smi` series; per-run artifacts under
|
||||
`~/Projects/vllm-omni-val/issue2/runs/<label>/` on the host.
|
||||
|
||||
| Run | Flags | Init | 1-ref | 2-ref | 4-ref | nvidia-smi peak |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| default | (none) | **OOM in 14 s** (23.51 GiB in use at load) | — | — | — | 23,939 MiB |
|
||||
| cpuoff | `--enable-cpu-offload` | ok | OK 34.2 s | OK 36.7 s | **OOM** in request | 24,015 MiB |
|
||||
| cpuoff-layerwise | `+ --enable-layerwise-offload` | ok | **OOM** | **OOM** | **OOM** | 24,137 MiB |
|
||||
| cpuoff-fp8v (yaml) | `+ --deploy-config deploy-fp8v.yaml` | ok | OK 34.3 s | OK 36.8 s | **OOM** | 24,015 MiB |
|
||||
| cpuoff-expandable | `+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` | ok | OK 33.8 s | OK 36.7 s | **OOM** | 24,095 MiB |
|
||||
| fp8 | `--enable-cpu-offload --quantization fp8` | ok | OK 29.9 s | OK 34.1 s (+3-ref OK 38.9 s) | OK 44.3 s — **output broken** | 23,983 MiB |
|
||||
|
||||
The `fp8` row (review follow-up; #1 validated this flag's capacity
|
||||
offline) also covered the previously untested three-reference count via
|
||||
`edit-compose-three` — derived from `edit-compose-four` minus
|
||||
`ref-shop-hesper` and its Subject line, four→three wording, same
|
||||
instrument (recorded in `fixtures.json`). Measured under fp8, seeds
|
||||
2301–2304: four 44.06–44.35 s (peak 22,918 MB), pair 34.62–35.05 s
|
||||
(peak 21,504 MB), three 39.44–39.69 s (peak 23,532 MB) — every request
|
||||
succeeds capacity-wise, and Marlin W8A16 is *faster* than BF16
|
||||
(4-ref 44.3 s vs Diffusers' 145.8 s = 3.3×). The quality outcome is in
|
||||
the next section.
|
||||
|
||||
- 4-ref OOM signature (identical ask across configs): `Tried to allocate
|
||||
490.00 MiB` inside the image-branch MLP forward
|
||||
(`qwen_image_21_transformer.py:293`, via `predict_noise` — prefill
|
||||
encoding has already succeeded). At cpuoff: 21.02 GiB allocated + 1.84
|
||||
GiB reserved-unallocated; at cpuoff-expandable: 22.29 GiB allocated,
|
||||
505 MiB reserved-unallocated — fragmentation removed, still short:
|
||||
genuine capacity gap.
|
||||
- Layerwise offload is strictly counterproductive here: even one reference
|
||||
OOMs (21.76 GiB allocated during the request).
|
||||
- Determinism: seed-2301 outputs are byte-identical (sha256
|
||||
`45fac0b8…` pair, `3c4eb2ef…` single) across cpuoff, cpuoff-fp8v,
|
||||
cpuoff-expandable — the frozen-seed instrument is sound.
|
||||
|
||||
### FP8 quality collapse (and a correction to issue 1)
|
||||
|
||||
Every FP8-weight output this probe produced is structurally broken, not
|
||||
merely noisier:
|
||||
|
||||
- Signature (identical everywhere): RGBA output ~96.5–97% transparent
|
||||
(opaque fraction 0.008–0.035, alpha mean ≈115), near-flat grey RGB
|
||||
(mean ≈120–130), PSNR vs the Diffusers baseline 6.6–9.0 dB. The BF16
|
||||
path on the same requests is 100% opaque and 26–29 dB.
|
||||
- Scope: the serve path at 1, 2, 3, and 4 references (all seeds measured),
|
||||
the offline path with `fp8` minus `img_mlp`, and the offline path with
|
||||
**transformer-only** FP8 (`quantization_config={"transformer":
|
||||
{"method": "fp8"}}` — the text encoder stays BF16). Transformer-only
|
||||
still collapsing pins the culprit on the Marlin-quantized DiT, not the
|
||||
encoder and not the image-MLP. An encoder-only isolator could not
|
||||
initialize (the encoder's bf16→fp8 conversion transient lands on the GPU
|
||||
on top of the resident BF16 DiT and exceeds the card).
|
||||
- **Correction to issue 1**: #1's offline FP8 validation
|
||||
(`qwen_image_21_fp8_all_rtx3090.png`, `…_fp8_imgmlp_…`) shows the same
|
||||
collapse on reinspection (opaque fraction 0.035 both) — that run
|
||||
validated rc=0 and image production, not output structure, so "FP8
|
||||
weight-only works on this card" from #1 overstated it. #1's 18.4 GB
|
||||
capacity measurement stands; its quality implication does not.
|
||||
- Upstream relevance: the recipe's H200 FP8 numbers quote PSNR only for
|
||||
text-encoder FP8 (32.1 dB T2I); DiT-FP8 output quality appears
|
||||
unmeasured upstream, and this probe adds the Ampere/Marlin cell:
|
||||
online FP8 DiT weights yield near-noise latents for Qwen-Image-2.1.
|
||||
Until that is fixed (or validated as Ampere-specific), FP8 cannot serve
|
||||
as the capacity unlock on this card — which is why the 3090 verdict
|
||||
remains `not-built` despite the fp8 capacity row passing.
|
||||
|
||||
### Offline knob runs (3090, `Omni(...)` engine kwargs)
|
||||
|
||||
The one knob that reaches the cache — `extras.prefix_kv_cache_dtype` —
|
||||
was exercised offline (`offline_fp8v.py`, `--enforce-eager
|
||||
--enable-cpu-offload` + engine kwarg):
|
||||
|
||||
| Run | 4-ref seed 2301 | Detail |
|
||||
| --- | --- | --- |
|
||||
| offline fp8_v | **OOM** | 20.86 GiB allocated + 2.14 GiB reserved-unallocated; same 490 MiB img-MLP ask |
|
||||
| offline fp8_v + expandable | **OOM** | 22.22 GiB allocated, 483 MiB reserved-unallocated |
|
||||
|
||||
FP8-V halves the V half of a cache that contributes only ~0.3 GiB per
|
||||
reference — an order of magnitude too small to bridge the gap. The
|
||||
conditioning KV cannot rescue four references on this card by quantization
|
||||
or by anything else; only a genuinely leaner resident working set
|
||||
(quantized DiT weights) would, and on this commit that is blocked by the
|
||||
init path (see 5080).
|
||||
|
||||
### Measured run + PSNR (3090)
|
||||
|
||||
`edit-compose-pair` (2 refs, 1248×832), seeds 2301–2304, on
|
||||
cpuoff-expandable, versus the sealed Diffusers attempt's pair samples
|
||||
(same references, 20 steps; Diffusers side ran an int8-quantized DiT with
|
||||
KV cache off, wall 81.16 s):
|
||||
|
||||
- Wall: **36.6–36.9 s per image** (2.2× faster than Diffusers nocache;
|
||||
server `denoise_step_latency` ≈ 1.83 s × 20 steps).
|
||||
- Peak: 21,976–23,332 MB server-reported; 24,095 MiB nvidia-smi series
|
||||
max.
|
||||
- PSNR vs Diffusers (RGBA-on-RGBA, #1's comparator): **26.09 / 28.75 /
|
||||
26.51 / 28.92 dB; mean 27.57 dB** (n=4). Cross-implementation agreement
|
||||
at matched seed/inputs, not parity — expected given the different
|
||||
samplers/kernels and the Diffusers side's int8 DiT.
|
||||
|
||||
The four-reference PSNR cell the issue nominally asks for is
|
||||
**capacity-blocked at BF16** (every served flag set, and offline with
|
||||
fp8_v). Under the fp8 serve row it becomes obtainable but meaningless as
|
||||
quality: measured seeds 2301–2304 give 7.03–7.21 dB (mean 7.11) against
|
||||
the sealed 4r baseline because the outputs are the collapsed ~97%
|
||||
transparent renders of the FP8 quality section — recorded in
|
||||
`evidence/rtx3090/psnr-fp8-rows.json` as the broken-output datum, not as
|
||||
a usable comparison. The pair row above (27.57 dB at BF16) remains the
|
||||
highest-reference-count usable comparison this card can produce.
|
||||
`edit-compose-four` wall for reference: Diffusers 145.8 s (nocache);
|
||||
vLLM 44.3 s under fp8 (fast but broken) — no valid BF16 number exists.
|
||||
|
||||
## Run matrix — RTX 5080
|
||||
|
||||
Environment built fresh per #1's recipe (userspace CPython 3.13.16 venv —
|
||||
the host only ships 3.14 and has no `/usr/bin/python3.13`; clone pinned to
|
||||
`8484d85d1`; weights/suite verified, the `q3-4r` attempt copied from the
|
||||
3090 so both legs compare against the identical baseline).
|
||||
|
||||
| Run | Flags | Init result |
|
||||
| --- | --- | --- |
|
||||
| default | (none) | OOM in 12 s (15.22 GiB allocated at load) |
|
||||
| cpuoff | `--enable-cpu-offload` | **OOM in 9 s** — DiT stays GPU-resident under cpu-offload; 14.99 GiB allocated |
|
||||
| cpuoff-layerwise | `+ --enable-layerwise-offload` | **OOM in 10 s** — layerwise streams per-forward, after load |
|
||||
| fp8 | `--enable-cpu-offload --quantization fp8` | **OOM in ~10 s** — FP8 path engaged (DeepGEMM init visible) but init still fills the card |
|
||||
| fp8-expandable | `+ expandable_segments` | **OOM** |
|
||||
| int8 | `--enable-cpu-offload --quantization int8` | **OOM in 8 s** — 15.55 GiB in use |
|
||||
|
||||
Common failure: `torch.OutOfMemoryError` at a 96 MiB allocation in
|
||||
`vllm_omni/diffusion/models/utils.py:159-168 init_parameters`
|
||||
(`empty_like` under a device context) — the unquantized parameter
|
||||
skeleton is materialized on-device at pipeline construction, before
|
||||
weight loading, quantization, or offload can act. Host RAM (30 GiB + zram)
|
||||
is a separate ceiling for full weight offload (~32 GiB of BF16 weights)
|
||||
but is not what kills these runs.
|
||||
|
||||
## Propagation gap found (3090 leg)
|
||||
|
||||
The recipe-documented deploy-YAML route for
|
||||
`extras.prefix_kv_cache_dtype` **does not reach the transformer through
|
||||
`vllm serve` on this commit**: with and without the YAML, serve outputs
|
||||
are byte-identical, and the drop is verified offline at the exact
|
||||
serve-time construction (`build_diffusion_config` →
|
||||
`stage_configs[0].diffusion_config.extras` carries only
|
||||
`auxiliary_text_encoder`/`default_llama_model_id` defaults; deploy stage
|
||||
`engine_extras.extras` is parsed correctly by `_stage_engine_overrides`
|
||||
but lost in the structured-config bridge). The offline `Omni(extras=…)`
|
||||
engine kwarg is accepted (load proceeds; the knob then behaves as
|
||||
documented). Shapes tried: flat stage key, `engine_extras.<key>`, and
|
||||
`engine_extras.extras.<key>`. This is a one-line-ish fix candidate in the
|
||||
config bridge, independent of any KV-spill contribution.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
Scripts live in this directory (`serve.sh`, `probe.py`, `psnr.py`,
|
||||
`offline_fp8v.py`, `deploy-fp8v.yaml`) and were pushed verbatim to
|
||||
`~/Projects/vllm-omni-val/issue2/` on each host; `fixtures.json` (rendered
|
||||
brief-prose/1 prompts, reference bindings, dimensions) was generated on
|
||||
the 3090 from the helper repo (`evaluation/media/image/suite.py` +
|
||||
`services/media/image_service/renderer.py`) and copied to the 5080.
|
||||
Server lifecycle ran under transient user units
|
||||
(`systemd-run --user --unit=issue2-serve`); per-run serve/vram logs and
|
||||
request records are under `runs/<label>/` on each host; condensed
|
||||
summaries in `evidence/rtx3090/runs-summary.json` and
|
||||
`evidence/rtx5080/runs-summary.json`.
|
||||
|
||||
## Helper-system follow-through (input to multi-model-helper-server #27 and #164 — not implemented here)
|
||||
|
||||
- **#27 (serving-profile candidacy)**: on the 3090 the vLLM-Omni path
|
||||
serves 1–2 references at 33–37 s/image with the conditioning cache on —
|
||||
2.2× the sealed Diffusers nocache baseline — and per-request cache
|
||||
pressure of ~0.3 GiB/reference means concurrent multi-reference sessions
|
||||
do not accumulate Diffusers-style 2.8 GiB/reference retention. Candidacy
|
||||
holds for ≤2-reference profiles on 24 GB; 4-reference profiles remain
|
||||
Diffusers-only: FP8 unlocks the capacity (all four counts fit, 3.3×
|
||||
baseline speed) but its output is broken on this commit, so the unlock
|
||||
order is upstream FP8-output correctness first, then serving.
|
||||
- **#164 (topology input)**: the 5080 (16 GB) cannot host this stack at
|
||||
all — init-time parameter skeleton exceeds the card before any flag
|
||||
acts (`diffusion/models/utils.py:159-168`). Any topology plan pairing
|
||||
the 5080 with Qwen-Image-2.1 on vLLM-Omni needs upstream load-path work
|
||||
(quantized or CPU-materialized init) first; the B-series torchao-int8
|
||||
Diffusers stack remains that card's only viable image backend.
|
||||
- Upstream contribution seam, if ever pursued: prefill-write/decode-read
|
||||
at `qwen_image_21_transformer.py:541-598` or the batch helpers at
|
||||
`pipeline_qwen_image_21.py:1218-1325`; but note the cache itself is not
|
||||
the binding constraint on either card — the 3090 blocks on the resident
|
||||
BF16 working set + img-MLP activation, the 5080 on init. A spill path
|
||||
alone would not change either verdict, which answers the issue's
|
||||
"does it still serve us before committing to anything upstream-sized":
|
||||
**no — not as the next step**; the load/init path comes first.
|
||||
|
||||
## Out-of-scope confirmation
|
||||
|
||||
No code changes; no upstream PRs; no TP/multi-GPU paths touched; no
|
||||
helper-system deployment evidence produced. The only artifacts are this
|
||||
report, the probe scripts, and per-host run records.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Issue 2 probe: Qwen-Image-2.1 with FP8 V-only prefix-KV storage.
|
||||
# Merged over the pipeline's declared single-stage topology; every other
|
||||
# field falls back to StageDeployConfig defaults.
|
||||
stages:
|
||||
- stage_id: 0
|
||||
engine_extras:
|
||||
prefix_kv_cache_dtype: "fp8_v"
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"cpuoff-expandable": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 26.09,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 28.75,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 26.51,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 28.92,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 27.57,
|
||||
"min_db": 26.09,
|
||||
"max_db": 28.92,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
{
|
||||
"edit-compose-four": {
|
||||
"cpuoff-fp8": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.03,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 7.11,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 7.1,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 7.21,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.11,
|
||||
"min_db": 7.03,
|
||||
"max_db": 7.21,
|
||||
"n": 4
|
||||
},
|
||||
"offline-fp8-imgmlp": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.27,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"error": "missing"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"error": "missing"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"error": "missing"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.27,
|
||||
"min_db": 7.27,
|
||||
"max_db": 7.27,
|
||||
"n": 1
|
||||
},
|
||||
"offline-fp8-dit": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 6.69,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"error": "missing"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"error": "missing"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"error": "missing"
|
||||
}
|
||||
],
|
||||
"mean_db": 6.69,
|
||||
"min_db": 6.69,
|
||||
"max_db": 6.69,
|
||||
"n": 1
|
||||
}
|
||||
},
|
||||
"edit-compose-pair": {
|
||||
"cpuoff-fp8": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 6.57,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 7.88,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 9.02,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 8.95,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 8.11,
|
||||
"min_db": 6.57,
|
||||
"max_db": 9.02,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"runs": {
|
||||
"cpuoff": {
|
||||
"flags": "--enable-cpu-offload",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 9,
|
||||
"vram_peak_mib": 15856
|
||||
},
|
||||
"cpuoff-layerwise": {
|
||||
"flags": "--enable-cpu-offload --enable-layerwise-offload",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 10,
|
||||
"vram_peak_mib": 15856
|
||||
},
|
||||
"default": {
|
||||
"flags": "flags:",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 12,
|
||||
"vram_peak_mib": 15926
|
||||
},
|
||||
"fp8": {
|
||||
"flags": "--enable-cpu-offload --quantization fp8",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 13,
|
||||
"vram_peak_mib": 15930
|
||||
},
|
||||
"fp8-expandable": {
|
||||
"flags": "--enable-cpu-offload --quantization fp8",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 9,
|
||||
"vram_peak_mib": 15876
|
||||
},
|
||||
"int8": {
|
||||
"flags": "--enable-cpu-offload --quantization int8",
|
||||
"serve_rc": 1,
|
||||
"serve_wall_s": 8,
|
||||
"vram_peak_mib": 15930
|
||||
}
|
||||
}
|
||||
}
|
||||
File renamed without changes.
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 2 probe: offline Qwen-Image-2.1 edit driver with prefix-KV extras.
|
||||
|
||||
The served path cannot reach `extras.prefix_kv_cache_dtype` on commit
|
||||
8484d85d (the structured-config bridge drops deploy-YAML extras), so this
|
||||
driver exercises the knob through the offline `Omni(...)` engine kwargs —
|
||||
the other route the recipe names. It mirrors image_edit.py's request
|
||||
construction: same prompt builder, same sampling fields, one request at a
|
||||
time.
|
||||
|
||||
Usage: offline_fp8v.py --out RUN_DIR [--kv-dtype fp8_v] [--fixture FID]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import time
|
||||
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
from vllm_omni.model_extras import (
|
||||
build_image_to_image_prompt,
|
||||
get_model_class_name,
|
||||
)
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
|
||||
SUITE = (
|
||||
pathlib.Path.home()
|
||||
/ ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
|
||||
)
|
||||
SNAP = (
|
||||
pathlib.Path.home()
|
||||
/ ".local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758"
|
||||
)
|
||||
STEPS = 20
|
||||
CFG = 1.0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--out", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--kv-dtype", default="fp8_v")
|
||||
parser.add_argument(
|
||||
"--quant",
|
||||
default="none",
|
||||
choices=["none", "fp8", "fp8-imgmlp", "fp8-dit", "fp8-encoder"],
|
||||
help="weight quantization: none, fp8 (all), fp8 minus img_mlp, transformer-only, or text-encoder-only",
|
||||
)
|
||||
parser.add_argument("--fixture", default="edit-compose-four")
|
||||
parser.add_argument("--seeds", type=lambda s: tuple(int(x) for x in s.split(",")), default=(2301,))
|
||||
parser.add_argument("--no-extras", action="store_true", help="run without the extras knob (control)")
|
||||
args = parser.parse_args()
|
||||
|
||||
fixture = json.loads((W / "fixtures.json").read_text())[args.fixture]
|
||||
refs = [Image.open(SUITE / f"{r}.png").convert("RGB") for r in fixture["references"]]
|
||||
|
||||
omni_kwargs = dict(
|
||||
model=str(SNAP),
|
||||
enforce_eager=True,
|
||||
enable_cpu_offload=True,
|
||||
init_timeout=1200,
|
||||
)
|
||||
if args.quant == "fp8":
|
||||
omni_kwargs["quantization"] = "fp8"
|
||||
elif args.quant == "fp8-imgmlp":
|
||||
omni_kwargs["quantization_config"] = {
|
||||
"method": "fp8",
|
||||
"ignored_layers": ["img_mlp"],
|
||||
}
|
||||
elif args.quant == "fp8-dit":
|
||||
omni_kwargs["quantization_config"] = {"transformer": {"method": "fp8"}}
|
||||
elif args.quant == "fp8-encoder":
|
||||
omni_kwargs["quantization_config"] = {"text_encoder": {"method": "fp8"}}
|
||||
if not args.no_extras:
|
||||
omni_kwargs["extras"] = {"prefix_kv_cache_dtype": args.kv_dtype}
|
||||
t0 = time.time()
|
||||
omni = Omni(**omni_kwargs)
|
||||
print(
|
||||
f"load_s={time.time() - t0:.1f} quant={args.quant} extras={omni_kwargs.get('extras')}",
|
||||
flush=True,
|
||||
)
|
||||
model_class_name = get_model_class_name(omni)
|
||||
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
records = []
|
||||
for seed in args.seeds:
|
||||
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(seed)
|
||||
prompt_dict = build_image_to_image_prompt(
|
||||
model_class_name=model_class_name,
|
||||
prompt=fixture["prompt"],
|
||||
negative_prompt=None,
|
||||
input_image=refs[0] if len(refs) == 1 else refs,
|
||||
height=fixture["height"],
|
||||
width=fixture["width"],
|
||||
)
|
||||
params = OmniDiffusionSamplingParams(
|
||||
generator=generator,
|
||||
seed=seed,
|
||||
true_cfg_scale=CFG,
|
||||
num_inference_steps=STEPS,
|
||||
height=fixture["height"],
|
||||
width=fixture["width"],
|
||||
)
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
t1 = time.time()
|
||||
rec = {"fixture_id": args.fixture, "seed": seed, "n_references": len(refs), "kv_dtype": args.kv_dtype, "quant": args.quant}
|
||||
try:
|
||||
outputs = omni.generate(prompt_dict, sampling_params_list=[params])
|
||||
wall = time.time() - t1
|
||||
images = None
|
||||
for output in outputs:
|
||||
images = getattr(output, "images", None)
|
||||
if images:
|
||||
break
|
||||
if not images:
|
||||
raise RuntimeError("no images in output")
|
||||
out_path = args.out / "measured" / f"{args.fixture}.s{seed}.png"
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
images[0].save(out_path)
|
||||
png = out_path.read_bytes()
|
||||
rec.update(
|
||||
wall_s=round(wall, 2),
|
||||
peak_gpu_mb=round(torch.cuda.max_memory_allocated() / 2**20, 1),
|
||||
bytes=len(png),
|
||||
sha256=hashlib.sha256(png).hexdigest(),
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - recorded as the probe result
|
||||
rec.update(wall_s=round(time.time() - t1, 2), error=f"{type(exc).__name__}: {exc}"[:2000])
|
||||
records.append(rec)
|
||||
print(json.dumps(rec), flush=True)
|
||||
|
||||
(args.out / f"offline-{args.fixture}.json").write_text(json.dumps(records, indent=1) + "\n")
|
||||
ok = all("sha256" in r for r in records)
|
||||
print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,23 +1,15 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 probe client: reference-conditioned requests against the int8
|
||||
weight-only compressed-tensors artifact, with the structure gate applied to
|
||||
every response.
|
||||
"""Issue 2 probe client: reference-conditioned requests through the API.
|
||||
|
||||
Matrix pass (the issue's capacity matrix): one-, two-, three-, and
|
||||
four-reference fixtures at seeds 2301-2304 — wall time and per-image
|
||||
structure metrics land in the record for every request. Every request
|
||||
mirrors the sealed Diffusers baseline attempt 2026-10-07T170729Z-q3-4r
|
||||
(same frozen references, same rendered prompt, 20 steps, true_cfg_scale
|
||||
1.0, fixture resolution, no negative prompt), submitted one at a time
|
||||
through POST /v1/images/edits.
|
||||
Functional pass: one-, two-, and four-reference fixtures at seed 2301.
|
||||
Measured pass: the four-reference fixture at seeds 2301-2304.
|
||||
|
||||
The structure gate (the lesson from issue 1's FP8 misread): a run only
|
||||
counts when the output is structurally sound — fully opaque RGBA and
|
||||
non-flat RGB. Broken FP8 output measured ~96.5-97% transparent with
|
||||
near-flat grey RGB; the BF16 path is 100% opaque. The gate numbers are
|
||||
recorded on every request regardless; `gate.py` summarizes them.
|
||||
Every request mirrors the sealed Diffusers baseline attempt
|
||||
2026-10-07T170729Z-q3-4r (same frozen references, same rendered prompt,
|
||||
20 steps, true_cfg_scale 1.0, fixture resolution, no negative prompt),
|
||||
submitted one at a time through POST /v1/images/edits.
|
||||
|
||||
Usage: probe.py --out RUN_DIR [--matrix] [--host H:P]
|
||||
Usage: probe.py --out RUN_DIR [--functional] [--measured] [--host H:P]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,42 +17,27 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import requests
|
||||
from PIL import Image
|
||||
|
||||
W = pathlib.Path.home() / "Projects/vllm-omni-val/issue5"
|
||||
W = pathlib.Path.home() / "Projects/vllm-omni-val/issue2"
|
||||
SUITE = (
|
||||
pathlib.Path.home()
|
||||
/ ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
|
||||
)
|
||||
STEPS = 20
|
||||
CFG = 1.0
|
||||
MATRIX_FIXTURES = (
|
||||
FUNCTIONAL_FIXTURES = (
|
||||
"edit-expression-ysolde", # 1 reference
|
||||
"edit-compose-pair", # 2 references
|
||||
"edit-compose-three", # 3 references (derived; see fixtures.json)
|
||||
"edit-compose-four", # 4 references
|
||||
)
|
||||
MATRIX_SEEDS = (2301, 2302, 2303, 2304)
|
||||
|
||||
|
||||
def structure_metrics(png: bytes) -> dict:
|
||||
"""Opacity + non-flatness numbers for the structure gate."""
|
||||
image = Image.open(io.BytesIO(png))
|
||||
array = np.asarray(image.convert("RGBA"))
|
||||
alpha = array[..., 3]
|
||||
rgb = array[..., :3].astype(np.float64)
|
||||
return {
|
||||
"opaque_fraction": round(float((alpha == 255).mean()), 4),
|
||||
"alpha_mean": round(float(alpha.mean()), 1),
|
||||
"rgb_std": [round(float(rgb[..., i].std()), 1) for i in range(3)],
|
||||
}
|
||||
MEASURED_FIXTURE = "edit-compose-four"
|
||||
MEASURED_SEEDS = (2301, 2302, 2303, 2304)
|
||||
|
||||
|
||||
def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
|
||||
@@ -94,7 +71,7 @@ def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
|
||||
t0 = time.time()
|
||||
try:
|
||||
response = requests.post(
|
||||
f"http://{host}/v1/images/edits", data=data, files=files, timeout=1800
|
||||
f"http://{host}/v1/images/edits", data=data, files=files, timeout=1200
|
||||
)
|
||||
rec["wall_s"] = round(time.time() - t0, 2)
|
||||
rec["http"] = response.status_code
|
||||
@@ -103,7 +80,6 @@ def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
|
||||
png = base64.b64decode(body["data"][0]["b64_json"])
|
||||
rec["bytes"] = len(png)
|
||||
rec["sha256"] = hashlib.sha256(png).hexdigest()
|
||||
rec["structure"] = structure_metrics(png)
|
||||
rec["usage"] = body.get("usage")
|
||||
for key in ("stage_metrics", "metrics", "timings"):
|
||||
if body.get(key):
|
||||
@@ -124,17 +100,18 @@ def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--out", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--host", default="127.0.0.1:8091")
|
||||
parser.add_argument("--matrix", action="store_true")
|
||||
parser.add_argument("--functional", action="store_true")
|
||||
parser.add_argument("--measured", action="store_true")
|
||||
parser.add_argument(
|
||||
"--fixture",
|
||||
default="edit-compose-four",
|
||||
help="single fixture id for --single",
|
||||
default=MEASURED_FIXTURE,
|
||||
help="fixture id for --measured (default: edit-compose-four)",
|
||||
)
|
||||
parser.add_argument("--single", action="store_true", help="one fixture, all seeds")
|
||||
parser.add_argument(
|
||||
"--seeds",
|
||||
type=lambda s: tuple(int(x) for x in s.split(",")),
|
||||
default=MATRIX_SEEDS,
|
||||
default=MEASURED_SEEDS,
|
||||
help="comma-separated seeds for --measured",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
@@ -142,20 +119,19 @@ def main() -> int:
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
plan: list[tuple[str, pathlib.Path, int]] = []
|
||||
if args.matrix:
|
||||
for fixture_id in MATRIX_FIXTURES:
|
||||
for seed in args.seeds:
|
||||
plan.append(
|
||||
(
|
||||
fixture_id,
|
||||
args.out / "matrix" / f"{fixture_id}.s{seed}.png",
|
||||
seed,
|
||||
)
|
||||
)
|
||||
if args.single:
|
||||
if args.functional:
|
||||
for fixture_id in FUNCTIONAL_FIXTURES:
|
||||
plan.append(
|
||||
(fixture_id, args.out / "functional" / f"{fixture_id}.png", 2301)
|
||||
)
|
||||
if args.measured:
|
||||
for seed in args.seeds:
|
||||
plan.append(
|
||||
(args.fixture, args.out / "matrix" / f"{args.fixture}.s{seed}.png", seed)
|
||||
(
|
||||
args.fixture,
|
||||
args.out / "measured" / f"{args.fixture}.s{seed}.png",
|
||||
seed,
|
||||
)
|
||||
)
|
||||
|
||||
records = []
|
||||
@@ -170,8 +146,15 @@ def main() -> int:
|
||||
elif rec.get("http") == 200:
|
||||
print(f"warning: 200 but no image for {fixture_id} seed {seed}", flush=True)
|
||||
|
||||
suffix = "matrix" if args.matrix else f"single-{args.fixture}"
|
||||
(args.out / f"{suffix}.json").write_text(json.dumps(records, indent=1) + "\n")
|
||||
suffix_parts = []
|
||||
if args.functional:
|
||||
suffix_parts.append("functional")
|
||||
if args.measured:
|
||||
suffix_parts.append(f"measured-{args.fixture}")
|
||||
suffix = "-".join(suffix_parts)
|
||||
(args.out / f"{suffix}.json").write_text(
|
||||
json.dumps(records, indent=1) + "\n"
|
||||
)
|
||||
ok = all(r.get("http") == 200 and "sha256" in r for r in records)
|
||||
print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
|
||||
return 0 if ok else 1
|
||||
@@ -1,14 +1,12 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 probe: PSNR of the int8 artifact's vLLM outputs against the
|
||||
sealed Diffusers baseline attempt 2026-10-07T170729Z-q3-4r.
|
||||
"""Issue 2 probe: PSNR of vLLM outputs against the sealed Diffusers
|
||||
four-reference baseline attempt 2026-10-07T170729Z-q3-4r.
|
||||
|
||||
Usage: psnr.py --fixture FIXTURE_ID RUN_DIR [RUN_DIR ...]
|
||||
|
||||
Compares each RUN_DIR/matrix/FIXTURE_ID.sSEED.png against the baseline
|
||||
Compares each RUN_DIR/measured/FIXTURE_ID.sSEED.png against the baseline
|
||||
sample with the same fixture and seed. RGBA outputs are composited over
|
||||
white, matching issue 1's comparator. The sealed baseline covers the 1-,
|
||||
2-, and 4-reference fixtures; the derived 3-reference fixture has no
|
||||
sealed row (gate.py + visual check cover it).
|
||||
white, matching issue 1's comparator.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -59,7 +57,7 @@ def main() -> int:
|
||||
for run_dir in args.runs:
|
||||
rows = []
|
||||
for seed, base_path in sorted(baseline.items()):
|
||||
ours = run_dir / "matrix" / f"{fixture}.s{seed}.png"
|
||||
ours = run_dir / "measured" / f"{fixture}.s{seed}.png"
|
||||
if not ours.exists():
|
||||
rows.append({"seed": seed, "error": "missing"})
|
||||
continue
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
# Issue 2 probe: serve Qwen-Image-2.1 on the RTX 3090 with one labeled
|
||||
# flag set. VRAM is polled once per second for the whole server lifetime,
|
||||
# matching issue 1's instrument (nvidia-smi csv series).
|
||||
#
|
||||
# Usage: serve.sh LABEL [extra vllm flags...]
|
||||
set -uo pipefail
|
||||
|
||||
BASE=$HOME/Projects/vllm-omni-val
|
||||
W=$BASE/issue2
|
||||
LABEL=$1; shift
|
||||
RUN=$W/runs/$LABEL
|
||||
mkdir -p "$RUN"
|
||||
|
||||
SNAP=$HOME/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758
|
||||
export HF_HOME=$HOME/.local/state/multi-model-helper-server/state/image-service/weights
|
||||
export HF_HUB_OFFLINE=1
|
||||
|
||||
( while sleep 1; do nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader; done ) > "$RUN/vram.log" 2>&1 &
|
||||
POLL=$!
|
||||
T0=$(date +%s)
|
||||
echo "flags: $*" > "$RUN/flags.txt"
|
||||
|
||||
$BASE/venv/bin/vllm serve "$SNAP" --omni --port 8091 \
|
||||
--served-model-name Qwen/Qwen-Image-2.1 \
|
||||
--enforce-eager "$@" > "$RUN/serve.log" 2>&1
|
||||
RC=$?
|
||||
kill $POLL 2>/dev/null
|
||||
echo "serve_rc=$RC wall=$(( $(date +%s) - T0 ))s" >> "$RUN/serve.log"
|
||||
@@ -1,246 +0,0 @@
|
||||
# Issue 5 probe: int8 quantized DiT serve paths on the RTX 3090 (24 GB)
|
||||
|
||||
## Round two (owner directive + amendment): W8A8 int8-dynamic — the capacity route is YES
|
||||
|
||||
The 08:37 amendment narrowed the remaining work to one decisive experiment:
|
||||
a compressed-tensors **W8A8 int8-dynamic** artifact (int8 weights + dynamic
|
||||
per-token symmetric int8 activations, int-quantized format) whose kernel on
|
||||
sm_86 is the cutlass int8 scaled-MM — no Marlin anywhere in the path.
|
||||
|
||||
**Result: the non-Marlin int8 path works end-to-end.** The artifact serves,
|
||||
passes the structure gate on all 16 matrix requests with healthy numbers,
|
||||
and lands at BF16-parity PSNR — including the first usable four-reference
|
||||
quality comparison this card has ever produced:
|
||||
|
||||
| Refs | Wall (mean, s) | Peak VRAM | Opacity / alpha | PSNR vs sealed Diffusers (dB) |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 | 20.5 | 23,879 MiB (run max) | 0.984–0.986 / 255.0 | 33.72 |
|
||||
| 2 | 25.1 | ” | 0.947–0.966 / 254.9–255.0 | 28.34 |
|
||||
| 3 | 29.4 | ” | 0.965–0.988 / 255.0 | (no sealed row; derived fixture) |
|
||||
| 4 | 34.0 | ” | 0.967–0.976 / 255.0 | **27.50** |
|
||||
|
||||
Comparison rows: BF16 served 1–2 refs at 34.2/36.7 s and OOMs at 4;
|
||||
online-FP8 and int8-wo-Marlin run 29.9–46.4 s but produce collapsed output;
|
||||
**W8A8-dynamic is both the fastest row measured (4-ref 34.0 s ≈ 4.3× the
|
||||
sealed Diffusers nocache baseline's 145.8 s) and the only structurally sound
|
||||
quantized row.** 3–4 references at cached speed on one 3090: delivered.
|
||||
|
||||
Gate calibration note: round one's pass rule (`opaque ≥ 0.99`) was an
|
||||
instrument calibration derived from the broken (0.008–0.20) vs BF16 (1.0)
|
||||
contrast. The W8A8 outputs sit at 0.95–0.99 opaque with alpha mean ≈255 and
|
||||
healthy RGB spread — soft-edge alpha, not the broken signature — so the
|
||||
recorded verdict rests on the raw numbers, visual inspection (four-character
|
||||
apothecary scene and the 1-ref portrait both render faithfully), and PSNR in
|
||||
the 26–29 dB BF16 band. The raw per-request structure metrics are in
|
||||
`evidence/rtx3090-round2/` unedited.
|
||||
|
||||
Kernel-path verification for this round (`w8a8_isolate.py`,
|
||||
`cutlass_ops_probe.py`): the raw ops (`scaled_int8_quant` +
|
||||
`cutlass_scaled_mm`) match a float32 reference at 0.17% rel-L2 across
|
||||
(64,256,128) / (512,4096,4096) / (512,4096,12288), and the full layer path
|
||||
(load → transpose-repack → apply, fused QKV and row-parallel, synthetic
|
||||
weights, loaded bitwise-verified) matches at 0.17% too. (Round-two lesson
|
||||
recorded for the instrument: an intermediate 97%-error scare in
|
||||
`w8a8_isolate.py` was a missing ×act_scale factor in the probe's own
|
||||
reference, not a kernel defect — the out/ref norm ratio equaled E[act_scale]
|
||||
exactly, which is what gave it away.)
|
||||
|
||||
Artifacts (outside Git): `…/artifacts/qwen-image-2.1-dit-w8a8dyn/` (int8
|
||||
weights unpacked + fp32 per-channel scales, 224 tensors, 7.26 GiB, manifest
|
||||
with digests). The round-one int8-wo artifact stays for the Marlin evidence.
|
||||
|
||||
The upstream Marlin report is drafted and ready to file in the #3 lane on
|
||||
the owner's call: `UPSTREAM-REPORT-DRAFT.md` (capability-query overclaim +
|
||||
NaN kernel + synthetic repro + the PR #4 FP8 convergence). The hard-stop
|
||||
branch did not trigger.
|
||||
|
||||
Helper-system follow-through for multi-model-helper-server #27 (input only):
|
||||
the 3090's serving-profile candidacy now extends to 3–4-reference profiles
|
||||
via the W8A8-dynamic artifact — 4-ref at 34.0 s/image, peak 23.9 GiB,
|
||||
BF16-parity PSNR (27.5 dB 4-ref vs the sealed int8-DiT Diffusers baseline),
|
||||
provided serving ships the artifact and the branch's two loader/factory
|
||||
fixes. This supersedes round one's "blocked on Marlin" conclusion.
|
||||
|
||||
## Round one: int8 weight-only via compressed-tensors — capacity yes, Marlin W8A16 collapses
|
||||
|
||||
Probed 2026-10-08 on `cachyos-alexander` (RTX 3090, sm_86, driver 615.71.09,
|
||||
CUDA 13.4), at upstream main `8484d85d1` plus this branch's two loader/factory
|
||||
fixes, vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||||
transformers 5.14.1, diffusers git `122b1e11`, compressed-tensors 0.17.0).
|
||||
The DiT was quantized **offline** from the sealed snapshot
|
||||
`Qwen/Qwen-Image-2.1@790c9263` to int8 weight-only (W8A16), per-output-channel
|
||||
symmetric, stored as a compressed-tensors pack-quantized checkpoint; encoder
|
||||
and VAE stay BF16 on disk via the artifact's per-component config wrapper.
|
||||
The probe mirrors the sealed Diffusers attempt `2026-10-07T170729Z-q3-4r`
|
||||
fixture-for-fixture (same frozen references-suite-2 images, same rendered
|
||||
prompts, seeds 2301–2304, 20 steps, `true_cfg_scale 1.0`).
|
||||
|
||||
## Verdict (superseded by round two above — kept as the round-one record)
|
||||
|
||||
**Capacity: delivered. Quality: collapsed in the serve kernel — the int8
|
||||
sidestep does not rescue 3–4 references on this card with this stack, and the
|
||||
blocker is now sharply isolated: the 8-bit Marlin W8A16 path itself.**
|
||||
|
||||
- Every step of the artifact chain is verified correct: the quantizer's
|
||||
round-trip reproduces the sealed BF16 weights at ~1.1% rel-L2
|
||||
(`evidence/rtx3090/artifact-verify.txt`), and the serve path loads the
|
||||
packed tensors **bitwise-identically** — including the fused q/k/v stacked
|
||||
shards — with finite scales before and after the Marlin repack.
|
||||
- Serving works mechanically: init ~40 s, all 16 capacity-matrix requests
|
||||
(1/2/3/4 references × seeds 2301–2304) complete at BF16-comparable speed
|
||||
(4-ref 46.4 s; BF16 OOMs at 4 refs, online-FP8 was 44.3 s), peak 23,935 MiB.
|
||||
- But the output is structurally broken on **every** counted run (gate FAIL
|
||||
16/16: opaque 0.0–0.20 vs 1.0 at BF16; PSNR vs the sealed baseline
|
||||
7.1–7.5 dB — the same collapsed band as online FP8). The images are
|
||||
horizontal-stripe noise.
|
||||
- A one-layer offline isolation (`marlin_isolate.py`) pins it: vLLM parallel
|
||||
linears built exactly as the qwen_image_21 transformer builds them, loaded
|
||||
with the artifact's verified tensors, emit **NaN** for all four layer
|
||||
geometries (fused QKV, to_out, img_mlp proj/out), for both bf16 and fp16
|
||||
activations — and do the same for **synthetic random int8 weights**, so the
|
||||
checkpoint and its quantization are exonerated. The 8-bit Marlin W8A16 path
|
||||
on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) is broken even though
|
||||
`query_marlin_supported_quant_types` accepts `uint8b128` on the card, and
|
||||
`choose_mp_linear_kernel` offers no non-Marlin W8A16 fallback on sm_86
|
||||
(Machete is sm90+, ExLlama int4-only).
|
||||
|
||||
The torchao-int8-in-Diffusers prior (multi-model-helper-server #160, blinded
|
||||
review within label noise of BF16) stands unchallenged: int8 quantization is
|
||||
fine; the serve-path kernel is not. The online-FP8 collapse from PR #4 and
|
||||
this int8 collapse now share one root suspect — the Marlin mixed-precision
|
||||
path on this fork/card — with the fp8 case previously isolated to the
|
||||
Marlin-quantized DiT and this run extending it to int8 with a fully verified
|
||||
checkpoint on the far side of the kernel.
|
||||
|
||||
## Two serve-path fixes this issue required (in this PR)
|
||||
|
||||
1. **Quant factory could not build compressed-tensors at all.**
|
||||
`_normalize_method_name` canonicalizes `compressed-tensors` to
|
||||
`compressed_tensors`, which then missed vLLM's hyphenated registry — so
|
||||
both `--quantization compressed-tensors` and checkpoint auto-detection
|
||||
raised "Unknown quantization method". The registry lookup now resolves
|
||||
both spellings, and registry configs whose `__init__` does not take the
|
||||
serialized schema fall back to `from_config` (`vllm_omni/quantization/factory.py`,
|
||||
tests in `tests/quantization/test_quant_factory.py`).
|
||||
2. **The diffusion loader misclassified pre-quantized checkpoints as online
|
||||
quantization under CPU offload.** With `--enable-cpu-offload` it loaded
|
||||
every component on the GPU before offloading (encoder 17 GiB + VAE + the
|
||||
packed DiT skeleton > 24 GiB → init OOM). Serialized formats are now
|
||||
detected (`is_prequantized_config`, component-wrapper aware) and load
|
||||
directly on the host, with each quantized module round-tripped through the
|
||||
accelerator only for the post-load Marlin repack
|
||||
(`vllm_omni/diffusion/model_loader/diffusers_loader.py`). With these two
|
||||
fixes the artifact serves: init healthy in ~40 s, logs
|
||||
`Per-component quantization: {'transformer': 'compressed-tensors'}`.
|
||||
|
||||
## Artifact (outside Git, under the helper state root)
|
||||
|
||||
`~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo/`
|
||||
— quantized `transformer/` (6.76 GiB packed int8 + BF16 boundary layers;
|
||||
224 tensors = 32 blocks × {to_q,to_k,to_v,to_out.0,proj,gate_layer,out};
|
||||
6.98 B params quantized, 0.14 B BF16), `model_index.json`/`scheduler`/
|
||||
`text_encoder`/`vae`/`processor` symlinked to the sealed snapshot, and
|
||||
`MANIFEST.json` with inputs (snapshot path, revision, tree digest), versions,
|
||||
per-file sha256, and quantization stats (mean rel-L2 1.10%, max 1.73%).
|
||||
Built by `quantize_dit_int8wo.py` using compressed-tensors' own
|
||||
`pack_to_int32` (uint8 biased by 128, 4 per int32) and `QuantizationConfig`
|
||||
serialization, so the storage convention matches llm-compressor bit-for-bit.
|
||||
`transformer/config.json` carries the quantization metadata wrapped
|
||||
per-component (`{"transformer": {...}}`) so auto-detection quantizes the DiT
|
||||
only — the encoder and VAE resolve to None and stay BF16
|
||||
(the plain global form would hand the encoder a compressed-tensors config
|
||||
it has no scales for).
|
||||
|
||||
## Run matrix — RTX 3090
|
||||
|
||||
Single run `int8wo`: `vllm serve <artifact> --omni --port 8091
|
||||
--served-model-name Qwen/Qwen-Image-2.1 --enforce-eager
|
||||
--enable-cpu-offload`, readiness via `/health`, VRAM via the 1 Hz
|
||||
`nvidia-smi` series (issue 2's instrument). Two precursor runs OOM'd at init
|
||||
for reasons fixed on this branch (see Verdict item 2); their logs survive
|
||||
under `runs/int8wo-expandable/` on the host (the first attempt's log was
|
||||
overwritten by the successful rerun in the same label).
|
||||
|
||||
| Refs | Fixture | Wall (mean, s) | Peak (MiB) | Gate | PSNR vs Diffusers (dB) |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | edit-expression-ysolde | 31.0 | 23,935 (run max) | FAIL 4/4 | 7.14 |
|
||||
| 2 | edit-compose-pair | 36.6 | ” | FAIL 4/4 | 7.46 |
|
||||
| 3 | edit-compose-three | 41.4 | ” | FAIL 4/4 | no sealed row (derived fixture) |
|
||||
| 4 | edit-compose-four | 46.4 | ” | FAIL 4/4 | 7.11 |
|
||||
|
||||
BF16 comparison rows (issue 2): 1-ref 34.2 s, 2-ref 36.7 s, 4-ref OOM.
|
||||
Online-fp8 (broken output, capacity reference): 29.9 / 34.1 / 38.9 / 44.3 s.
|
||||
So the int8 artifact matches the fp8 capacity/speed profile exactly —
|
||||
including its broken-output signature (PSNR band 7.0–7.5 dB) — while the
|
||||
checkpoint side is proven numerically sound, moving the defect wholly into
|
||||
the kernel.
|
||||
|
||||
## Structure gate and quality read
|
||||
|
||||
- Gate rule (issue 1's lesson): `opaque_fraction ≥ 0.99` and `mean(rgb_std)
|
||||
≥ 10`; BF16 outputs measure 1.0 opaque, 26–29 dB. Every int8-artifact run
|
||||
fails (opaque 0.0–0.20, alpha mean 114–137; rgb_std spans 3–93 — several
|
||||
outputs have healthy RGB spread yet near-total transparency, and one
|
||||
(seed 2302) is near-flat grey like the fp8 signature). Outputs are
|
||||
horizontal-stripe noise on visual inspection.
|
||||
- PSNR vs the sealed 4-reference Diffusers baseline (RGBA-on-RGBA,
|
||||
#1's comparator): 7.14 / 7.46 / 7.11 dB mean for 1/2/4 refs — the
|
||||
collapsed band, not the 26–29 dB BF16-row parity the issue hoped for.
|
||||
|
||||
## Kernel isolation (the issue's "different kernels" prior, answered)
|
||||
|
||||
`marlin_isolate.py` builds vLLM `QKVParallelLinear` / `RowParallelLinear` /
|
||||
`ColumnParallelLinear` exactly as `qwen_image_21_transformer.py` does, loads
|
||||
the artifact tensors through the production weight-loader machinery
|
||||
(bitwise-verified after load, fused q/k/v shards included), runs
|
||||
`process_weights_after_loading` (the Marlin repack) on the GPU, and compares
|
||||
`layer(x)` against `float32 x @ dequant(W)ᵀ` computed from the artifact's own
|
||||
unpacked weights:
|
||||
|
||||
| Layer | bf16 acts | fp16 acts | packed bitwise | scales post-repack |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| attn.to_qkv (fused) | NaN | NaN | equal | finite (6.6e-4…1.2e-2) |
|
||||
| attn.to_out.0 (row) | NaN | NaN | equal | finite |
|
||||
| img_mlp.proj (column) | NaN | NaN | equal | finite |
|
||||
| img_mlp.out (row, wide-in) | NaN | NaN | equal | finite |
|
||||
|
||||
Synthetic random int8 weights reproduce the NaN identically, and the runtime
|
||||
capability query accepts `uint8b128` on this card — so this is a defect of
|
||||
the 8-bit Marlin W8A16 path on this stack (repack or kernel; sm_86,
|
||||
vllm 0.31.0, torch 2.13.0+cu130), not of the checkpoint, the quantizer, the
|
||||
loader, or the fused-QKV stacking. No alternative W8A16 kernel exists on
|
||||
sm_86 in this build to switch to.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
Scripts in this directory: `quantize_dit_int8wo.py` (artifact + manifest),
|
||||
`verify_artifact.py` (round-trip check), `serve.sh` / `probe.py` (adapted
|
||||
issue-2 kit; probe records the structure gate inline per request), `gate.py`
|
||||
(gate + capacity summarizer), `psnr.py`, `marlin_isolate.py` (kernel
|
||||
isolation, `--synthetic` for the stack-only variant). All were pushed
|
||||
verbatim to `~/Projects/vllm-omni-val/issue5/` on the host and run from the
|
||||
validation clone checked out on this branch. Per-run serve/vram logs and all
|
||||
output PNGs live under `~/Projects/vllm-omni-val/issue5/runs/int8wo/`;
|
||||
condensed evidence in `evidence/rtx3090/`.
|
||||
|
||||
## Helper-system follow-through (input to multi-model-helper-server #27 — not implemented here)
|
||||
|
||||
- The 3090's 3–4 reference serving path remains blocked, now with a single
|
||||
named suspect: the 8-bit Marlin W8A16 kernel path on sm_86 (this fork's
|
||||
vllm 0.31.0 / torch 2.13.0 stack). Both quantized routes (online FP8,
|
||||
offline int8 compressed-tensors with a verified checkpoint) collapse at the
|
||||
same place. The unlock order for #27 candidacy is now: upstream Marlin
|
||||
8-bit fix (or a non-Marlin W8A16 kernel on sm_86), then serving — capacity
|
||||
and speed are already proven sufficient (4-ref at 46.4 s, 2.9× the sealed
|
||||
Diffusers nocache baseline).
|
||||
- The two vllm-omni loader/factory fixes on this branch are prerequisites for
|
||||
any compressed-tensors artifact serving under CPU offload and are
|
||||
independent of the kernel defect.
|
||||
|
||||
## Out-of-scope confirmation
|
||||
|
||||
No upstream PRs; no W8A8/int4/NVFP4 artifacts built; no helper-system
|
||||
deployment evidence. Root-causing the Marlin 8-bit defect remains the
|
||||
separate lane (PR #4 carries the fp8 isolation; this branch adds the int8 +
|
||||
synthetic + bitwise-load isolation). The only code changes are the two
|
||||
serve-path fixes required to make the issue's plan steps executable at all.
|
||||
@@ -1,71 +0,0 @@
|
||||
# Upstream report draft: Marlin 8-bit W8A16 yields NaN on sm_86 while the capability query accepts it
|
||||
|
||||
Status: DRAFT — prepared for the #3 upstream lane; file when the owner calls it.
|
||||
Evidence: issue 5 round one (this directory) + PR #4's online-FP8 isolation.
|
||||
Reproduction: `probe/issue-5/marlin_isolate.py --synthetic` (no checkpoint needed).
|
||||
|
||||
## Summary
|
||||
|
||||
On vLLM 0.31.0 (torch 2.13.0+cu130, CUDA 13.4, RTX 3090 / sm_86, driver
|
||||
615.71.09), the 8-bit Marlin W8A16 path selected for compressed-tensors
|
||||
int8 weight-only checkpoints produces NaN outputs for every tested layer
|
||||
geometry, activation dtype, and input distribution — including synthetic
|
||||
random weights — even though `query_marlin_supported_quant_types` accepts
|
||||
`uint8b128` on the device and `choose_mp_linear_kernel` has no other W8A16
|
||||
candidate on sm_86 (Machete is sm90+, ExLlama int4-only).
|
||||
|
||||
## Isolation chain (each step independently verified)
|
||||
|
||||
1. **Checkpoint exonerated.** The int8 weight-only compressed-tensors
|
||||
artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 using
|
||||
compressed-tensors' own `unpack_from_int32` + dequantize
|
||||
(`verify_artifact.py`, output in `evidence/rtx3090/artifact-verify.txt`).
|
||||
2. **Loading exonerated.** `marlin_isolate.py` loads the artifact's
|
||||
`weight_packed` tensors through the production vLLM weight-loader
|
||||
machinery into `QKVParallelLinear` / `RowParallelLinear` /
|
||||
`ColumnParallelLinear` built exactly as the qwen_image_21 transformer
|
||||
builds them; after loading, the parameters are **bitwise-equal** to the
|
||||
checkpoint (including the fused q/k/v stacked shards), and scales are
|
||||
finite before and after `process_weights_after_loading`.
|
||||
3. **Kernel indicted.** `layer(x)` is NaN for all four layer geometries
|
||||
(fused 12288-out QKV, 4096-out row, 12288-out column, wide-in row), for
|
||||
both bf16 and fp16 activations, at M=512.
|
||||
4. **Checkpoint-independence.** `--synthetic` (random int8 weights + random
|
||||
per-channel scales, quantized and packed in-memory) reproduces the NaN
|
||||
identically — no checkpoint, quantizer, or scale file involved.
|
||||
5. **Same-family serve evidence.** PR #4's online-FP8 path (also Marlin
|
||||
W8A16 on sm_86) produces structurally broken output (≈97% transparent
|
||||
RGBA, PSNR 7 dB vs BF16) across every reference count; transformer-only
|
||||
FP8 isolates the collapse to the Marlin-quantized DiT. Two independent
|
||||
quantizers, one kernel family.
|
||||
|
||||
## Suspicion boundary (what is and is not pinned)
|
||||
|
||||
The NaN arises between `gptq_marlin_repack`/`marlin_permute_scales` and
|
||||
`torch.ops.vllm.marlin_gemm` — the isolate stops at that boundary. Not yet
|
||||
dissected: whether the repacked weight bytes are wrong (repack/layout bug
|
||||
for 8-bit on sm_86) or the kernel itself miscomputes from correct bytes.
|
||||
The 4-bit Marlin path was not tested (out of scope for issue 5).
|
||||
|
||||
## Minimal reproduction (from this repo)
|
||||
|
||||
```sh
|
||||
python probe/issue-5/marlin_isolate.py --synthetic # NaN, all geometries
|
||||
```
|
||||
|
||||
Requires: vLLM 0.31.0 importable with CUDA visible; ~2 GiB GPU memory.
|
||||
Expected on an unaffected stack: rel-L2 ≈ 0.01 per layer.
|
||||
Observed on the affected stack: NaN for every layer, both activation dtypes
|
||||
(`evidence/rtx3090/marlin-isolate-synthetic.json`).
|
||||
|
||||
## Context that may matter upstream
|
||||
|
||||
- torch 2.13.0+cu130 / CUDA 13.4 is a bleeding-edge combination; if CI
|
||||
covers Marlin int8 on sm_80/86 with an older stack, a torch/CUDA 13
|
||||
interaction is a candidate.
|
||||
- The 4-bit Marlin path is heavily exercised by CI; the 8-bit
|
||||
(`uint8b128`) path is comparatively new. If 8-bit Marlin support was
|
||||
validated only on sm_80/sm_90, sm_86 may be untested.
|
||||
- The capability query accepting `uint8b128` on sm_86 while the kernel
|
||||
yields NaN is the dispatch-overclaim angle: either the query or the
|
||||
kernel is wrong for this cell.
|
||||
@@ -1,67 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 round two: raw-op probe of the cutlass W8A8 int8 path on this stack.
|
||||
|
||||
Bypasses every layer/loader and tests exactly what CutlassInt8ScaledMMLinearKernel
|
||||
calls: ops.scaled_int8_quant (dynamic per-token symmetric) then
|
||||
ops.cutlass_scaled_mm, against a float32 reference. A failure here means the
|
||||
cutlass int8 kernels themselves are broken on this stack (CUDA 13.4 / sm_86 /
|
||||
torch 2.13.0+cu130 / vllm 0.31.0), independent of any checkpoint or loader.
|
||||
|
||||
Usage: cutlass_ops_probe.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import torch
|
||||
from vllm import _custom_ops as ops
|
||||
|
||||
|
||||
def rel(a: torch.Tensor, b: torch.Tensor) -> float:
|
||||
return float((a - b).norm() / b.norm())
|
||||
|
||||
|
||||
def main() -> int:
|
||||
torch.manual_seed(2301)
|
||||
device = torch.device("cuda")
|
||||
results = []
|
||||
|
||||
for m, k, n in ((64, 256, 128), (512, 4096, 4096), (512, 4096, 12288)):
|
||||
w = torch.randn(n, k, device=device)
|
||||
w_scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
w_q = torch.round(w / w_scale).clamp(-127, 127).to(torch.int8)
|
||||
|
||||
x = torch.randn(m, k, dtype=torch.bfloat16, device=device)
|
||||
x_q, x_s, _ = ops.scaled_int8_quant(x.contiguous(), None, None, symmetric=True)
|
||||
|
||||
# Reference: dequantized per-token acts @ dequantized per-channel weights.
|
||||
ref = (x_q.float() * x_s.float()) @ (w_q.float() * w_scale.float()).T
|
||||
|
||||
w_q_t = w_q.t()
|
||||
out = ops.cutlass_scaled_mm(
|
||||
x_q, w_q_t, scale_a=x_s, scale_b=w_scale, out_dtype=torch.bfloat16
|
||||
).float()
|
||||
|
||||
# Also check the quantizer op itself against a float32 reference.
|
||||
act_scale_ref = x.float().abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
x_q_ref = torch.round(x.float() / act_scale_ref).clamp(-127, 127).to(torch.int8)
|
||||
quant_match = bool(torch.equal(x_q, x_q_ref)) and bool(
|
||||
torch.allclose(x_s.float(), act_scale_ref, rtol=1e-3, atol=1e-7)
|
||||
)
|
||||
|
||||
results.append({
|
||||
"mk": [m, k, n],
|
||||
"quant_op_matches_reference": quant_match,
|
||||
"gemm_rel_l2": round(rel(out, ref), 5),
|
||||
"out_finite": bool(torch.isfinite(out).all()),
|
||||
"scale_a_shape": list(x_s.shape),
|
||||
"scale_b_shape": list(w_scale.shape),
|
||||
})
|
||||
|
||||
print(json.dumps(results, indent=1))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,56 +0,0 @@
|
||||
[
|
||||
{
|
||||
"mk": [
|
||||
64,
|
||||
256,
|
||||
128
|
||||
],
|
||||
"quant_op_matches_reference": false,
|
||||
"gemm_rel_l2": 0.00165,
|
||||
"out_finite": true,
|
||||
"scale_a_shape": [
|
||||
64,
|
||||
1
|
||||
],
|
||||
"scale_b_shape": [
|
||||
128,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"mk": [
|
||||
512,
|
||||
4096,
|
||||
4096
|
||||
],
|
||||
"quant_op_matches_reference": false,
|
||||
"gemm_rel_l2": 0.00166,
|
||||
"out_finite": true,
|
||||
"scale_a_shape": [
|
||||
512,
|
||||
1
|
||||
],
|
||||
"scale_b_shape": [
|
||||
4096,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"mk": [
|
||||
512,
|
||||
4096,
|
||||
12288
|
||||
],
|
||||
"quant_op_matches_reference": false,
|
||||
"gemm_rel_l2": 0.00166,
|
||||
"out_finite": true,
|
||||
"scale_a_shape": [
|
||||
512,
|
||||
1
|
||||
],
|
||||
"scale_b_shape": [
|
||||
12288,
|
||||
1
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,15 +0,0 @@
|
||||
[
|
||||
{
|
||||
"layer": "attn.to_qkv (fused q/k/v)",
|
||||
"rel_l2": 0.00171,
|
||||
"out_finite": true
|
||||
},
|
||||
{
|
||||
"layer": "attn.to_out.0 (row)",
|
||||
"rel_l2": 0.00172,
|
||||
"out_finite": true,
|
||||
"loaded_bitwise_ok": true,
|
||||
"loaded_scale_ok": true,
|
||||
"rel_l2_vs_layer_weight": 0.00172
|
||||
}
|
||||
]
|
||||
@@ -1,118 +0,0 @@
|
||||
== int8w8a8dyn ==
|
||||
1-ref: n=4 gate=FAIL wall 20.4-20.6s (mean 20.5s)
|
||||
edit-expression-ysolde s2301: 20.5s {"opaque_fraction": 0.9847, "alpha_mean": 255.0, "rgb_std": [45.6, 44.3, 47.2]} BROKEN
|
||||
edit-expression-ysolde s2302: 20.43s {"opaque_fraction": 0.9861, "alpha_mean": 255.0, "rgb_std": [47.0, 45.3, 48.5]} BROKEN
|
||||
edit-expression-ysolde s2303: 20.52s {"opaque_fraction": 0.9844, "alpha_mean": 255.0, "rgb_std": [46.5, 45.2, 48.1]} BROKEN
|
||||
edit-expression-ysolde s2304: 20.56s {"opaque_fraction": 0.9854, "alpha_mean": 255.0, "rgb_std": [46.9, 44.8, 47.7]} BROKEN
|
||||
2-ref: n=4 gate=FAIL wall 25.0-25.2s (mean 25.1s)
|
||||
edit-compose-pair s2301: 25.02s {"opaque_fraction": 0.9638, "alpha_mean": 255.0, "rgb_std": [69.4, 70.2, 71.8]} BROKEN
|
||||
edit-compose-pair s2302: 25.11s {"opaque_fraction": 0.9659, "alpha_mean": 255.0, "rgb_std": [61.9, 63.6, 65.7]} BROKEN
|
||||
edit-compose-pair s2303: 25.16s {"opaque_fraction": 0.9572, "alpha_mean": 255.0, "rgb_std": [61.6, 63.2, 64.9]} BROKEN
|
||||
edit-compose-pair s2304: 25.17s {"opaque_fraction": 0.9471, "alpha_mean": 254.9, "rgb_std": [57.5, 55.8, 55.6]} BROKEN
|
||||
3-ref: n=4 gate=FAIL wall 29.4-29.5s (mean 29.4s)
|
||||
edit-compose-three s2301: 29.37s {"opaque_fraction": 0.9785, "alpha_mean": 255.0, "rgb_std": [42.9, 32.4, 23.1]} BROKEN
|
||||
edit-compose-three s2302: 29.44s {"opaque_fraction": 0.9882, "alpha_mean": 255.0, "rgb_std": [37.9, 30.5, 21.6]} BROKEN
|
||||
edit-compose-three s2303: 29.47s {"opaque_fraction": 0.9757, "alpha_mean": 255.0, "rgb_std": [48.5, 40.2, 30.1]} BROKEN
|
||||
edit-compose-three s2304: 29.49s {"opaque_fraction": 0.9645, "alpha_mean": 255.0, "rgb_std": [48.5, 35.7, 25.1]} BROKEN
|
||||
4-ref: n=4 gate=FAIL wall 34.0-34.1s (mean 34.0s)
|
||||
edit-compose-four s2301: 34.08s {"opaque_fraction": 0.976, "alpha_mean": 255.0, "rgb_std": [49.0, 37.3, 29.3]} BROKEN
|
||||
edit-compose-four s2302: 34.0s {"opaque_fraction": 0.9713, "alpha_mean": 255.0, "rgb_std": [47.2, 39.5, 29.8]} BROKEN
|
||||
edit-compose-four s2303: 34.02s {"opaque_fraction": 0.9706, "alpha_mean": 255.0, "rgb_std": [56.4, 44.8, 35.4]} BROKEN
|
||||
edit-compose-four s2304: 34.04s {"opaque_fraction": 0.9674, "alpha_mean": 255.0, "rgb_std": [50.9, 38.3, 29.5]} BROKEN
|
||||
nvidia-smi peak: 23879 MiB; server peak: None MB
|
||||
STRUCTURE GATE: FAIL
|
||||
|
||||
== psnr edit-expression-ysolde ==
|
||||
{
|
||||
"int8w8a8dyn": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 33.45,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 34.47,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 34.68,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 32.26,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 33.72,
|
||||
"min_db": 32.26,
|
||||
"max_db": 34.68,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-pair ==
|
||||
{
|
||||
"int8w8a8dyn": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 31.94,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 27.97,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 23.42,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 30.05,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 28.34,
|
||||
"min_db": 23.42,
|
||||
"max_db": 31.94,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-four ==
|
||||
{
|
||||
"int8w8a8dyn": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 28.01,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 25.7,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 29.82,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 26.48,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 27.5,
|
||||
"min_db": 25.7,
|
||||
"max_db": 29.82,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
23879 MiB, 100 %
|
||||
@@ -1,102 +0,0 @@
|
||||
{
|
||||
"schema": "issue5-w8a8dyn-v1",
|
||||
"created_utc": "2026-10-08T12:52:49+00:00",
|
||||
"description": "Qwen-Image-2.1 DiT w8a8dyn W8A8 int8-dynamic, int-quantized, compressed-tensors",
|
||||
"inputs": {
|
||||
"snapshot": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"revision": "790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"tree_manifest": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/trees/790c92633540aa0cb11d9abf19eb46d861714758.json",
|
||||
"tree_sha256": "1f35d0a5a99f2c79bf2a6818b96582828094feb8544a992bc81c1d4100aed436"
|
||||
},
|
||||
"versions": {
|
||||
"python": "3.13.15",
|
||||
"torch": "2.13.0+cu130",
|
||||
"compressed_tensors": "0.17.0"
|
||||
},
|
||||
"quantization": {
|
||||
"method": "compressed-tensors",
|
||||
"format": "pack-quantized",
|
||||
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||
"config": {
|
||||
"config_groups": {
|
||||
"group_0": {
|
||||
"targets": [
|
||||
"Linear"
|
||||
],
|
||||
"weights": {
|
||||
"num_bits": 8,
|
||||
"type": "int",
|
||||
"symmetric": true,
|
||||
"group_size": null,
|
||||
"strategy": "channel",
|
||||
"block_structure": null,
|
||||
"dynamic": false,
|
||||
"actorder": null,
|
||||
"scale_dtype": null,
|
||||
"zp_dtype": null,
|
||||
"observer": "minmax",
|
||||
"observer_kwargs": {}
|
||||
},
|
||||
"input_activations": {
|
||||
"num_bits": 8,
|
||||
"type": "int",
|
||||
"symmetric": true,
|
||||
"group_size": null,
|
||||
"strategy": "token",
|
||||
"block_structure": null,
|
||||
"dynamic": true,
|
||||
"actorder": null,
|
||||
"scale_dtype": null,
|
||||
"zp_dtype": null,
|
||||
"observer": null,
|
||||
"observer_kwargs": {}
|
||||
},
|
||||
"output_activations": null,
|
||||
"format": null
|
||||
}
|
||||
},
|
||||
"quant_method": "compressed-tensors",
|
||||
"kv_cache_scheme": null,
|
||||
"format": "int-quantized",
|
||||
"quantization_status": "frozen",
|
||||
"global_compression_ratio": null,
|
||||
"ignore": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
]
|
||||
},
|
||||
"ignored_layers": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
],
|
||||
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers"
|
||||
},
|
||||
"stats": {
|
||||
"quantized_tensors": 224,
|
||||
"quantized_params": 6979321856,
|
||||
"bf16_params": 135802880,
|
||||
"mean_rel_l2": 0.010959165191577216,
|
||||
"max_rel_l2": 0.017284497618675232,
|
||||
"weights_file_bytes": 7256754928
|
||||
},
|
||||
"files": [
|
||||
{
|
||||
"path": "transformer/config.json",
|
||||
"bytes": 1786,
|
||||
"sha256": "321512e271d79f729790890a64773464baafe7e7c702fc10442de89f79a7d163"
|
||||
},
|
||||
{
|
||||
"path": "transformer/diffusion_pytorch_model.safetensors",
|
||||
"bytes": 7256754928,
|
||||
"sha256": "ab2814308c55eb95c648aad94d47f59c28386113677767d791fea72a4506ce5c"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,978 +0,0 @@
|
||||
[
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2301,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 20.5,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "31af175ab74c8a86746d7a69df771fd62552def574cc08b2a673220c69f302da",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9847,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
45.6,
|
||||
44.3,
|
||||
47.2
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 20433.584690093994,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1021.6792345046997,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 20433.70294570923,
|
||||
"image_time_to_first_output_ms": 20433.545112609863,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.12007599434582517,
|
||||
"stage_0_gen_ms": 20433.584690093994
|
||||
},
|
||||
"peak_memory_mb": 21550.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2302,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 20.43,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "9f24d8b8ff78fd10c6f17c036a69b86574023fab13f772d3999497341016d321",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9861,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
47.0,
|
||||
45.3,
|
||||
48.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 20371.083736419678,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1018.5541868209839,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 20371.186017990112,
|
||||
"image_time_to_first_output_ms": 20371.060371398926,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.09984300413634628,
|
||||
"stage_0_gen_ms": 20371.083736419678
|
||||
},
|
||||
"peak_memory_mb": 21550.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2303,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 20.52,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "cf9ec7928fa22375c476640a9920b2e05ef439200ada14b3d073bce896d5ea64",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9844,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
46.5,
|
||||
45.2,
|
||||
48.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 20455.31988143921,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1022.7659940719604,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 20455.437421798706,
|
||||
"image_time_to_first_output_ms": 20455.294847488403,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11576600081752986,
|
||||
"stage_0_gen_ms": 20455.31988143921
|
||||
},
|
||||
"peak_memory_mb": 21550.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2304,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 20.56,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "288b63f713ac0664b7e0035e5f8ada494eeeb413052a35d8f3324035cb4b2bc7",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9854,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
46.9,
|
||||
44.8,
|
||||
47.7
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 20500.65541267395,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1025.0327706336975,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 20500.685691833496,
|
||||
"image_time_to_first_output_ms": 20500.566005706787,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.09380799747304991,
|
||||
"stage_0_gen_ms": 20500.65541267395
|
||||
},
|
||||
"peak_memory_mb": 21550.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2301,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 25.02,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "3df72b7d1334454fddc1768c6b6ff172a86cdb3ab8b075c3ff784260260bd1d1",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9638,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
69.4,
|
||||
70.2,
|
||||
71.8
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 24954.949617385864,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1247.7474808692932,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 24954.51283454895,
|
||||
"image_time_to_first_output_ms": 24954.333305358887,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15277299826266244,
|
||||
"stage_0_gen_ms": 24954.949617385864
|
||||
},
|
||||
"peak_memory_mb": 22342.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2302,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 25.11,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "151ddf53141622414678c27efeafe7b928297a4cfc29038e41607069ca6968f6",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9659,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
61.9,
|
||||
63.6,
|
||||
65.7
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 25045.337438583374,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1252.2668719291687,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 25045.44496536255,
|
||||
"image_time_to_first_output_ms": 25045.308351516724,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11107399768661708,
|
||||
"stage_0_gen_ms": 25045.337438583374
|
||||
},
|
||||
"peak_memory_mb": 22280.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2303,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 25.16,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "023407fe5dd8530aa1c640c3949b1b8da0893a15f07ae061a7ece4c132a7931b",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9572,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
61.6,
|
||||
63.2,
|
||||
64.9
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 25092.447519302368,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1254.6223759651184,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 25092.57960319519,
|
||||
"image_time_to_first_output_ms": 25092.416524887085,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.13619099627248943,
|
||||
"stage_0_gen_ms": 25092.447519302368
|
||||
},
|
||||
"peak_memory_mb": 22280.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2304,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 25.17,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "7e46495775ee73c2ccc21e252c351311c7e0976a59ef0d9483b14cfcc0fa27f7",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9471,
|
||||
"alpha_mean": 254.9,
|
||||
"rgb_std": [
|
||||
57.5,
|
||||
55.8,
|
||||
55.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 25105.11350631714,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1255.255675315857,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 25105.209827423096,
|
||||
"image_time_to_first_output_ms": 25105.08370399475,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10113100142916664,
|
||||
"stage_0_gen_ms": 25105.11350631714
|
||||
},
|
||||
"peak_memory_mb": 22280.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2301,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 29.37,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "1b3430e99b76ed0483c37fbdbacdb3a890e08af69a5a9c77388cb1665c54fa19",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9785,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
42.9,
|
||||
32.4,
|
||||
23.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 29299.254417419434,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1464.9627208709717,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 29299.4122505188,
|
||||
"image_time_to_first_output_ms": 29299.22580718994,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1593470005900599,
|
||||
"stage_0_gen_ms": 29299.254417419434
|
||||
},
|
||||
"peak_memory_mb": 23432.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2302,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 29.44,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "67ed3ba31c5db970ddda42dc38e5f75e395d83aae1c467625060a5bc1034906d",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9882,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
37.9,
|
||||
30.5,
|
||||
21.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 29367.507219314575,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1468.3753609657288,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 29367.653369903564,
|
||||
"image_time_to_first_output_ms": 29367.475748062134,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15194500156212598,
|
||||
"stage_0_gen_ms": 29367.507219314575
|
||||
},
|
||||
"peak_memory_mb": 23432.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2303,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 29.47,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "ed8493e310da3d892eeba849519fe8697e638c215eb01bccad96608b9040e70b",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9757,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
48.5,
|
||||
40.2,
|
||||
30.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 29401.696920394897,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1470.0848460197449,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 29401.797771453857,
|
||||
"image_time_to_first_output_ms": 29401.665925979614,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10474099690327421,
|
||||
"stage_0_gen_ms": 29401.696920394897
|
||||
},
|
||||
"peak_memory_mb": 23432.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2304,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 29.49,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "eca7238f811f5980b514adde1f0727fc6ce6d33efe16bd620cef184831ed1b44",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9645,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
48.5,
|
||||
35.7,
|
||||
25.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 29419.892072677612,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1470.9946036338806,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 29419.984102249146,
|
||||
"image_time_to_first_output_ms": 29419.86107826233,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.09873499948298559,
|
||||
"stage_0_gen_ms": 29419.892072677612
|
||||
},
|
||||
"peak_memory_mb": 23432.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2301,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 34.08,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "ed2978bbee6c6d667a1e113fddb6e5d7c06d9b0c7c0d8eda77ad330b92bdc45f",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.976,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
49.0,
|
||||
37.3,
|
||||
29.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 34010.05530357361,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1700.5027651786804,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 34010.1637840271,
|
||||
"image_time_to_first_output_ms": 34010.02025604248,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11569900379981846,
|
||||
"stage_0_gen_ms": 34010.05530357361
|
||||
},
|
||||
"peak_memory_mb": 23598.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2302,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 34.0,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "f4433739a48bdb32c24f0b53d4feee604adad91327d0b25656cb0b28d4051e1b",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9713,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
47.2,
|
||||
39.5,
|
||||
29.8
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 33924.51333999634,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1696.225666999817,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 33924.62372779846,
|
||||
"image_time_to_first_output_ms": 33924.47805404663,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11932000052183867,
|
||||
"stage_0_gen_ms": 33924.51333999634
|
||||
},
|
||||
"peak_memory_mb": 23590.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2303,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 34.02,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "9b673d3e82dc29d8781dc9d2e0a40bdd7dcad746332f9566e80e5bd8a055bd30",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9706,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
56.4,
|
||||
44.8,
|
||||
35.4
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 33950.94609260559,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1697.5473046302795,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 33951.04670524597,
|
||||
"image_time_to_first_output_ms": 33950.846672058105,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.17462699906900525,
|
||||
"stage_0_gen_ms": 33950.94609260559
|
||||
},
|
||||
"peak_memory_mb": 23590.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2304,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 34.04,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "22d9558389165abb8e58c02e7f1c4d49d4f312ff178062246c9907d519336ad6",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.9674,
|
||||
"alpha_mean": 255.0,
|
||||
"rgb_std": [
|
||||
50.9,
|
||||
38.3,
|
||||
29.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 33973.530292510986,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 1698.6765146255493,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 33973.62184524536,
|
||||
"image_time_to_first_output_ms": 33973.4947681427,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.09957399743143469,
|
||||
"stage_0_gen_ms": 33973.530292510986
|
||||
},
|
||||
"peak_memory_mb": 23590.0
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -1,89 +0,0 @@
|
||||
{
|
||||
"schema": "issue5-int8wo-v1",
|
||||
"created_utc": "2026-10-08T11:16:41+00:00",
|
||||
"description": "Qwen-Image-2.1 DiT int8 weight-only (W8A16), compressed-tensors pack-quantized",
|
||||
"inputs": {
|
||||
"snapshot": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"revision": "790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"tree_manifest": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/trees/790c92633540aa0cb11d9abf19eb46d861714758.json",
|
||||
"tree_sha256": "1f35d0a5a99f2c79bf2a6818b96582828094feb8544a992bc81c1d4100aed436"
|
||||
},
|
||||
"versions": {
|
||||
"python": "3.13.15",
|
||||
"torch": "2.13.0+cu130",
|
||||
"compressed_tensors": "0.17.0"
|
||||
},
|
||||
"quantization": {
|
||||
"method": "compressed-tensors",
|
||||
"format": "pack-quantized",
|
||||
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||
"config": {
|
||||
"config_groups": {
|
||||
"group_0": {
|
||||
"targets": [
|
||||
"Linear"
|
||||
],
|
||||
"weights": {
|
||||
"num_bits": 8,
|
||||
"type": "int",
|
||||
"symmetric": true,
|
||||
"group_size": null,
|
||||
"strategy": "channel",
|
||||
"block_structure": null,
|
||||
"dynamic": false,
|
||||
"actorder": null,
|
||||
"scale_dtype": null,
|
||||
"zp_dtype": null,
|
||||
"observer": "minmax",
|
||||
"observer_kwargs": {}
|
||||
},
|
||||
"input_activations": null,
|
||||
"output_activations": null,
|
||||
"format": null
|
||||
}
|
||||
},
|
||||
"quant_method": "compressed-tensors",
|
||||
"kv_cache_scheme": null,
|
||||
"format": "pack-quantized",
|
||||
"quantization_status": "frozen",
|
||||
"global_compression_ratio": null,
|
||||
"ignore": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
]
|
||||
},
|
||||
"ignored_layers": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
],
|
||||
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers"
|
||||
},
|
||||
"stats": {
|
||||
"quantized_tensors": 224,
|
||||
"quantized_params": 6979321856,
|
||||
"bf16_params": 135802880,
|
||||
"mean_rel_l2": 0.010959165191577216,
|
||||
"max_rel_l2": 0.017284497618675232,
|
||||
"weights_file_bytes": 7256783616
|
||||
},
|
||||
"files": [
|
||||
{
|
||||
"path": "transformer/config.json",
|
||||
"bytes": 1405,
|
||||
"sha256": "43b8c94380afb473f252d2707d1398b2be180f0248acee31cad4d6613f4c0a84"
|
||||
},
|
||||
{
|
||||
"path": "transformer/diffusion_pytorch_model.safetensors",
|
||||
"bytes": 7256783616,
|
||||
"sha256": "947d9c0660776b1125ec98762817af55d59d326b8e223120aa1bdb0eced91aa9"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
transformer_blocks.0.attn.to_k.weight: shape (4096, 4096) rel_l2=0.01092 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_out.0.weight: shape (4096, 4096) rel_l2=0.00978 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_q.weight: shape (4096, 4096) rel_l2=0.01106 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_v.weight: shape (4096, 4096) rel_l2=0.00976 scale (4096, 1)/torch.float32
|
||||
checked 4 tensors
|
||||
@@ -1,119 +0,0 @@
|
||||
== int8wo ==
|
||||
1-ref: n=4 gate=FAIL wall 30.5-31.7s (mean 31.0s)
|
||||
edit-expression-ysolde s2301: 31.73s {"opaque_fraction": 0.0294, "alpha_mean": 114.2, "rgb_std": [59.9, 61.7, 63.8]} BROKEN
|
||||
edit-expression-ysolde s2302: 30.53s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.2, 2.9, 3.3]} BROKEN
|
||||
edit-expression-ysolde s2303: 30.88s {"opaque_fraction": 0.0391, "alpha_mean": 125.5, "rgb_std": [63.5, 66.2, 67.0]} BROKEN
|
||||
edit-expression-ysolde s2304: 30.95s {"opaque_fraction": 0.0352, "alpha_mean": 122.4, "rgb_std": [63.6, 66.3, 67.1]} BROKEN
|
||||
2-ref: n=4 gate=FAIL wall 36.5-36.7s (mean 36.6s)
|
||||
edit-compose-pair s2301: 36.46s {"opaque_fraction": 0.0303, "alpha_mean": 115.4, "rgb_std": [59.2, 61.0, 63.3]} BROKEN
|
||||
edit-compose-pair s2302: 36.57s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.1, 2.9, 3.2]} BROKEN
|
||||
edit-compose-pair s2303: 36.61s {"opaque_fraction": 0.03, "alpha_mean": 115.7, "rgb_std": [76.1, 77.9, 69.1]} BROKEN
|
||||
edit-compose-pair s2304: 36.69s {"opaque_fraction": 0.1257, "alpha_mean": 127.1, "rgb_std": [64.3, 64.3, 64.3]} BROKEN
|
||||
3-ref: n=4 gate=FAIL wall 41.3-41.6s (mean 41.4s)
|
||||
edit-compose-three s2301: 41.57s {"opaque_fraction": 0.2029, "alpha_mean": 137.2, "rgb_std": [93.0, 92.7, 91.0]} BROKEN
|
||||
edit-compose-three s2302: 41.32s {"opaque_fraction": 0.0193, "alpha_mean": 131.9, "rgb_std": [40.8, 45.8, 44.3]} BROKEN
|
||||
edit-compose-three s2303: 41.34s {"opaque_fraction": 0.0188, "alpha_mean": 131.6, "rgb_std": [41.0, 45.9, 44.6]} BROKEN
|
||||
edit-compose-three s2304: 41.39s {"opaque_fraction": 0.0189, "alpha_mean": 131.6, "rgb_std": [41.1, 46.0, 44.6]} BROKEN
|
||||
4-ref: n=4 gate=FAIL wall 46.4-46.5s (mean 46.4s)
|
||||
edit-compose-four s2301: 46.4s {"opaque_fraction": 0.0195, "alpha_mean": 131.6, "rgb_std": [41.4, 46.3, 45.0]} BROKEN
|
||||
edit-compose-four s2302: 46.42s {"opaque_fraction": 0.0196, "alpha_mean": 131.9, "rgb_std": [41.1, 46.0, 44.5]} BROKEN
|
||||
edit-compose-four s2303: 46.44s {"opaque_fraction": 0.0195, "alpha_mean": 132.0, "rgb_std": [41.1, 46.1, 44.7]} BROKEN
|
||||
edit-compose-four s2304: 46.48s {"opaque_fraction": 0.0195, "alpha_mean": 131.7, "rgb_std": [41.0, 46.1, 44.5]} BROKEN
|
||||
nvidia-smi peak: 23935 MiB; server peak: None MB
|
||||
STRUCTURE GATE: FAIL
|
||||
== psnr edit-expression-ysolde ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 6.95,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 8.3,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 6.67,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 6.65,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.14,
|
||||
"min_db": 6.65,
|
||||
"max_db": 8.3,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-pair ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.03,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 8.86,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 6.77,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 7.19,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.46,
|
||||
"min_db": 6.77,
|
||||
"max_db": 8.86,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-four ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.02,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 7.11,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 7.1,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 7.2,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.11,
|
||||
"min_db": 7.02,
|
||||
"max_db": 7.2,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
23935 MiB, 100 %
|
||||
(APIServer pid=473436) INFO 10-08 07:36:28 [data.py:1483] Auto-detected quantization 'component' from model config
|
||||
(DiffusionWorker pid=473436) INFO 10-08 07:36:28 [diffusers_loader.py:719] Offline-quantized model with CPU offload, loading weights directly on CPU
|
||||
@@ -1,82 +0,0 @@
|
||||
[
|
||||
{
|
||||
"layer": "attn.to_qkv (fused q/k/v)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0006612942670471966,
|
||||
"max": 0.012303149327635765
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
12288
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
12288
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "attn.to_out.0 (row)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.00037870631786063313,
|
||||
"max": 0.011257382109761238
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
4096
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
4096
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.proj (column)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0005651759565807879,
|
||||
"max": 0.005628691054880619
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
12288
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
12288
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.out (row, wide-in)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0008073942153714597,
|
||||
"max": 0.014763779938220978
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
4096
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
4096
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,58 +0,0 @@
|
||||
[
|
||||
{
|
||||
"layer": "attn.to_qkv (fused q/k/v)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02410734072327614,
|
||||
"max": 0.043380916118621826
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "attn.to_out.0 (row)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02417764998972416,
|
||||
"max": 0.042702700942754745
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.proj (column)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.024067752063274384,
|
||||
"max": 0.04336588457226753
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.out (row, wide-in)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02682814560830593,
|
||||
"max": 0.04223968833684921
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -1,978 +0,0 @@
|
||||
[
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2301,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 31.73,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "a62cc903858a5026c08f284bdfea58108068fd6ab6425b54c246d65990e0bec4",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0294,
|
||||
"alpha_mean": 114.2,
|
||||
"rgb_std": [
|
||||
59.9,
|
||||
61.7,
|
||||
63.8
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 31616.730451583862,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1580.8365225791931,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 31616.933584213257,
|
||||
"image_time_to_first_output_ms": 31616.7151927948,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.18030800129054114,
|
||||
"stage_0_gen_ms": 31616.730451583862
|
||||
},
|
||||
"peak_memory_mb": 21594.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2302,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.53,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "ae5c51a9a36f8892c15116b90bcd2b85f90a83bae3daea2196f2602b5968e9af",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0,
|
||||
"alpha_mean": 126.9,
|
||||
"rgb_std": [
|
||||
3.2,
|
||||
2.9,
|
||||
3.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30463.57011795044,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1523.178505897522,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30463.69981765747,
|
||||
"image_time_to_first_output_ms": 30463.540077209473,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.12335599603829905,
|
||||
"stage_0_gen_ms": 30463.57011795044
|
||||
},
|
||||
"peak_memory_mb": 21900.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2303,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.88,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "d05825c0d525ff8522d83db2a9fc6337cc06a3960f7eb2a112898c5b71aa34a1",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0391,
|
||||
"alpha_mean": 125.5,
|
||||
"rgb_std": [
|
||||
63.5,
|
||||
66.2,
|
||||
67.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30779.66594696045,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1538.9832973480225,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30779.80923652649,
|
||||
"image_time_to_first_output_ms": 30779.641151428223,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14197500422596931,
|
||||
"stage_0_gen_ms": 30779.66594696045
|
||||
},
|
||||
"peak_memory_mb": 21906.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2304,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.95,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "10e85cf24451f4fb7bf2d8e07b4039158af08e36467a2a14d2baee1d68cd98fa",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0352,
|
||||
"alpha_mean": 122.4,
|
||||
"rgb_std": [
|
||||
63.6,
|
||||
66.3,
|
||||
67.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30871.540069580078,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1543.577003479004,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30871.644735336304,
|
||||
"image_time_to_first_output_ms": 30871.514797210693,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10297299741068855,
|
||||
"stage_0_gen_ms": 30871.540069580078
|
||||
},
|
||||
"peak_memory_mb": 21906.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2301,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.46,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "4bfdf57ecb0e1ed1c2c4dcb10ce0545d89040ad08ed754e01adbe2ddd2b407d2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0303,
|
||||
"alpha_mean": 115.4,
|
||||
"rgb_std": [
|
||||
59.2,
|
||||
61.0,
|
||||
63.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36376.55758857727,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1818.8278794288635,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36376.71780586243,
|
||||
"image_time_to_first_output_ms": 36376.53112411499,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15923400496831164,
|
||||
"stage_0_gen_ms": 36376.55758857727
|
||||
},
|
||||
"peak_memory_mb": 22676.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2302,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.57,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "e5ad17605624fdb636dc02e272c0dba2d3286dcfa4c27f297a605e785f3177a2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0,
|
||||
"alpha_mean": 126.9,
|
||||
"rgb_std": [
|
||||
3.1,
|
||||
2.9,
|
||||
3.2
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36508.753061294556,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1825.4376530647278,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36508.851528167725,
|
||||
"image_time_to_first_output_ms": 36508.72349739075,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1011220010695979,
|
||||
"stage_0_gen_ms": 36508.753061294556
|
||||
},
|
||||
"peak_memory_mb": 22556.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2303,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.61,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "f58a982cf1f3afd4a81fcc6b349aa3b147fecea0594e33f49cab5673f300b2ac",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.03,
|
||||
"alpha_mean": 115.7,
|
||||
"rgb_std": [
|
||||
76.1,
|
||||
77.9,
|
||||
69.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36523.38361740112,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1826.1691808700562,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36523.181676864624,
|
||||
"image_time_to_first_output_ms": 36523.05722236633,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.0983119971351698,
|
||||
"stage_0_gen_ms": 36523.38361740112
|
||||
},
|
||||
"peak_memory_mb": 22554.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2304,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.69,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "871c8147b27310508fb2c9e72b3b47499d7f8ab4ea705719aa30c465c8a5476a",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.1257,
|
||||
"alpha_mean": 127.1,
|
||||
"rgb_std": [
|
||||
64.3,
|
||||
64.3,
|
||||
64.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36608.7908744812,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1830.43954372406,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36608.89911651611,
|
||||
"image_time_to_first_output_ms": 36608.76417160034,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10716100223362446,
|
||||
"stage_0_gen_ms": 36608.7908744812
|
||||
},
|
||||
"peak_memory_mb": 22556.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2301,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.57,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "810cb635438bac76de76fa44172687e2f18b212379458580a32bbddd028bd78c",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.2029,
|
||||
"alpha_mean": 137.2,
|
||||
"rgb_std": [
|
||||
93.0,
|
||||
92.7,
|
||||
91.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41476.71937942505,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2073.8359689712524,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41476.8271446228,
|
||||
"image_time_to_first_output_ms": 41476.691246032715,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1088700009859167,
|
||||
"stage_0_gen_ms": 41476.71937942505
|
||||
},
|
||||
"peak_memory_mb": 23468.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2302,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.32,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "2f085a5089ab0618a2f17df7c29c52fde091247cb4a1ccf0cd8d662171b18fc9",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0193,
|
||||
"alpha_mean": 131.9,
|
||||
"rgb_std": [
|
||||
40.8,
|
||||
45.8,
|
||||
44.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41233.43777656555,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2061.6718888282776,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41233.57892036438,
|
||||
"image_time_to_first_output_ms": 41233.40702056885,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14587400073651224,
|
||||
"stage_0_gen_ms": 41233.43777656555
|
||||
},
|
||||
"peak_memory_mb": 23460.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2303,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.34,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "81e9b7436d392c8420d1529976a0d88e5de03af1e70823cc8c7b279b3d7bb3fa",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0188,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.0,
|
||||
45.9,
|
||||
44.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41250.11157989502,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2062.505578994751,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41250.218868255615,
|
||||
"image_time_to_first_output_ms": 41250.08296966553,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10969900176860392,
|
||||
"stage_0_gen_ms": 41250.11157989502
|
||||
},
|
||||
"peak_memory_mb": 23394.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2304,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.39,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "2221932a1783102d87c01157e231160803cebac2113c86b9416e537e0e738747",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0189,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.0,
|
||||
44.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41297.605991363525,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2064.8802995681763,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41297.31011390686,
|
||||
"image_time_to_first_output_ms": 41297.15037345886,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.134524998429697,
|
||||
"stage_0_gen_ms": 41297.605991363525
|
||||
},
|
||||
"peak_memory_mb": 23394.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2301,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.4,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "abd057bee84e594437f895cc9f51f90f118adfeb4dbf738686791b41487e93b2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.4,
|
||||
46.3,
|
||||
45.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46304.86750602722,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2315.243375301361,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46304.97694015503,
|
||||
"image_time_to_first_output_ms": 46304.8369884491,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11391400039428845,
|
||||
"stage_0_gen_ms": 46304.86750602722
|
||||
},
|
||||
"peak_memory_mb": 23534.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2302,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.42,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "24a783a9ca3a002745351476e8000be87854d1364ce92ca944c2986a983922f5",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0196,
|
||||
"alpha_mean": 131.9,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.0,
|
||||
44.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46325.687408447266,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2316.2843704223633,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46324.424266815186,
|
||||
"image_time_to_first_output_ms": 46324.246883392334,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15303100371966138,
|
||||
"stage_0_gen_ms": 46325.687408447266
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2303,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.44,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "db507c15651324404b79d6e5446e43fb460797204b302c0c258be37f1e2a7a80",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 132.0,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.1,
|
||||
44.7
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46352.77342796326,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2317.638671398163,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46352.887868881226,
|
||||
"image_time_to_first_output_ms": 46352.74004936218,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1095690022339113,
|
||||
"stage_0_gen_ms": 46352.77342796326
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2304,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.48,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "5481e029af2cb8ef3140565c615a5abc97ed755ff180b1970badf3a997c60ac4",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 131.7,
|
||||
"rgb_std": [
|
||||
41.0,
|
||||
46.1,
|
||||
44.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46385.611057281494,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2319.2805528640747,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46385.305404663086,
|
||||
"image_time_to_first_output_ms": 46385.130643844604,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14768500113859773,
|
||||
"stage_0_gen_ms": 46385.611057281494
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -1,80 +0,0 @@
|
||||
{
|
||||
"schema": "issue5-runs-summary-v2",
|
||||
"probed_utc": "2026-10-08",
|
||||
"round_two": {
|
||||
"directive": "issue 5 comment 2026-10-08T07:51 (amended 08:37): one decisive experiment - W8A8 int8-dynamic, non-Marlin",
|
||||
"artifact": "~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-w8a8dyn",
|
||||
"kernel": "CutlassInt8ScaledMMLinearKernel (ops.scaled_int8_quant + ops.cutlass_scaled_mm)",
|
||||
"kernel_verification": {
|
||||
"raw_ops_rel_l2": "0.0017 at (64,256,128)/(512,4096,4096)/(512,4096,12288) vs float32 reference (cutlass-ops-probe.json in evidence/rtx3090-round2)",
|
||||
"layer_path_rel_l2": "0.0017 for fused QKV and row-parallel, synthetic weights, loaded bitwise-verified (w8a8-layer-isolate.json)"
|
||||
},
|
||||
"serve_run": {
|
||||
"label": "int8w8a8dyn",
|
||||
"init": "healthy in ~40 s",
|
||||
"wall_s_mean": {"1-ref": 20.5, "2-ref": 25.1, "3-ref": 29.4, "4-ref": 34.0},
|
||||
"nvidia_smi_peak_mib": 23879,
|
||||
"structure": "opaque 0.947-0.988, alpha mean 254.9-255.0, rgb_std 21-72 across all 16 requests; visual checks render faithful scenes",
|
||||
"gate_note": "round-one 0.99-opacity rule was a broken-vs-BF16 calibration; W8A8 outputs are soft-edge alpha, not the broken signature (0.008-0.20 band); verdict rests on raw numbers + visual + PSNR",
|
||||
"psnr_vs_sealed_diffusers_mean_db": {"1-ref": 33.72, "2-ref": 28.34, "4-ref": 27.50},
|
||||
"bf16_reference_pair_db": 27.57
|
||||
},
|
||||
"verdict": "YES - W8A8 int8-dynamic delivers 3-4 reference capacity at the fastest measured speed with BF16-parity quality; hard stop did not trigger; Marlin W8A16 root-cause draft staged for the #3 lane (UPSTREAM-REPORT-DRAFT.md)"
|
||||
},
|
||||
"host": "cachyos-alexander (RTX 3090 24 GB, sm_86, driver 615.71.09, CUDA 13.4)",
|
||||
"stack": {
|
||||
"vllm_omni": "ISSUE-5 branch e778d21f..67e17786 over main 8484d85d1",
|
||||
"vllm": "0.31.0",
|
||||
"torch": "2.13.0+cu130",
|
||||
"compressed_tensors": "0.17.0",
|
||||
"transformers": "5.14.1",
|
||||
"diffusers": "0.41.0.dev0 (git 122b1e11)"
|
||||
},
|
||||
"artifact": {
|
||||
"path": "~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo",
|
||||
"quantized_params": 6979321856,
|
||||
"bf16_boundary_params": 135802880,
|
||||
"weights_file_bytes": 7256783616,
|
||||
"mean_rel_l2_quant_error": 0.010959,
|
||||
"max_rel_l2_quant_error": 0.017284,
|
||||
"roundtrip_verification": "unpack+dequant vs sealed BF16: rel_l2 0.0098-0.0117 per tensor (see artifact-verify.txt)"
|
||||
},
|
||||
"serve_run": {
|
||||
"label": "int8wo",
|
||||
"command": "vllm serve <artifact> --omni --port 8091 --served-model-name Qwen/Qwen-Image-2.1 --enforce-eager --enable-cpu-offload",
|
||||
"init": "healthy in ~40 s; per-component quantization auto-detected from transformer/config.json (transformer -> compressed-tensors, encoder/VAE BF16); CompressedTensorsWNA16 + MarlinLinearKernel selected on sm_86",
|
||||
"failed_precursor_runs": [
|
||||
"int8wo (first): init OOM - loader treated the pre-quantized checkpoint as online quantization and loaded all components on the GPU (encoder 17 GiB + VAE + packed DiT skeleton > 24 GiB); fixed by the offline-detection loader change on this branch",
|
||||
"int8wo-expandable: same OOM with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (genuine capacity, not fragmentation); also polluted by a leftover 586 MiB DiffusionWorker from an interrupted test run"
|
||||
]
|
||||
},
|
||||
"capacity_matrix": {
|
||||
"fixtures": {"1-ref": "edit-expression-ysolde", "2-ref": "edit-compose-pair", "3-ref": "edit-compose-three", "4-ref": "edit-compose-four"},
|
||||
"seeds": [2301, 2302, 2303, 2304],
|
||||
"requests": 16,
|
||||
"http_200": 16,
|
||||
"wall_s_mean": {"1-ref": 31.0, "2-ref": 36.6, "3-ref": 41.4, "4-ref": 46.4},
|
||||
"nvidia_smi_peak_mib": 23935,
|
||||
"bf16_reference_walls": {"1-ref": 34.2, "2-ref": 36.7, "4-ref": "OOM at BF16 (issue 2)"},
|
||||
"fp8_reference_walls": {"1-ref": 29.9, "2-ref": 34.1, "3-ref": 38.9, "4-ref": 44.3}
|
||||
},
|
||||
"structure_gate": {
|
||||
"rule": "opaque_fraction >= 0.99 AND mean(rgb_std) >= 10",
|
||||
"result": "FAIL 16/16",
|
||||
"signature": "opaque 0.0-0.20 (vs 1.0 at BF16), alpha mean 114-137, rgb_std 3-93; outputs are horizontal-stripe noise, not rendered scenes",
|
||||
"detail": "gate-and-psnr-summary.txt; per-request records in matrix.json"
|
||||
},
|
||||
"psnr_vs_sealed_diffusers_baseline": {
|
||||
"edit-expression-ysolde_1ref_mean_db": 7.14,
|
||||
"edit-compose-pair_2ref_mean_db": 7.46,
|
||||
"edit-compose-four_4ref_mean_db": 7.11,
|
||||
"bf16_pair_reference_mean_db": 27.57,
|
||||
"fp8_broken_reference_mean_db": 7.11
|
||||
},
|
||||
"marlin_isolation": {
|
||||
"method": "marlin_isolate.py: vLLM QKVParallelLinear/RowParallelLinear/ColumnParallelLinear built exactly as the qwen_image_21 transformer does; artifact tensors loaded through the production weight-loaders; Marlin repack on GPU; layer(x) compared against float32 x @ dequant(W)^T",
|
||||
"results_block0_artifact": "NaN output, all four layer kinds, bf16 AND fp16 activations; loaded weight_packed bitwise-equal to artifact (incl. fused q/k/v stacked shards); scales finite pre and post repack (marlin-isolate-block0.json)",
|
||||
"results_synthetic_random_int8": "NaN identically - the checkpoint and its quantization are exonerated (marlin-isolate-synthetic.json)",
|
||||
"conclusion": "the 8-bit Marlin W8A16 path on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) produces NaN for int8 weight-only layers although query_marlin_supported_quant_types accepts uint8b128 on the card; choose_mp_linear_kernel offers no non-Marlin fallback for W8A16 on sm_86 (Machete is sm90+, ExLlama int4-only)"
|
||||
}
|
||||
}
|
||||
@@ -1,94 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 structure gate + capacity summarizer.
|
||||
|
||||
Applies the gate rule from issue 1's FP8 misread to every recorded request
|
||||
and folds a run directory into the capacity matrix row: per reference
|
||||
count, wall time, peak VRAM, and the server-reported peak.
|
||||
|
||||
Gate rule (a run only counts when structurally sound):
|
||||
opaque_fraction >= 0.99 AND mean(rgb_std) >= 10
|
||||
|
||||
Broken FP8 output measured opaque 0.008-0.035 with flat grey RGB (std of a
|
||||
few); the BF16 path is 100% opaque with healthy channel spread.
|
||||
|
||||
Usage: gate.py RUN_DIR [RUN_DIR ...] # reads matrix.json / matrix-*.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
|
||||
def peak_vram_mb(vram_log: pathlib.Path) -> int | None:
|
||||
if not vram_log.exists():
|
||||
return None
|
||||
peak = 0
|
||||
for line in vram_log.read_text().splitlines():
|
||||
match = re.match(r"\s*(\d+)\s+MiB\s*,", line) or re.match(r"\s*(\d+),", line)
|
||||
if match:
|
||||
peak = max(peak, int(match.group(1)))
|
||||
return peak or None
|
||||
|
||||
|
||||
def server_peak_mb(serve_log: pathlib.Path) -> int | None:
|
||||
if not serve_log.exists():
|
||||
return None
|
||||
peaks = [int(m) for m in re.findall(r"peak.*?(\d[\d,]+)\s*MB", serve_log.read_text())]
|
||||
if not peaks:
|
||||
return None
|
||||
return max(peaks)
|
||||
|
||||
|
||||
def gate(structure: dict) -> bool:
|
||||
return (
|
||||
structure["opaque_fraction"] >= 0.99
|
||||
and sum(structure["rgb_std"]) / len(structure["rgb_std"]) >= 10
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("runs", nargs="+", type=pathlib.Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
for run_dir in args.runs:
|
||||
records: list[dict] = []
|
||||
for candidate in sorted(run_dir.glob("matrix*.json")):
|
||||
records.extend(json.loads(candidate.read_text()))
|
||||
by_refs: dict[int, list[dict]] = {}
|
||||
for rec in records:
|
||||
by_refs.setdefault(rec.get("n_references", 0), []).append(rec)
|
||||
|
||||
print(f"== {run_dir.name} ==")
|
||||
all_pass = bool(records)
|
||||
for n_refs in sorted(by_refs):
|
||||
rows = by_refs[n_refs]
|
||||
walls = [r["wall_s"] for r in rows if "wall_s" in r]
|
||||
gates = [gate(r["structure"]) for r in rows if "structure" in r]
|
||||
row_pass = bool(rows) and all(gates) and len(gates) == len(rows)
|
||||
all_pass = all_pass and row_pass
|
||||
print(
|
||||
f" {n_refs}-ref: n={len(rows)} gate={'PASS' if row_pass else 'FAIL'} "
|
||||
f"wall {min(walls):.1f}-{max(walls):.1f}s"
|
||||
+ (f" (mean {sum(walls) / len(walls):.1f}s)" if walls else "")
|
||||
)
|
||||
for rec in rows:
|
||||
s = rec.get("structure")
|
||||
tag = "" if s is None else (" ok" if gate(s) else " BROKEN")
|
||||
err = f" error={rec['error'][:80]}" if rec.get("error") else ""
|
||||
print(
|
||||
f" {rec.get('fixture_id')} s{rec.get('seed')}: "
|
||||
f"{rec.get('wall_s')}s {json.dumps(s) if s else 'no-image'}{tag}{err}"
|
||||
)
|
||||
peak = peak_vram_mb(run_dir / "vram.log")
|
||||
server_peak = server_peak_mb(run_dir / "serve.log")
|
||||
print(f" nvidia-smi peak: {peak} MiB; server peak: {server_peak} MB")
|
||||
print(f" STRUCTURE GATE: {'PASS' if all_pass else 'FAIL'}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,239 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 diagnostic: isolate the compressed-tensors W8A16 serve path per layer.
|
||||
|
||||
Builds vLLM parallel linear layers exactly as the qwen_image_21 transformer
|
||||
does (fused QKV, row-parallel to_out, img_mlp proj/out), loads the int8
|
||||
artifact's packed tensors through the production weight-loader machinery,
|
||||
runs the Marlin repack (process_weights_after_loading) on the GPU, and
|
||||
compares each layer's forward output against a float32 reference computed
|
||||
from the artifact's own unpacked+dequantized weights.
|
||||
|
||||
Expected rel-L2 when the load->repack->apply chain is correct: ~1-2%
|
||||
(int8 quantization error plus bf16 activation rounding). A large error
|
||||
pins corruption inside the serve path with the checkpoint already
|
||||
verified correct (verify_artifact.py).
|
||||
|
||||
Usage: marlin_isolate.py --artifact ARTIFACT_DIR [--block N]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
import torch
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
|
||||
from safetensors.torch import load_file
|
||||
|
||||
|
||||
def dequantized_weight(art: dict, base: str) -> torch.Tensor:
|
||||
shape = tuple(art[f"{base}_shape"].tolist())
|
||||
q = unpack_from_int32(art[f"{base}_packed"].cpu(), 8, shape)
|
||||
return q.to(torch.float32) * art[f"{base}_scale"].cpu().to(torch.float32)
|
||||
|
||||
|
||||
def load_tensor(layer: torch.nn.Module, param_name: str, tensor: torch.Tensor, shard: str | None) -> None:
|
||||
param = getattr(layer, param_name)
|
||||
if shard is None:
|
||||
param.weight_loader(param, tensor)
|
||||
else:
|
||||
param.weight_loader(param, tensor, shard)
|
||||
|
||||
|
||||
def check(name: str, layer: torch.nn.Module, art: dict, base: str, shards: list[tuple[str, str]], in_features: int, dtype: torch.dtype) -> dict:
|
||||
device = torch.device("cuda")
|
||||
for suffix in ("weight_packed", "weight_scale", "weight_shape"):
|
||||
for ckpt_base, shard in shards:
|
||||
load_tensor(layer, suffix, art[f"{ckpt_base}.{suffix}"].to(device), shard)
|
||||
|
||||
# Bitwise check of the loaded packed weights against the artifact.
|
||||
loaded = layer.weight_packed.detach().cpu()
|
||||
if len(shards) > 1:
|
||||
ref_packed = torch.cat([art[f"{b}.weight_packed"] for b, _ in shards], dim=0)
|
||||
else:
|
||||
ref_packed = art[f"{base}.weight_packed"]
|
||||
packed_ok = bool(torch.equal(loaded, ref_packed))
|
||||
scale_loaded = layer.weight_scale.detach().float()
|
||||
scale_ok = bool(torch.isfinite(scale_loaded).all())
|
||||
|
||||
layer.quant_method.process_weights_after_loading(layer)
|
||||
repacked_scale = getattr(layer, "weight_scale", None)
|
||||
scale_post = (
|
||||
{
|
||||
"finite": bool(torch.isfinite(repacked_scale.detach().float()).all()),
|
||||
"min": float(repacked_scale.detach().float().min()),
|
||||
"max": float(repacked_scale.detach().float().max()),
|
||||
}
|
||||
if repacked_scale is not None
|
||||
else {}
|
||||
)
|
||||
|
||||
torch.manual_seed(2301)
|
||||
ref_w = torch.cat([dequantized_weight(art, f"{b}.weight") for b, _ in shards], dim=0) if len(shards) > 1 else dequantized_weight(art, f"{base}.weight")
|
||||
outs = {}
|
||||
for act_dtype in (torch.bfloat16, torch.float16):
|
||||
x = torch.randn(512, in_features, dtype=act_dtype, device=device)
|
||||
result = layer(x)
|
||||
out = (result[0] if isinstance(result, tuple) else result).float()
|
||||
ref = x.float() @ ref_w.T.to(device)
|
||||
finite = bool(torch.isfinite(out).all())
|
||||
rel = float((out - ref).norm() / ref.norm())
|
||||
outs[str(act_dtype).replace("torch.", "")] = rel if finite else "NaN"
|
||||
|
||||
return {
|
||||
"layer": name,
|
||||
"rel_l2_by_act_dtype": outs,
|
||||
"packed_bitwise_ok": packed_ok,
|
||||
"scale_finite_pre_repack": scale_ok,
|
||||
"scale_post_repack": scale_post,
|
||||
}
|
||||
|
||||
|
||||
def synthetic_artifact(block: int) -> dict[str, torch.Tensor]:
|
||||
"""Random int8-quantized tensors in the artifact's layout, for stack-only isolation."""
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import pack_to_int32
|
||||
|
||||
def quant(name: str, out: int, inp: int) -> None:
|
||||
w = torch.randn(out, inp, dtype=torch.float32)
|
||||
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||
art[f"{name}.weight_packed"] = pack_to_int32(q.contiguous(), 8).contiguous()
|
||||
art[f"{name}.weight_scale"] = scale.to(torch.float32).contiguous()
|
||||
art[f"{name}.weight_shape"] = torch.tensor([out, inp], dtype=torch.int64)
|
||||
|
||||
art: dict[str, torch.Tensor] = {}
|
||||
p = f"transformer_blocks.{block}"
|
||||
for part in ("to_q", "to_k", "to_v"):
|
||||
quant(f"{p}.attn.{part}", 4096, 4096)
|
||||
quant(f"{p}.attn.to_out.0", 4096, 4096)
|
||||
quant(f"{p}.img_mlp.proj", 12288, 4096)
|
||||
quant(f"{p}.img_mlp.out", 4096, 12288)
|
||||
return art
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--artifact", type=pathlib.Path, required=True)
|
||||
parser.add_argument("--block", type=int, default=0)
|
||||
parser.add_argument(
|
||||
"--synthetic",
|
||||
action="store_true",
|
||||
help="ignore --artifact; use random int8 weights (isolates the stack from the checkpoint)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
import vllm.distributed
|
||||
from vllm.config import VllmConfig, set_current_vllm_config
|
||||
from vllm.distributed import (
|
||||
destroy_distributed_environment,
|
||||
destroy_model_parallel,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm.model_executor.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
|
||||
vllm.distributed.init_distributed_environment(
|
||||
backend="gloo",
|
||||
world_size=1,
|
||||
rank=0,
|
||||
local_rank=0,
|
||||
distributed_init_method="tcp://127.0.0.1:29601",
|
||||
)
|
||||
with set_current_vllm_config(VllmConfig()):
|
||||
initialize_model_parallel(tensor_model_parallel_size=1)
|
||||
|
||||
config = json.loads((args.artifact / "transformer" / "config.json").read_text())
|
||||
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||
|
||||
quant_config = resolve_component_quant_config(
|
||||
build_quant_config(config["quantization_config"]), "transformer"
|
||||
)
|
||||
if args.synthetic:
|
||||
art = synthetic_artifact(args.block)
|
||||
else:
|
||||
art = load_file(str(args.artifact / "transformer" / "diffusion_pytorch_model.safetensors"))
|
||||
b = args.block
|
||||
p = f"transformer.transformer_blocks.{b}"
|
||||
|
||||
results = []
|
||||
with torch.device("cuda"):
|
||||
to_qkv = QKVParallelLinear(
|
||||
hidden_size=4096, head_size=128, total_num_heads=32, bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_qkv",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"attn.to_qkv (fused q/k/v)",
|
||||
to_qkv,
|
||||
art,
|
||||
f"transformer_blocks.{b}.attn.to_qkv",
|
||||
[
|
||||
(f"transformer_blocks.{b}.attn.to_q", "q"),
|
||||
(f"transformer_blocks.{b}.attn.to_k", "k"),
|
||||
(f"transformer_blocks.{b}.attn.to_v", "v"),
|
||||
],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
to_out = RowParallelLinear(
|
||||
4096, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_out.0",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"attn.to_out.0 (row)",
|
||||
to_out,
|
||||
art,
|
||||
f"transformer_blocks.{b}.attn.to_out.0",
|
||||
[(f"transformer_blocks.{b}.attn.to_out.0", None)],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
proj = ColumnParallelLinear(
|
||||
4096, 12288, bias=False, gather_output=False, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.img_mlp.proj",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"img_mlp.proj (column)",
|
||||
proj,
|
||||
art,
|
||||
f"transformer_blocks.{b}.img_mlp.proj",
|
||||
[(f"transformer_blocks.{b}.img_mlp.proj", None)],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
mlp_out = RowParallelLinear(
|
||||
12288, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.img_mlp.out",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"img_mlp.out (row, wide-in)",
|
||||
mlp_out,
|
||||
art,
|
||||
f"transformer_blocks.{b}.img_mlp.out",
|
||||
[(f"transformer_blocks.{b}.img_mlp.out", None)],
|
||||
12288,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
destroy_model_parallel()
|
||||
destroy_distributed_environment()
|
||||
print(json.dumps(results, indent=1))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,252 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5: quantize the Qwen-Image-2.1 DiT to int8 weight-only (W8A16) and
|
||||
save it as a compressed-tensors pack-quantized checkpoint.
|
||||
|
||||
Quantizes exactly the linears the vllm-omni transformer constructs with
|
||||
``quant_config`` — per-block attention (to_q/to_k/to_v/to_out.0) and
|
||||
img_mlp (proj/gate_layer/out) — and copies every boundary layer (img_in,
|
||||
txt_in, time_text_embed, modulation, norm_out, proj_out) plus the RMSNorm
|
||||
q/k scales through in BF16. Per-output-channel symmetric int8, scales in
|
||||
fp32, weights packed 4x uint8 (biased by 128) per int32 via
|
||||
compressed-tensors' own ``pack_to_int32`` so the artifact matches the
|
||||
llm-compressor storage convention bit for bit.
|
||||
|
||||
The artifact is a full diffusers-layout model directory: the quantized
|
||||
``transformer/`` is real, the remaining components are symlinks into the
|
||||
sealed snapshot. ``transformer/config.json`` carries the quantization
|
||||
metadata wrapped per-component (``{"transformer": {...}}``) so vLLM-Omni's
|
||||
auto-detection quantizes the DiT only — the text encoder and VAE stay BF16.
|
||||
|
||||
A MANIFEST.json (inputs, versions, per-file sha256, quantization stats) is
|
||||
written beside the weights. Nothing here enters Git; the artifact lives
|
||||
under the helper-system state root.
|
||||
|
||||
Usage:
|
||||
quantize_dit_int8wo.py --snapshot SNAPSHOT_DIR --out ARTIFACT_DIR
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import platform
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import pack_to_int32
|
||||
from compressed_tensors.config import CompressionFormat
|
||||
from compressed_tensors.quantization import (
|
||||
QuantizationArgs,
|
||||
QuantizationConfig,
|
||||
QuantizationScheme,
|
||||
QuantizationStatus,
|
||||
QuantizationStrategy,
|
||||
QuantizationType,
|
||||
)
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
SNAPSHOT_COMMIT = "790c92633540aa0cb11d9abf19eb46d861714758"
|
||||
# Boundary layers the vllm-omni model pins to quant_config=None (BF16), plus
|
||||
# the non-linear tensors; everything else in a block is quantized.
|
||||
IGNORED_TARGETS = ["img_in", "txt_in", "time_text_embed", "modulation", "norm_out", "proj_out"]
|
||||
QUANTIZED_KEY = re.compile(
|
||||
r"^transformer_blocks\.\d+\."
|
||||
r"(attn\.(to_q|to_k|to_v|to_out\.0)|img_mlp\.(proj|gate_layer|out))\.weight$"
|
||||
)
|
||||
SYMLINKED_COMPONENTS = ("scheduler", "text_encoder", "vae", "processor")
|
||||
SYMLINKED_FILES = ("model_index.json",)
|
||||
|
||||
|
||||
def build_quantization_dict(scheme: str) -> dict:
|
||||
"""Serialized compressed-tensors metadata (llm-compressor schema).
|
||||
|
||||
scheme="int8wo": int8 weight-only, pack-quantized storage.
|
||||
scheme="w8a8dyn": int8 weights (static per-channel) + dynamic per-token
|
||||
symmetric int8 activations, int-quantized storage — the non-Marlin
|
||||
cutlass path on sm_86.
|
||||
"""
|
||||
weights = QuantizationArgs(
|
||||
num_bits=8,
|
||||
type=QuantizationType.INT,
|
||||
symmetric=True,
|
||||
strategy=QuantizationStrategy.CHANNEL,
|
||||
dynamic=False,
|
||||
observer="minmax",
|
||||
)
|
||||
input_activations = (
|
||||
QuantizationArgs(
|
||||
num_bits=8,
|
||||
type=QuantizationType.INT,
|
||||
symmetric=True,
|
||||
strategy=QuantizationStrategy.TOKEN,
|
||||
dynamic=True,
|
||||
)
|
||||
if scheme == "w8a8dyn"
|
||||
else None
|
||||
)
|
||||
scheme_obj = QuantizationScheme(targets=["Linear"], weights=weights, input_activations=input_activations)
|
||||
config = QuantizationConfig(
|
||||
config_groups={"group_0": scheme_obj},
|
||||
format=CompressionFormat.int_quantized if scheme == "w8a8dyn" else CompressionFormat.pack_quantized,
|
||||
ignore=list(IGNORED_TARGETS),
|
||||
quant_method="compressed-tensors",
|
||||
quantization_status=QuantizationStatus.FROZEN,
|
||||
)
|
||||
return config.to_dict()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def quantize_weight(weight: torch.Tensor, scheme: str) -> tuple[dict[str, torch.Tensor], dict]:
|
||||
"""Per-output-channel symmetric int8.
|
||||
|
||||
scheme="int8wo": q = round(W/s), s = rowmax(|W|)/127, packed via
|
||||
compressed-tensors' packer (uint8 biased by 128, 4 per int32).
|
||||
scheme="w8a8dyn": same quantization, stored unpacked as plain int8
|
||||
(int-quantized format) with fp32 scales.
|
||||
"""
|
||||
w = weight.detach().to(torch.float32)
|
||||
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||
dequant = q.to(torch.float32) * scale
|
||||
err = (w - dequant).norm() / w.norm().clamp_min(1e-12)
|
||||
stats = {"rel_l2": float(err), "out": int(w.shape[0]), "in": int(w.shape[1])}
|
||||
if scheme == "w8a8dyn":
|
||||
return {"weight": q.contiguous(), "weight_scale": scale.to(torch.float32).contiguous()}, stats
|
||||
return {
|
||||
"weight_packed": pack_to_int32(q.contiguous(), 8).contiguous(),
|
||||
"weight_scale": scale.to(torch.float32).contiguous(),
|
||||
"weight_shape": torch.tensor(weight.shape, dtype=torch.int64),
|
||||
}, stats
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--snapshot", type=Path, required=True, help="sealed snapshot dir")
|
||||
parser.add_argument("--out", type=Path, required=True, help="artifact dir to create")
|
||||
parser.add_argument(
|
||||
"--scheme",
|
||||
choices=("int8wo", "w8a8dyn"),
|
||||
default="int8wo",
|
||||
help="int8 weight-only (pack-quantized, Marlin path) or W8A8 int8-dynamic (int-quantized, cutlass path)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
snapshot = args.snapshot.resolve()
|
||||
if not (snapshot / "transformer" / "config.json").exists():
|
||||
sys.exit(f"snapshot transformer config not found under {snapshot}")
|
||||
if args.out.exists():
|
||||
sys.exit(f"refusing to overwrite existing {args.out}")
|
||||
artifact = args.out
|
||||
(artifact / "transformer").mkdir(parents=True)
|
||||
|
||||
index = json.loads(
|
||||
(snapshot / "transformer" / "diffusion_pytorch_model.safetensors.index.json").read_text()
|
||||
)
|
||||
shards = sorted(set(index["weight_map"].values()))
|
||||
state: dict[str, torch.Tensor] = {}
|
||||
quant_stats: dict[str, dict] = {}
|
||||
quant_params = bf16_params = 0
|
||||
for shard in shards:
|
||||
tensors = load_file(str(snapshot / "transformer" / shard))
|
||||
for name, tensor in tensors.items():
|
||||
if QUANTIZED_KEY.match(name):
|
||||
emitted, stats = quantize_weight(tensor, args.scheme)
|
||||
if args.scheme == "w8a8dyn":
|
||||
# int-quantized storage: plain int8 weight + scale, mirroring
|
||||
# the vLLM layer parameter names (weight / weight_scale).
|
||||
state[name] = emitted["weight"]
|
||||
state[f"{name}_scale"] = emitted["weight_scale"]
|
||||
else:
|
||||
state[f"{name}_packed"] = emitted["weight_packed"]
|
||||
state[f"{name}_scale"] = emitted["weight_scale"]
|
||||
state[f"{name}_shape"] = emitted["weight_shape"]
|
||||
quant_stats[name] = stats
|
||||
quant_params += tensor.numel()
|
||||
print(f"q {name} {tuple(tensor.shape)} rel_l2={stats['rel_l2']:.5f}", flush=True)
|
||||
else:
|
||||
state[name] = tensor
|
||||
bf16_params += tensor.numel()
|
||||
del tensors
|
||||
|
||||
weights_path = artifact / "transformer" / "diffusion_pytorch_model.safetensors"
|
||||
save_file(state, str(weights_path), metadata={"format": "pt"})
|
||||
total_bytes = weights_path.stat().st_size
|
||||
|
||||
config = json.loads((snapshot / "transformer" / "config.json").read_text())
|
||||
config["quantization_config"] = {"transformer": build_quantization_dict(args.scheme)}
|
||||
(artifact / "transformer" / "config.json").write_text(json.dumps(config, indent=2) + "\n")
|
||||
|
||||
for name in SYMLINKED_FILES:
|
||||
target = snapshot / name
|
||||
(artifact / name).symlink_to(target)
|
||||
for name in SYMLINKED_COMPONENTS:
|
||||
target = snapshot / name
|
||||
(artifact / name).symlink_to(target, target_is_directory=True)
|
||||
|
||||
files = [
|
||||
{"path": "transformer/config.json", "bytes": (artifact / "transformer" / "config.json").stat().st_size},
|
||||
{"path": "transformer/diffusion_pytorch_model.safetensors", "bytes": total_bytes},
|
||||
]
|
||||
for entry in files:
|
||||
entry["sha256"] = sha256_file(artifact / entry["path"])
|
||||
tree_json = snapshot.parent.parent / "trees" / f"{SNAPSHOT_COMMIT}.json"
|
||||
manifest = {
|
||||
"schema": f"issue5-{args.scheme}-v1",
|
||||
"created_utc": dt.datetime.now(dt.UTC).replace(microsecond=0).isoformat(),
|
||||
"description": (
|
||||
f"Qwen-Image-2.1 DiT {args.scheme} "
|
||||
+ (
|
||||
"W8A8 int8-dynamic, int-quantized"
|
||||
if args.scheme == "w8a8dyn"
|
||||
else "int8 weight-only W8A16, pack-quantized"
|
||||
)
|
||||
+ ", compressed-tensors"
|
||||
),
|
||||
"inputs": {
|
||||
"snapshot": str(snapshot),
|
||||
"revision": SNAPSHOT_COMMIT,
|
||||
"tree_manifest": str(tree_json),
|
||||
"tree_sha256": sha256_file(tree_json) if tree_json.exists() else None,
|
||||
},
|
||||
"versions": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"compressed_tensors": __import__("compressed_tensors").__version__,
|
||||
},
|
||||
"quantization": {
|
||||
"method": "compressed-tensors",
|
||||
"format": "pack-quantized",
|
||||
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||
"config": build_quantization_dict(args.scheme),
|
||||
"ignored_layers": list(IGNORED_TARGETS),
|
||||
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers",
|
||||
},
|
||||
"stats": {
|
||||
"quantized_tensors": len(quant_stats),
|
||||
"quantized_params": quant_params,
|
||||
"bf16_params": bf16_params,
|
||||
"mean_rel_l2": sum(s["rel_l2"] for s in quant_stats.values()) / max(len(quant_stats), 1),
|
||||
"max_rel_l2": max((s["rel_l2"] for s in quant_stats.values()), default=0.0),
|
||||
"weights_file_bytes": total_bytes,
|
||||
},
|
||||
"files": files,
|
||||
}
|
||||
(artifact / "MANIFEST.json").write_text(json.dumps(manifest, indent=1) + "\n")
|
||||
print(json.dumps(manifest["stats"], indent=1))
|
||||
print(f"artifact: {artifact}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,33 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Issue 5 probe: serve the int8 weight-only compressed-tensors artifact on
|
||||
# the RTX 3090 with one labeled flag set. VRAM is polled once per second for
|
||||
# the whole server lifetime, matching issue 2's instrument (nvidia-smi csv
|
||||
# series). The default flag set is cpu-offload + eager; pass extras after
|
||||
# the label to override/extend (they are appended, vLLM last-one-wins).
|
||||
#
|
||||
# Usage: serve.sh LABEL [extra vllm flags...]
|
||||
set -uo pipefail
|
||||
|
||||
BASE=$HOME/Projects/vllm-omni-val
|
||||
W=$BASE/issue5
|
||||
LABEL=$1; shift
|
||||
RUN=$W/runs/$LABEL
|
||||
mkdir -p "$RUN"
|
||||
|
||||
STATE=$HOME/.local/state/multi-model-helper-server/state/image-service
|
||||
ARTIFACT=${ARTIFACT:-$STATE/artifacts/qwen-image-2.1-dit-int8wo}
|
||||
export HF_HOME=$STATE/weights
|
||||
export HF_HUB_OFFLINE=1
|
||||
|
||||
( while sleep 1; do nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader; done ) > "$RUN/vram.log" 2>&1 &
|
||||
POLL=$!
|
||||
T0=$(date +%s)
|
||||
echo "model: $ARTIFACT" > "$RUN/flags.txt"
|
||||
echo "flags: $*" >> "$RUN/flags.txt"
|
||||
|
||||
$BASE/venv/bin/vllm serve "$ARTIFACT" --omni --port 8091 \
|
||||
--served-model-name Qwen/Qwen-Image-2.1 \
|
||||
--enforce-eager --enable-cpu-offload "$@" > "$RUN/serve.log" 2>&1
|
||||
RC=$?
|
||||
kill $POLL 2>/dev/null
|
||||
echo "serve_rc=$RC wall=$(( $(date +%s) - T0 ))s" >> "$RUN/serve.log"
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5: verify the int8 weight-only artifact round-trips to the BF16 original.
|
||||
|
||||
Unpacks ``weight_packed`` with compressed-tensors' own unpacker, dequantizes
|
||||
with the stored per-channel scales, and compares against the sealed snapshot's
|
||||
BF16 tensors. Expected rel-L2 ~1% (the quantizer's measured error); a large
|
||||
error means the artifact itself is corrupt, a small error means any serve-path
|
||||
collapse lives downstream of the checkpoint (loading/repack/kernel).
|
||||
|
||||
Usage: verify_artifact.py --artifact ARTIFACT_DIR --snapshot SNAPSHOT_DIR [--tensors N]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
import torch
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
|
||||
from safetensors.torch import load_file
|
||||
|
||||
QUANTIZED_KEY = "weight_packed" # checkpoint suffix; stripping it yields the base ".weight" name
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--artifact", type=pathlib.Path, required=True)
|
||||
parser.add_argument("--snapshot", type=pathlib.Path, required=True)
|
||||
parser.add_argument("--tensors", type=int, default=6, help="number of quantized tensors to check")
|
||||
args = parser.parse_args()
|
||||
|
||||
art = load_file(str(args.artifact / "transformer" / "diffusion_pytorch_model.safetensors"))
|
||||
index = json.loads(
|
||||
(args.snapshot / "transformer" / "diffusion_pytorch_model.safetensors.index.json").read_text()
|
||||
)["weight_map"]
|
||||
shard_cache: dict[str, dict] = {}
|
||||
|
||||
checked = 0
|
||||
for name in sorted(k.removesuffix("_packed") for k in art if k.endswith(f".{QUANTIZED_KEY}")):
|
||||
if checked >= args.tensors:
|
||||
break
|
||||
base_name = name
|
||||
shard = index[base_name]
|
||||
if shard not in shard_cache:
|
||||
shard_cache.clear() # keep at most one ~7 GB shard resident
|
||||
shard_cache[shard] = load_file(str(args.snapshot / "transformer" / shard))
|
||||
ref = shard_cache[shard][base_name].to(torch.float32)
|
||||
shape = tuple(art[f"{base_name}_shape"].tolist())
|
||||
q = unpack_from_int32(art[f"{base_name}_packed"], 8, shape)
|
||||
dequant = q.to(torch.float32) * art[f"{base_name}_scale"]
|
||||
rel = float((dequant - ref).norm() / ref.norm())
|
||||
print(
|
||||
f"{base_name}: shape {shape} rel_l2={rel:.5f} "
|
||||
f"scale {tuple(art[f'{base_name}_scale'].shape)}/{art[f'{base_name}_scale'].dtype}"
|
||||
)
|
||||
checked += 1
|
||||
print(f"checked {checked} tensors")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,174 +0,0 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 round two: isolate the compressed-tensors W8A8-int8-dynamic path.
|
||||
|
||||
Mirrors marlin_isolate.py but for the int-quantized W8A8 dynamic scheme
|
||||
(int8 weights, static per-output-channel scales, dynamic symmetric per-token
|
||||
int8 activations) whose kernel on sm_86 is the cutlass int8 scaled-MM —
|
||||
no Marlin in the path. Loads synthetic int8 tensors through the production
|
||||
weight-loader machinery and compares layer(x) against a float32 reference
|
||||
computed from the same dequantized weights with per-token dynamic int8
|
||||
activation quantization applied to the reference input.
|
||||
|
||||
Expected rel-L2 when the chain is correct: a few percent (int8 weight error
|
||||
~1% plus int8 per-token activation rounding).
|
||||
|
||||
Usage: w8a8_isolate.py [--block N]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
def synth(out: int, inp: int, ones: bool = False) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Per-channel symmetric int8 weight + scale, mirroring the artifact builder.
|
||||
|
||||
ones=True emits an analytically known weight (q=127, scale=1/127 -> W=1):
|
||||
with an all-ones input row the layer output must be exactly in_features.
|
||||
"""
|
||||
if ones:
|
||||
q = torch.full((out, inp), 127, dtype=torch.int8)
|
||||
return q, torch.full((out, 1), 1.0 / 127.0, dtype=torch.float32)
|
||||
w = torch.randn(out, inp, dtype=torch.float32)
|
||||
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||
return q, scale.to(torch.float32)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--block", type=int, default=0)
|
||||
parser.add_argument("--ones", action="store_true",
|
||||
help="analytic weights (W=1) and ones input; output must equal in_features")
|
||||
parser.add_argument("--act-dtype", choices=("bfloat16", "float16"), default="bfloat16")
|
||||
args = parser.parse_args()
|
||||
|
||||
import vllm.distributed
|
||||
from vllm.config import VllmConfig, set_current_vllm_config
|
||||
from vllm.distributed import (
|
||||
destroy_distributed_environment,
|
||||
destroy_model_parallel,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm.model_executor.layers.linear import QKVParallelLinear, RowParallelLinear
|
||||
from compressed_tensors.quantization import (
|
||||
QuantizationArgs,
|
||||
QuantizationConfig,
|
||||
QuantizationScheme,
|
||||
QuantizationStatus,
|
||||
QuantizationStrategy,
|
||||
QuantizationType,
|
||||
)
|
||||
from compressed_tensors.config import CompressionFormat
|
||||
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||
|
||||
weights = QuantizationArgs(num_bits=8, type=QuantizationType.INT, symmetric=True,
|
||||
strategy=QuantizationStrategy.CHANNEL, dynamic=False, observer="minmax")
|
||||
acts = QuantizationArgs(num_bits=8, type=QuantizationType.INT, symmetric=True,
|
||||
strategy=QuantizationStrategy.TOKEN, dynamic=True)
|
||||
scheme = QuantizationScheme(targets=["Linear"], weights=weights, input_activations=acts)
|
||||
disk = QuantizationConfig(
|
||||
config_groups={"group_0": scheme}, format=CompressionFormat.int_quantized,
|
||||
ignore=["img_in", "txt_in", "time_text_embed", "modulation", "norm_out", "proj_out"],
|
||||
quant_method="compressed-tensors", quantization_status=QuantizationStatus.FROZEN,
|
||||
).to_dict()
|
||||
quant_config = resolve_component_quant_config(
|
||||
build_quant_config({"transformer": disk}), "transformer"
|
||||
)
|
||||
|
||||
vllm.distributed.init_distributed_environment(
|
||||
backend="gloo", world_size=1, rank=0, local_rank=0,
|
||||
distributed_init_method="tcp://127.0.0.1:29602",
|
||||
)
|
||||
with set_current_vllm_config(VllmConfig()):
|
||||
initialize_model_parallel(tensor_model_parallel_size=1)
|
||||
|
||||
device = torch.device("cuda")
|
||||
torch.manual_seed(2301)
|
||||
tensors = {
|
||||
"q": synth(4096, 4096, args.ones),
|
||||
"k": synth(4096, 4096, args.ones),
|
||||
"v": synth(4096, 4096, args.ones),
|
||||
"to_out": synth(4096, 4096, args.ones),
|
||||
}
|
||||
|
||||
b = args.block
|
||||
p = f"transformer.transformer_blocks.{b}"
|
||||
results = []
|
||||
with torch.device(device.type):
|
||||
to_qkv = QKVParallelLinear(
|
||||
hidden_size=4096, head_size=128, total_num_heads=32, bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_qkv",
|
||||
)
|
||||
for part, shard in (("q", "q"), ("k", "k"), ("v", "v")):
|
||||
q, s = tensors[part]
|
||||
to_qkv.weight.weight_loader(to_qkv.weight, q.to(device), shard)
|
||||
to_qkv.weight_scale.weight_loader(to_qkv.weight_scale, s.to(device), shard)
|
||||
to_qkv.quant_method.process_weights_after_loading(to_qkv)
|
||||
|
||||
to_out = RowParallelLinear(
|
||||
4096, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_out.0",
|
||||
)
|
||||
q, s = tensors["to_out"]
|
||||
to_out.weight.weight_loader(to_out.weight, q.to(device))
|
||||
to_out.weight_scale.weight_loader(to_out.weight_scale, s.to(device))
|
||||
loaded_bitwise_ok = bool(torch.equal(to_out.weight.detach().cpu(), q))
|
||||
loaded_scale_ok = bool(torch.allclose(to_out.weight_scale.detach().float().cpu(), s))
|
||||
to_out.quant_method.process_weights_after_loading(to_out)
|
||||
|
||||
for name, layer, parts, in_features in (
|
||||
("attn.to_qkv (fused q/k/v)", to_qkv, ("q", "k", "v"), 4096),
|
||||
("attn.to_out.0 (row)", to_out, ("to_out",), 4096),
|
||||
):
|
||||
if args.ones:
|
||||
x = torch.ones(4, in_features, dtype=getattr(torch, args.act_dtype), device=device)
|
||||
else:
|
||||
x = torch.randn(512, in_features, dtype=getattr(torch, args.act_dtype), device=device)
|
||||
result = layer(x)
|
||||
out = (result[0] if isinstance(result, tuple) else result).float()
|
||||
|
||||
if args.ones:
|
||||
expected = torch.full_like(out, float(in_features))
|
||||
rel = float((out - expected).norm() / expected.norm())
|
||||
finite = bool(torch.isfinite(out).all())
|
||||
results.append({
|
||||
"layer": name,
|
||||
"rel_l2": round(rel, 6) if finite else "NaN",
|
||||
"out_finite": finite,
|
||||
"out_row0_first8": [round(v, 2) for v in out[0, :8].tolist()],
|
||||
})
|
||||
continue
|
||||
|
||||
ref_w = torch.cat([tensors[k][0].float() * tensors[k][1] for k in parts], dim=0)
|
||||
# Reference with the same dynamic per-token symmetric int8 activation
|
||||
# quantization the kernel applies (dequantized acts, dequantized weights).
|
||||
xs = x.float()
|
||||
act_scale = xs.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
x_dq = torch.round(xs / act_scale).clamp(-127, 127) * act_scale
|
||||
ref = x_dq @ ref_w.T.to(device)
|
||||
finite = bool(torch.isfinite(out).all())
|
||||
rel = float((out - ref).norm() / ref.norm())
|
||||
row = {"layer": name, "rel_l2": round(rel, 5) if finite else "NaN", "out_finite": finite}
|
||||
if name.startswith("attn.to_out"):
|
||||
row["loaded_bitwise_ok"] = loaded_bitwise_ok
|
||||
row["loaded_scale_ok"] = loaded_scale_ok
|
||||
# Reference computed from the layer's own post-repack weight
|
||||
# parameter ((K, N) after the transpose) and scales.
|
||||
w_layer = to_out.weight.detach().float() # (K, N)
|
||||
s_layer = to_out.weight_scale.detach().float().squeeze(1) # (N,)
|
||||
ref_layer = x_dq @ (w_layer * s_layer.unsqueeze(0)).to(device)
|
||||
row["rel_l2_vs_layer_weight"] = round(float((out - ref_layer).norm() / ref_layer.norm()), 5)
|
||||
results.append(row)
|
||||
|
||||
destroy_model_parallel()
|
||||
destroy_distributed_environment()
|
||||
print(json.dumps(results, indent=1))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,93 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Tests for the quantization config factory's method-name and disk-dict resolution."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors import (
|
||||
CompressedTensorsConfig,
|
||||
)
|
||||
|
||||
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||
from vllm_omni.quantization.component_config import (
|
||||
ComponentQuantizationConfig,
|
||||
is_prequantized_config,
|
||||
resolve_encoder_quant_config,
|
||||
)
|
||||
from vllm_omni.quantization.factory import SUPPORTED_QUANTIZATION_METHODS, _build_single
|
||||
|
||||
|
||||
def _int8_weight_only_disk_config() -> dict:
|
||||
"""Serialized compressed-tensors int8 weight-only metadata as it appears on disk."""
|
||||
weights = {
|
||||
"num_bits": 8,
|
||||
"type": "int",
|
||||
"symmetric": True,
|
||||
"strategy": "channel",
|
||||
"dynamic": False,
|
||||
"observer": "minmax",
|
||||
}
|
||||
return {
|
||||
"quant_method": "compressed-tensors",
|
||||
"format": "pack-quantized",
|
||||
"config_groups": {"group_0": {"targets": ["Linear"], "weights": weights}},
|
||||
"ignore": ["img_in"],
|
||||
"quantization_status": "frozen",
|
||||
}
|
||||
|
||||
|
||||
def test_hyphenated_registry_method_builds_from_disk_schema() -> None:
|
||||
# A bare str spec cannot construct compressed-tensors (a serialized-format
|
||||
# method with no parameterless default); the serialized schema must build.
|
||||
with pytest.raises(TypeError, match="Cannot instantiate CompressedTensorsConfig"):
|
||||
_build_single("compressed-tensors")
|
||||
disk = _int8_weight_only_disk_config()
|
||||
disk.pop("quant_method")
|
||||
config = _build_single("compressed-tensors", **disk)
|
||||
assert isinstance(config, CompressedTensorsConfig)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spelling", ["compressed-tensors", "compressed_tensors"])
|
||||
def test_disk_dict_builds_via_from_config(spelling: str) -> None:
|
||||
disk = _int8_weight_only_disk_config()
|
||||
disk["quant_method"] = spelling
|
||||
config = build_quant_config(disk)
|
||||
assert isinstance(config, CompressedTensorsConfig)
|
||||
|
||||
|
||||
def test_per_component_dict_routes_transformer_only() -> None:
|
||||
disk = {"transformer": _int8_weight_only_disk_config()}
|
||||
config = build_quant_config(disk)
|
||||
assert isinstance(config, ComponentQuantizationConfig)
|
||||
assert isinstance(
|
||||
resolve_component_quant_config(config, "transformer"), CompressedTensorsConfig
|
||||
)
|
||||
encoder = resolve_encoder_quant_config(
|
||||
resolve_component_quant_config(config, "text_encoder")
|
||||
)
|
||||
assert encoder is None
|
||||
|
||||
|
||||
def test_unknown_method_still_raises() -> None:
|
||||
with pytest.raises(ValueError, match="Unknown quantization method"):
|
||||
_build_single("not-a-real-method")
|
||||
|
||||
|
||||
def test_supported_list_advertises_hyphenated_registry_names() -> None:
|
||||
assert "compressed-tensors" in SUPPORTED_QUANTIZATION_METHODS
|
||||
|
||||
|
||||
def test_is_prequantized_config_recognizes_serialized_formats() -> None:
|
||||
# compressed-tensors is always a serialized format: its linear methods
|
||||
# consume packed checkpoint tensors and there is no on-load quantization.
|
||||
assert is_prequantized_config(build_quant_config(_int8_weight_only_disk_config()))
|
||||
component = build_quant_config({"transformer": _int8_weight_only_disk_config()})
|
||||
assert isinstance(component, ComponentQuantizationConfig)
|
||||
assert is_prequantized_config(component)
|
||||
assert not is_prequantized_config(None)
|
||||
# Online (non-serialized) methods stay unrecognized so the loader keeps
|
||||
# the accelerator-side load-quantize-offload path for them.
|
||||
from vllm_omni.quantization.fp8_config import DiffusionFp8Config
|
||||
|
||||
assert not is_prequantized_config(DiffusionFp8Config())
|
||||
@@ -63,7 +63,6 @@ from vllm_omni.diffusion.offloader.module_collector import ModuleDiscovery, Pipe
|
||||
from vllm_omni.diffusion.offloader.offload_plan import get_offload_plan
|
||||
from vllm_omni.diffusion.registry import initialize_model
|
||||
from vllm_omni.model_executor.model_loader.weight_utils import download_weights_from_hf_specific
|
||||
from vllm_omni.quantization.component_config import is_prequantized_config
|
||||
from vllm_omni.transformers_utils.repo_utils import hf_api
|
||||
|
||||
|
||||
@@ -696,17 +695,16 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
||||
}
|
||||
if load_format is None:
|
||||
load_format = "default"
|
||||
# CPU offload + quantization: for offline-quantized models (e.g., AutoRound MXFP8,
|
||||
# compressed-tensors), weights are already quantized in the checkpoint — load
|
||||
# directly on CPU. For online quantization, load on device so quantization can
|
||||
# run on accelerator, then move back to CPU afterward.
|
||||
# CPU offload + quantization: for offline-quantized models (e.g., AutoRound MXFP8),
|
||||
# weights are already quantized in the checkpoint — load directly on CPU.
|
||||
# For online quantization, load on device so quantization can run on accelerator,
|
||||
# then move back to CPU afterward.
|
||||
offload_after_quant = False
|
||||
offline_quant_direct_cpu = False
|
||||
if load_device == "cpu" and self.quant_config is not None and device is not None:
|
||||
quant_cfg = self.quant_config
|
||||
is_offline = getattr(quant_cfg, "data_type", None) == "mx_fp" or getattr(
|
||||
quant_cfg, "is_checkpoint_quantized", False
|
||||
) or is_prequantized_config(quant_cfg)
|
||||
)
|
||||
if not is_offline:
|
||||
load_device = device.type
|
||||
offload_after_quant = True
|
||||
@@ -715,7 +713,6 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
||||
load_device,
|
||||
)
|
||||
else:
|
||||
offline_quant_direct_cpu = True
|
||||
logger.info("Offline-quantized model with CPU offload, loading weights directly on CPU")
|
||||
|
||||
target_device = torch.device(load_device)
|
||||
@@ -877,14 +874,7 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
||||
else:
|
||||
self.load_weights(model)
|
||||
self._maybe_fuse_distilled_lora(model)
|
||||
# Post-load transforms (e.g. the Marlin repack for
|
||||
# compressed-tensors) only run on the accelerator, so an
|
||||
# offline-quantized CPU-offload load round-trips each
|
||||
# quantized module through the device and back to the host.
|
||||
process_device = (
|
||||
device if offline_quant_direct_cpu and device is not None else target_device
|
||||
)
|
||||
self._process_weights_after_loading(model, process_device)
|
||||
self._process_weights_after_loading(model, target_device)
|
||||
|
||||
# A warm final-layout hit has already completed all
|
||||
# byte-changing work through the restorer. Shared runtime
|
||||
|
||||
@@ -33,26 +33,6 @@ PRE_QUANTIZED_METHODS: frozenset[str] = frozenset(
|
||||
)
|
||||
|
||||
|
||||
def is_prequantized_config(quant_config: QuantizationConfig | None) -> bool:
|
||||
"""True when the config can only describe pre-quantized (serialized) weights.
|
||||
|
||||
Such checkpoints need no accelerator-side quantization pass at load, so a
|
||||
CPU-offload load can read weights directly on the host. compressed-tensors
|
||||
is the canonical case: its linear methods consume packed checkpoint tensors
|
||||
(weight_packed/weight_scale) directly and vLLM has no on-load quantization
|
||||
path for the format. A component wrapper is pre-quantized when every routed
|
||||
component (and the default) is itself pre-quantized or unquantized.
|
||||
"""
|
||||
if quant_config is None:
|
||||
return False
|
||||
if isinstance(quant_config, ComponentQuantizationConfig):
|
||||
routed = [*quant_config.component_configs.values(), quant_config.default_config]
|
||||
return any(c is not None for c in routed) and all(
|
||||
c is None or is_prequantized_config(c) for c in routed
|
||||
)
|
||||
return quant_config.get_name() == "compressed-tensors"
|
||||
|
||||
|
||||
def resolve_component_quant_config(
|
||||
quant_config: QuantizationConfig | None,
|
||||
component: str,
|
||||
|
||||
@@ -326,36 +326,20 @@ def _build_from_method_and_config(method: str, config: Mapping[str, Any]) -> Qua
|
||||
return _build_single(method, **config)
|
||||
|
||||
|
||||
def _resolve_registry_method_name(method: str) -> str | None:
|
||||
"""Resolve a normalized (underscore) method name to vLLM's registry spelling.
|
||||
|
||||
vLLM's ``QUANTIZATION_METHODS`` keeps canonical names with their historical
|
||||
spelling — including hyphenated ones such as ``compressed-tensors`` — while
|
||||
``_normalize_method_name`` canonicalizes dashes to underscores for the
|
||||
``_OVERRIDES`` lookup. Try both spellings so hyphenated registry entries
|
||||
remain reachable.
|
||||
"""
|
||||
for candidate in (method, method.replace("_", "-")):
|
||||
if candidate in QUANTIZATION_METHODS:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
||||
"""Build a single QuantizationConfig by method name.
|
||||
|
||||
Resolution: _OVERRIDES first, then vLLM registry via from_config().
|
||||
"""
|
||||
normalized = _normalize_method_name(method)
|
||||
method = _normalize_method_name(method)
|
||||
|
||||
if normalized in _OVERRIDES:
|
||||
return _OVERRIDES[normalized](**kwargs)
|
||||
if method in _OVERRIDES:
|
||||
return _OVERRIDES[method](**kwargs)
|
||||
|
||||
registry_method = _resolve_registry_method_name(normalized)
|
||||
if registry_method is None:
|
||||
if method not in QUANTIZATION_METHODS:
|
||||
raise ValueError(f"Unknown quantization method: {method!r}. Supported: {SUPPORTED_QUANTIZATION_METHODS}")
|
||||
|
||||
config_cls = get_quantization_config(registry_method)
|
||||
config_cls = get_quantization_config(method)
|
||||
|
||||
from vllm.config.quantization import resolve_quantization_config
|
||||
from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig
|
||||
@@ -363,7 +347,7 @@ def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
||||
if config_cls is OnlineQuantizationConfig:
|
||||
if "args" in kwargs:
|
||||
return config_cls(**kwargs)
|
||||
args = resolve_quantization_config(registry_method, kwargs or None)
|
||||
args = resolve_quantization_config(method, kwargs or None)
|
||||
if args is None:
|
||||
raise ValueError("Online quantization requires quantization config arguments")
|
||||
return config_cls(args)
|
||||
@@ -371,10 +355,6 @@ def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
||||
try:
|
||||
return config_cls(**kwargs)
|
||||
except TypeError:
|
||||
# Registry configs built from checkpoint metadata (e.g. compressed-tensors)
|
||||
# take the serialized schema through from_config rather than __init__.
|
||||
if kwargs and hasattr(config_cls, "from_config"):
|
||||
return config_cls.from_config(dict(kwargs))
|
||||
sig = inspect.signature(config_cls.__init__)
|
||||
raise TypeError(
|
||||
f"Cannot instantiate {config_cls.__name__} with kwargs {kwargs}. Expected signature: {sig}"
|
||||
|
||||
Reference in new issue
Block a user