probe(evidence): int8-wo capacity matrix, structure gate, PSNR, Marlin 8-bit NaN isolation (issue 5)
This commit is contained in:
admin committed
1 parent
74b596cd05
commit
bf65a41ceb
9 files changed
+1813
No files matched your search
@@ -0,0 +1,183 @@
|
||||
# Issue 5 probe: int8 weight-only compressed-tensors DiT on the RTX 3090 (24 GB)
|
||||
|
||||
Probed 2026-10-08 on `cachyos-alexander` (RTX 3090, sm_86, driver 615.71.09,
|
||||
CUDA 13.4), at upstream main `8484d85d1` plus this branch's two loader/factory
|
||||
fixes, vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||||
transformers 5.14.1, diffusers git `122b1e11`, compressed-tensors 0.17.0).
|
||||
The DiT was quantized **offline** from the sealed snapshot
|
||||
`Qwen/Qwen-Image-2.1@790c9263` to int8 weight-only (W8A16), per-output-channel
|
||||
symmetric, stored as a compressed-tensors pack-quantized checkpoint; encoder
|
||||
and VAE stay BF16 on disk via the artifact's per-component config wrapper.
|
||||
The probe mirrors the sealed Diffusers attempt `2026-10-07T170729Z-q3-4r`
|
||||
fixture-for-fixture (same frozen references-suite-2 images, same rendered
|
||||
prompts, seeds 2301–2304, 20 steps, `true_cfg_scale 1.0`).
|
||||
|
||||
## Verdict
|
||||
|
||||
**Capacity: delivered. Quality: collapsed in the serve kernel — the int8
|
||||
sidestep does not rescue 3–4 references on this card with this stack, and the
|
||||
blocker is now sharply isolated: the 8-bit Marlin W8A16 path itself.**
|
||||
|
||||
- Every step of the artifact chain is verified correct: the quantizer's
|
||||
round-trip reproduces the sealed BF16 weights at ~1.1% rel-L2
|
||||
(`evidence/rtx3090/artifact-verify.txt`), and the serve path loads the
|
||||
packed tensors **bitwise-identically** — including the fused q/k/v stacked
|
||||
shards — with finite scales before and after the Marlin repack.
|
||||
- Serving works mechanically: init ~40 s, all 16 capacity-matrix requests
|
||||
(1/2/3/4 references × seeds 2301–2304) complete at BF16-comparable speed
|
||||
(4-ref 46.4 s; BF16 OOMs at 4 refs, online-FP8 was 44.3 s), peak 23,935 MiB.
|
||||
- But the output is structurally broken on **every** counted run (gate FAIL
|
||||
16/16: opaque 0.0–0.20 vs 1.0 at BF16; PSNR vs the sealed baseline
|
||||
7.1–7.5 dB — the same collapsed band as online FP8). The images are
|
||||
horizontal-stripe noise.
|
||||
- A one-layer offline isolation (`marlin_isolate.py`) pins it: vLLM parallel
|
||||
linears built exactly as the qwen_image_21 transformer builds them, loaded
|
||||
with the artifact's verified tensors, emit **NaN** for all four layer
|
||||
geometries (fused QKV, to_out, img_mlp proj/out), for both bf16 and fp16
|
||||
activations — and do the same for **synthetic random int8 weights**, so the
|
||||
checkpoint and its quantization are exonerated. The 8-bit Marlin W8A16 path
|
||||
on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) is broken even though
|
||||
`query_marlin_supported_quant_types` accepts `uint8b128` on the card, and
|
||||
`choose_mp_linear_kernel` offers no non-Marlin W8A16 fallback on sm_86
|
||||
(Machete is sm90+, ExLlama int4-only).
|
||||
|
||||
The torchao-int8-in-Diffusers prior (multi-model-helper-server #160, blinded
|
||||
review within label noise of BF16) stands unchallenged: int8 quantization is
|
||||
fine; the serve-path kernel is not. The online-FP8 collapse from PR #4 and
|
||||
this int8 collapse now share one root suspect — the Marlin mixed-precision
|
||||
path on this fork/card — with the fp8 case previously isolated to the
|
||||
Marlin-quantized DiT and this run extending it to int8 with a fully verified
|
||||
checkpoint on the far side of the kernel.
|
||||
|
||||
## Two serve-path fixes this issue required (in this PR)
|
||||
|
||||
1. **Quant factory could not build compressed-tensors at all.**
|
||||
`_normalize_method_name` canonicalizes `compressed-tensors` to
|
||||
`compressed_tensors`, which then missed vLLM's hyphenated registry — so
|
||||
both `--quantization compressed-tensors` and checkpoint auto-detection
|
||||
raised "Unknown quantization method". The registry lookup now resolves
|
||||
both spellings, and registry configs whose `__init__` does not take the
|
||||
serialized schema fall back to `from_config` (`vllm_omni/quantization/factory.py`,
|
||||
tests in `tests/quantization/test_quant_factory.py`).
|
||||
2. **The diffusion loader misclassified pre-quantized checkpoints as online
|
||||
quantization under CPU offload.** With `--enable-cpu-offload` it loaded
|
||||
every component on the GPU before offloading (encoder 17 GiB + VAE + the
|
||||
packed DiT skeleton > 24 GiB → init OOM). Serialized formats are now
|
||||
detected (`is_prequantized_config`, component-wrapper aware) and load
|
||||
directly on the host, with each quantized module round-tripped through the
|
||||
accelerator only for the post-load Marlin repack
|
||||
(`vllm_omni/diffusion/model_loader/diffusers_loader.py`). With these two
|
||||
fixes the artifact serves: init healthy in ~40 s, logs
|
||||
`Per-component quantization: {'transformer': 'compressed-tensors'}`.
|
||||
|
||||
## Artifact (outside Git, under the helper state root)
|
||||
|
||||
`~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo/`
|
||||
— quantized `transformer/` (6.76 GiB packed int8 + BF16 boundary layers;
|
||||
224 tensors = 32 blocks × {to_q,to_k,to_v,to_out.0,proj,gate_layer,out};
|
||||
6.98 B params quantized, 0.14 B BF16), `model_index.json`/`scheduler`/
|
||||
`text_encoder`/`vae`/`processor` symlinked to the sealed snapshot, and
|
||||
`MANIFEST.json` with inputs (snapshot path, revision, tree digest), versions,
|
||||
per-file sha256, and quantization stats (mean rel-L2 1.10%, max 1.73%).
|
||||
Built by `quantize_dit_int8wo.py` using compressed-tensors' own
|
||||
`pack_to_int32` (uint8 biased by 128, 4 per int32) and `QuantizationConfig`
|
||||
serialization, so the storage convention matches llm-compressor bit-for-bit.
|
||||
`transformer/config.json` carries the quantization metadata wrapped
|
||||
per-component (`{"transformer": {...}}`) so auto-detection quantizes the DiT
|
||||
only — the encoder and VAE resolve to None and stay BF16
|
||||
(the plain global form would hand the encoder a compressed-tensors config
|
||||
it has no scales for).
|
||||
|
||||
## Run matrix — RTX 3090
|
||||
|
||||
Single run `int8wo`: `vllm serve <artifact> --omni --port 8091
|
||||
--served-model-name Qwen/Qwen-Image-2.1 --enforce-eager
|
||||
--enable-cpu-offload`, readiness via `/health`, VRAM via the 1 Hz
|
||||
`nvidia-smi` series (issue 2's instrument). Two precursor runs OOM'd at init
|
||||
for reasons fixed on this branch (see Verdict item 2); their logs survive
|
||||
under `runs/int8wo-expandable/` on the host (the first attempt's log was
|
||||
overwritten by the successful rerun in the same label).
|
||||
|
||||
| Refs | Fixture | Wall (mean, s) | Peak (MiB) | Gate | PSNR vs Diffusers (dB) |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | edit-expression-ysolde | 31.0 | 23,935 (run max) | FAIL 4/4 | 7.14 |
|
||||
| 2 | edit-compose-pair | 36.6 | ” | FAIL 4/4 | 7.46 |
|
||||
| 3 | edit-compose-three | 41.4 | ” | FAIL 4/4 | no sealed row (derived fixture) |
|
||||
| 4 | edit-compose-four | 46.4 | ” | FAIL 4/4 | 7.11 |
|
||||
|
||||
BF16 comparison rows (issue 2): 1-ref 34.2 s, 2-ref 36.7 s, 4-ref OOM.
|
||||
Online-fp8 (broken output, capacity reference): 29.9 / 34.1 / 38.9 / 44.3 s.
|
||||
So the int8 artifact matches the fp8 capacity/speed profile exactly —
|
||||
including its broken-output signature (PSNR band 7.0–7.5 dB) — while the
|
||||
checkpoint side is proven numerically sound, moving the defect wholly into
|
||||
the kernel.
|
||||
|
||||
## Structure gate and quality read
|
||||
|
||||
- Gate rule (issue 1's lesson): `opaque_fraction ≥ 0.99` and `mean(rgb_std)
|
||||
≥ 10`; BF16 outputs measure 1.0 opaque, 26–29 dB. Every int8-artifact run
|
||||
fails (opaque 0.0–0.20, alpha mean 114–137; rgb_std spans 3–93 — several
|
||||
outputs have healthy RGB spread yet near-total transparency, and one
|
||||
(seed 2302) is near-flat grey like the fp8 signature). Outputs are
|
||||
horizontal-stripe noise on visual inspection.
|
||||
- PSNR vs the sealed 4-reference Diffusers baseline (RGBA-on-RGBA,
|
||||
#1's comparator): 7.14 / 7.46 / 7.11 dB mean for 1/2/4 refs — the
|
||||
collapsed band, not the 26–29 dB BF16-row parity the issue hoped for.
|
||||
|
||||
## Kernel isolation (the issue's "different kernels" prior, answered)
|
||||
|
||||
`marlin_isolate.py` builds vLLM `QKVParallelLinear` / `RowParallelLinear` /
|
||||
`ColumnParallelLinear` exactly as `qwen_image_21_transformer.py` does, loads
|
||||
the artifact tensors through the production weight-loader machinery
|
||||
(bitwise-verified after load, fused q/k/v shards included), runs
|
||||
`process_weights_after_loading` (the Marlin repack) on the GPU, and compares
|
||||
`layer(x)` against `float32 x @ dequant(W)ᵀ` computed from the artifact's own
|
||||
unpacked weights:
|
||||
|
||||
| Layer | bf16 acts | fp16 acts | packed bitwise | scales post-repack |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| attn.to_qkv (fused) | NaN | NaN | equal | finite (6.6e-4…1.2e-2) |
|
||||
| attn.to_out.0 (row) | NaN | NaN | equal | finite |
|
||||
| img_mlp.proj (column) | NaN | NaN | equal | finite |
|
||||
| img_mlp.out (row, wide-in) | NaN | NaN | equal | finite |
|
||||
|
||||
Synthetic random int8 weights reproduce the NaN identically, and the runtime
|
||||
capability query accepts `uint8b128` on this card — so this is a defect of
|
||||
the 8-bit Marlin W8A16 path on this stack (repack or kernel; sm_86,
|
||||
vllm 0.31.0, torch 2.13.0+cu130), not of the checkpoint, the quantizer, the
|
||||
loader, or the fused-QKV stacking. No alternative W8A16 kernel exists on
|
||||
sm_86 in this build to switch to.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
Scripts in this directory: `quantize_dit_int8wo.py` (artifact + manifest),
|
||||
`verify_artifact.py` (round-trip check), `serve.sh` / `probe.py` (adapted
|
||||
issue-2 kit; probe records the structure gate inline per request), `gate.py`
|
||||
(gate + capacity summarizer), `psnr.py`, `marlin_isolate.py` (kernel
|
||||
isolation, `--synthetic` for the stack-only variant). All were pushed
|
||||
verbatim to `~/Projects/vllm-omni-val/issue5/` on the host and run from the
|
||||
validation clone checked out on this branch. Per-run serve/vram logs and all
|
||||
output PNGs live under `~/Projects/vllm-omni-val/issue5/runs/int8wo/`;
|
||||
condensed evidence in `evidence/rtx3090/`.
|
||||
|
||||
## Helper-system follow-through (input to multi-model-helper-server #27 — not implemented here)
|
||||
|
||||
- The 3090's 3–4 reference serving path remains blocked, now with a single
|
||||
named suspect: the 8-bit Marlin W8A16 kernel path on sm_86 (this fork's
|
||||
vllm 0.31.0 / torch 2.13.0 stack). Both quantized routes (online FP8,
|
||||
offline int8 compressed-tensors with a verified checkpoint) collapse at the
|
||||
same place. The unlock order for #27 candidacy is now: upstream Marlin
|
||||
8-bit fix (or a non-Marlin W8A16 kernel on sm_86), then serving — capacity
|
||||
and speed are already proven sufficient (4-ref at 46.4 s, 2.9× the sealed
|
||||
Diffusers nocache baseline).
|
||||
- The two vllm-omni loader/factory fixes on this branch are prerequisites for
|
||||
any compressed-tensors artifact serving under CPU offload and are
|
||||
independent of the kernel defect.
|
||||
|
||||
## Out-of-scope confirmation
|
||||
|
||||
No upstream PRs; no W8A8/int4/NVFP4 artifacts built; no helper-system
|
||||
deployment evidence. Root-causing the Marlin 8-bit defect remains the
|
||||
separate lane (PR #4 carries the fp8 isolation; this branch adds the int8 +
|
||||
synthetic + bitwise-load isolation). The only code changes are the two
|
||||
serve-path fixes required to make the issue's plan steps executable at all.
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"schema": "issue5-int8wo-v1",
|
||||
"created_utc": "2026-10-08T11:16:41+00:00",
|
||||
"description": "Qwen-Image-2.1 DiT int8 weight-only (W8A16), compressed-tensors pack-quantized",
|
||||
"inputs": {
|
||||
"snapshot": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"revision": "790c92633540aa0cb11d9abf19eb46d861714758",
|
||||
"tree_manifest": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/trees/790c92633540aa0cb11d9abf19eb46d861714758.json",
|
||||
"tree_sha256": "1f35d0a5a99f2c79bf2a6818b96582828094feb8544a992bc81c1d4100aed436"
|
||||
},
|
||||
"versions": {
|
||||
"python": "3.13.15",
|
||||
"torch": "2.13.0+cu130",
|
||||
"compressed_tensors": "0.17.0"
|
||||
},
|
||||
"quantization": {
|
||||
"method": "compressed-tensors",
|
||||
"format": "pack-quantized",
|
||||
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||
"config": {
|
||||
"config_groups": {
|
||||
"group_0": {
|
||||
"targets": [
|
||||
"Linear"
|
||||
],
|
||||
"weights": {
|
||||
"num_bits": 8,
|
||||
"type": "int",
|
||||
"symmetric": true,
|
||||
"group_size": null,
|
||||
"strategy": "channel",
|
||||
"block_structure": null,
|
||||
"dynamic": false,
|
||||
"actorder": null,
|
||||
"scale_dtype": null,
|
||||
"zp_dtype": null,
|
||||
"observer": "minmax",
|
||||
"observer_kwargs": {}
|
||||
},
|
||||
"input_activations": null,
|
||||
"output_activations": null,
|
||||
"format": null
|
||||
}
|
||||
},
|
||||
"quant_method": "compressed-tensors",
|
||||
"kv_cache_scheme": null,
|
||||
"format": "pack-quantized",
|
||||
"quantization_status": "frozen",
|
||||
"global_compression_ratio": null,
|
||||
"ignore": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
]
|
||||
},
|
||||
"ignored_layers": [
|
||||
"img_in",
|
||||
"txt_in",
|
||||
"time_text_embed",
|
||||
"modulation",
|
||||
"norm_out",
|
||||
"proj_out"
|
||||
],
|
||||
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers"
|
||||
},
|
||||
"stats": {
|
||||
"quantized_tensors": 224,
|
||||
"quantized_params": 6979321856,
|
||||
"bf16_params": 135802880,
|
||||
"mean_rel_l2": 0.010959165191577216,
|
||||
"max_rel_l2": 0.017284497618675232,
|
||||
"weights_file_bytes": 7256783616
|
||||
},
|
||||
"files": [
|
||||
{
|
||||
"path": "transformer/config.json",
|
||||
"bytes": 1405,
|
||||
"sha256": "43b8c94380afb473f252d2707d1398b2be180f0248acee31cad4d6613f4c0a84"
|
||||
},
|
||||
{
|
||||
"path": "transformer/diffusion_pytorch_model.safetensors",
|
||||
"bytes": 7256783616,
|
||||
"sha256": "947d9c0660776b1125ec98762817af55d59d326b8e223120aa1bdb0eced91aa9"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
transformer_blocks.0.attn.to_k.weight: shape (4096, 4096) rel_l2=0.01092 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_out.0.weight: shape (4096, 4096) rel_l2=0.00978 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_q.weight: shape (4096, 4096) rel_l2=0.01106 scale (4096, 1)/torch.float32
|
||||
transformer_blocks.0.attn.to_v.weight: shape (4096, 4096) rel_l2=0.00976 scale (4096, 1)/torch.float32
|
||||
checked 4 tensors
|
||||
@@ -0,0 +1,119 @@
|
||||
== int8wo ==
|
||||
1-ref: n=4 gate=FAIL wall 30.5-31.7s (mean 31.0s)
|
||||
edit-expression-ysolde s2301: 31.73s {"opaque_fraction": 0.0294, "alpha_mean": 114.2, "rgb_std": [59.9, 61.7, 63.8]} BROKEN
|
||||
edit-expression-ysolde s2302: 30.53s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.2, 2.9, 3.3]} BROKEN
|
||||
edit-expression-ysolde s2303: 30.88s {"opaque_fraction": 0.0391, "alpha_mean": 125.5, "rgb_std": [63.5, 66.2, 67.0]} BROKEN
|
||||
edit-expression-ysolde s2304: 30.95s {"opaque_fraction": 0.0352, "alpha_mean": 122.4, "rgb_std": [63.6, 66.3, 67.1]} BROKEN
|
||||
2-ref: n=4 gate=FAIL wall 36.5-36.7s (mean 36.6s)
|
||||
edit-compose-pair s2301: 36.46s {"opaque_fraction": 0.0303, "alpha_mean": 115.4, "rgb_std": [59.2, 61.0, 63.3]} BROKEN
|
||||
edit-compose-pair s2302: 36.57s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.1, 2.9, 3.2]} BROKEN
|
||||
edit-compose-pair s2303: 36.61s {"opaque_fraction": 0.03, "alpha_mean": 115.7, "rgb_std": [76.1, 77.9, 69.1]} BROKEN
|
||||
edit-compose-pair s2304: 36.69s {"opaque_fraction": 0.1257, "alpha_mean": 127.1, "rgb_std": [64.3, 64.3, 64.3]} BROKEN
|
||||
3-ref: n=4 gate=FAIL wall 41.3-41.6s (mean 41.4s)
|
||||
edit-compose-three s2301: 41.57s {"opaque_fraction": 0.2029, "alpha_mean": 137.2, "rgb_std": [93.0, 92.7, 91.0]} BROKEN
|
||||
edit-compose-three s2302: 41.32s {"opaque_fraction": 0.0193, "alpha_mean": 131.9, "rgb_std": [40.8, 45.8, 44.3]} BROKEN
|
||||
edit-compose-three s2303: 41.34s {"opaque_fraction": 0.0188, "alpha_mean": 131.6, "rgb_std": [41.0, 45.9, 44.6]} BROKEN
|
||||
edit-compose-three s2304: 41.39s {"opaque_fraction": 0.0189, "alpha_mean": 131.6, "rgb_std": [41.1, 46.0, 44.6]} BROKEN
|
||||
4-ref: n=4 gate=FAIL wall 46.4-46.5s (mean 46.4s)
|
||||
edit-compose-four s2301: 46.4s {"opaque_fraction": 0.0195, "alpha_mean": 131.6, "rgb_std": [41.4, 46.3, 45.0]} BROKEN
|
||||
edit-compose-four s2302: 46.42s {"opaque_fraction": 0.0196, "alpha_mean": 131.9, "rgb_std": [41.1, 46.0, 44.5]} BROKEN
|
||||
edit-compose-four s2303: 46.44s {"opaque_fraction": 0.0195, "alpha_mean": 132.0, "rgb_std": [41.1, 46.1, 44.7]} BROKEN
|
||||
edit-compose-four s2304: 46.48s {"opaque_fraction": 0.0195, "alpha_mean": 131.7, "rgb_std": [41.0, 46.1, 44.5]} BROKEN
|
||||
nvidia-smi peak: 23935 MiB; server peak: None MB
|
||||
STRUCTURE GATE: FAIL
|
||||
== psnr edit-expression-ysolde ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 6.95,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 8.3,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 6.67,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 6.65,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.14,
|
||||
"min_db": 6.65,
|
||||
"max_db": 8.3,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-pair ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.03,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 8.86,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 6.77,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 7.19,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.46,
|
||||
"min_db": 6.77,
|
||||
"max_db": 8.86,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
== psnr edit-compose-four ==
|
||||
{
|
||||
"int8wo": {
|
||||
"rows": [
|
||||
{
|
||||
"seed": 2301,
|
||||
"psnr_db": 7.02,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2302,
|
||||
"psnr_db": 7.11,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2303,
|
||||
"psnr_db": 7.1,
|
||||
"modes": "RGBA/RGBA"
|
||||
},
|
||||
{
|
||||
"seed": 2304,
|
||||
"psnr_db": 7.2,
|
||||
"modes": "RGBA/RGBA"
|
||||
}
|
||||
],
|
||||
"mean_db": 7.11,
|
||||
"min_db": 7.02,
|
||||
"max_db": 7.2,
|
||||
"n": 4
|
||||
}
|
||||
}
|
||||
23935 MiB, 100 %
|
||||
(APIServer pid=473436) INFO 10-08 07:36:28 [data.py:1483] Auto-detected quantization 'component' from model config
|
||||
(DiffusionWorker pid=473436) INFO 10-08 07:36:28 [diffusers_loader.py:719] Offline-quantized model with CPU offload, loading weights directly on CPU
|
||||
@@ -0,0 +1,82 @@
|
||||
[
|
||||
{
|
||||
"layer": "attn.to_qkv (fused q/k/v)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0006612942670471966,
|
||||
"max": 0.012303149327635765
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
12288
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
12288
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "attn.to_out.0 (row)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.00037870631786063313,
|
||||
"max": 0.011257382109761238
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
4096
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
4096
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.proj (column)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0005651759565807879,
|
||||
"max": 0.005628691054880619
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
12288
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
12288
|
||||
]
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.out (row, wide-in)",
|
||||
"rel_l2": "NaN",
|
||||
"out_finite": false,
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.0008073942153714597,
|
||||
"max": 0.014763779938220978
|
||||
},
|
||||
"out_shape": [
|
||||
512,
|
||||
4096
|
||||
],
|
||||
"ref_shape": [
|
||||
512,
|
||||
4096
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,58 @@
|
||||
[
|
||||
{
|
||||
"layer": "attn.to_qkv (fused q/k/v)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02410734072327614,
|
||||
"max": 0.043380916118621826
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "attn.to_out.0 (row)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02417764998972416,
|
||||
"max": 0.042702700942754745
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.proj (column)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.024067752063274384,
|
||||
"max": 0.04336588457226753
|
||||
}
|
||||
},
|
||||
{
|
||||
"layer": "img_mlp.out (row, wide-in)",
|
||||
"rel_l2_by_act_dtype": {
|
||||
"bfloat16": "NaN",
|
||||
"float16": "NaN"
|
||||
},
|
||||
"packed_bitwise_ok": true,
|
||||
"scale_finite_pre_repack": true,
|
||||
"scale_post_repack": {
|
||||
"finite": true,
|
||||
"min": 0.02682814560830593,
|
||||
"max": 0.04223968833684921
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,978 @@
|
||||
[
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2301,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 31.73,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "a62cc903858a5026c08f284bdfea58108068fd6ab6425b54c246d65990e0bec4",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0294,
|
||||
"alpha_mean": 114.2,
|
||||
"rgb_std": [
|
||||
59.9,
|
||||
61.7,
|
||||
63.8
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 31616.730451583862,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1580.8365225791931,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 31616.933584213257,
|
||||
"image_time_to_first_output_ms": 31616.7151927948,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.18030800129054114,
|
||||
"stage_0_gen_ms": 31616.730451583862
|
||||
},
|
||||
"peak_memory_mb": 21594.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2302,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.53,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "ae5c51a9a36f8892c15116b90bcd2b85f90a83bae3daea2196f2602b5968e9af",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0,
|
||||
"alpha_mean": 126.9,
|
||||
"rgb_std": [
|
||||
3.2,
|
||||
2.9,
|
||||
3.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30463.57011795044,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1523.178505897522,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30463.69981765747,
|
||||
"image_time_to_first_output_ms": 30463.540077209473,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.12335599603829905,
|
||||
"stage_0_gen_ms": 30463.57011795044
|
||||
},
|
||||
"peak_memory_mb": 21900.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2303,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.88,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "d05825c0d525ff8522d83db2a9fc6337cc06a3960f7eb2a112898c5b71aa34a1",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0391,
|
||||
"alpha_mean": 125.5,
|
||||
"rgb_std": [
|
||||
63.5,
|
||||
66.2,
|
||||
67.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30779.66594696045,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1538.9832973480225,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30779.80923652649,
|
||||
"image_time_to_first_output_ms": 30779.641151428223,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14197500422596931,
|
||||
"stage_0_gen_ms": 30779.66594696045
|
||||
},
|
||||
"peak_memory_mb": 21906.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-expression-ysolde",
|
||||
"seed": 2304,
|
||||
"n_references": 1,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "864x1152",
|
||||
"wall_s": 30.95,
|
||||
"http": 200,
|
||||
"bytes": 3983827,
|
||||
"sha256": "10e85cf24451f4fb7bf2d8e07b4039158af08e36467a2a14d2baee1d68cd98fa",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0352,
|
||||
"alpha_mean": 122.4,
|
||||
"rgb_std": [
|
||||
63.6,
|
||||
66.3,
|
||||
67.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 30871.540069580078,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 995328,
|
||||
"denoise_step_latency_ms": 1543.577003479004,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 30871.644735336304,
|
||||
"image_time_to_first_output_ms": 30871.514797210693,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10297299741068855,
|
||||
"stage_0_gen_ms": 30871.540069580078
|
||||
},
|
||||
"peak_memory_mb": 21906.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2301,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.46,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "4bfdf57ecb0e1ed1c2c4dcb10ce0545d89040ad08ed754e01adbe2ddd2b407d2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0303,
|
||||
"alpha_mean": 115.4,
|
||||
"rgb_std": [
|
||||
59.2,
|
||||
61.0,
|
||||
63.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36376.55758857727,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1818.8278794288635,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36376.71780586243,
|
||||
"image_time_to_first_output_ms": 36376.53112411499,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15923400496831164,
|
||||
"stage_0_gen_ms": 36376.55758857727
|
||||
},
|
||||
"peak_memory_mb": 22676.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2302,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.57,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "e5ad17605624fdb636dc02e272c0dba2d3286dcfa4c27f297a605e785f3177a2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0,
|
||||
"alpha_mean": 126.9,
|
||||
"rgb_std": [
|
||||
3.1,
|
||||
2.9,
|
||||
3.2
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36508.753061294556,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1825.4376530647278,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36508.851528167725,
|
||||
"image_time_to_first_output_ms": 36508.72349739075,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1011220010695979,
|
||||
"stage_0_gen_ms": 36508.753061294556
|
||||
},
|
||||
"peak_memory_mb": 22556.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2303,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.61,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "f58a982cf1f3afd4a81fcc6b349aa3b147fecea0594e33f49cab5673f300b2ac",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.03,
|
||||
"alpha_mean": 115.7,
|
||||
"rgb_std": [
|
||||
76.1,
|
||||
77.9,
|
||||
69.1
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36523.38361740112,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1826.1691808700562,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36523.181676864624,
|
||||
"image_time_to_first_output_ms": 36523.05722236633,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.0983119971351698,
|
||||
"stage_0_gen_ms": 36523.38361740112
|
||||
},
|
||||
"peak_memory_mb": 22554.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-pair",
|
||||
"seed": 2304,
|
||||
"n_references": 2,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1248x832",
|
||||
"wall_s": 36.69,
|
||||
"http": 200,
|
||||
"bytes": 4155590,
|
||||
"sha256": "871c8147b27310508fb2c9e72b3b47499d7f8ab4ea705719aa30c465c8a5476a",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.1257,
|
||||
"alpha_mean": 127.1,
|
||||
"rgb_std": [
|
||||
64.3,
|
||||
64.3,
|
||||
64.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 36608.7908744812,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1038336,
|
||||
"denoise_step_latency_ms": 1830.43954372406,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 36608.89911651611,
|
||||
"image_time_to_first_output_ms": 36608.76417160034,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10716100223362446,
|
||||
"stage_0_gen_ms": 36608.7908744812
|
||||
},
|
||||
"peak_memory_mb": 22556.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2301,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.57,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "810cb635438bac76de76fa44172687e2f18b212379458580a32bbddd028bd78c",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.2029,
|
||||
"alpha_mean": 137.2,
|
||||
"rgb_std": [
|
||||
93.0,
|
||||
92.7,
|
||||
91.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41476.71937942505,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2073.8359689712524,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41476.8271446228,
|
||||
"image_time_to_first_output_ms": 41476.691246032715,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1088700009859167,
|
||||
"stage_0_gen_ms": 41476.71937942505
|
||||
},
|
||||
"peak_memory_mb": 23468.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2302,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.32,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "2f085a5089ab0618a2f17df7c29c52fde091247cb4a1ccf0cd8d662171b18fc9",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0193,
|
||||
"alpha_mean": 131.9,
|
||||
"rgb_std": [
|
||||
40.8,
|
||||
45.8,
|
||||
44.3
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41233.43777656555,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2061.6718888282776,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41233.57892036438,
|
||||
"image_time_to_first_output_ms": 41233.40702056885,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14587400073651224,
|
||||
"stage_0_gen_ms": 41233.43777656555
|
||||
},
|
||||
"peak_memory_mb": 23460.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2303,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.34,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "81e9b7436d392c8420d1529976a0d88e5de03af1e70823cc8c7b279b3d7bb3fa",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0188,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.0,
|
||||
45.9,
|
||||
44.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41250.11157989502,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2062.505578994751,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41250.218868255615,
|
||||
"image_time_to_first_output_ms": 41250.08296966553,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.10969900176860392,
|
||||
"stage_0_gen_ms": 41250.11157989502
|
||||
},
|
||||
"peak_memory_mb": 23394.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-three",
|
||||
"seed": 2304,
|
||||
"n_references": 3,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 41.39,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "2221932a1783102d87c01157e231160803cebac2113c86b9416e537e0e738747",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0189,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.0,
|
||||
44.6
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 41297.605991363525,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2064.8802995681763,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 41297.31011390686,
|
||||
"image_time_to_first_output_ms": 41297.15037345886,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.134524998429697,
|
||||
"stage_0_gen_ms": 41297.605991363525
|
||||
},
|
||||
"peak_memory_mb": 23394.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2301,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.4,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "abd057bee84e594437f895cc9f51f90f118adfeb4dbf738686791b41487e93b2",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 131.6,
|
||||
"rgb_std": [
|
||||
41.4,
|
||||
46.3,
|
||||
45.0
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46304.86750602722,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2315.243375301361,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46304.97694015503,
|
||||
"image_time_to_first_output_ms": 46304.8369884491,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.11391400039428845,
|
||||
"stage_0_gen_ms": 46304.86750602722
|
||||
},
|
||||
"peak_memory_mb": 23534.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2302,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.42,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "24a783a9ca3a002745351476e8000be87854d1364ce92ca944c2986a983922f5",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0196,
|
||||
"alpha_mean": 131.9,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.0,
|
||||
44.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46325.687408447266,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2316.2843704223633,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46324.424266815186,
|
||||
"image_time_to_first_output_ms": 46324.246883392334,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.15303100371966138,
|
||||
"stage_0_gen_ms": 46325.687408447266
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2303,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.44,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "db507c15651324404b79d6e5446e43fb460797204b302c0c258be37f1e2a7a80",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 132.0,
|
||||
"rgb_std": [
|
||||
41.1,
|
||||
46.1,
|
||||
44.7
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46352.77342796326,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2317.638671398163,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46352.887868881226,
|
||||
"image_time_to_first_output_ms": 46352.74004936218,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.1095690022339113,
|
||||
"stage_0_gen_ms": 46352.77342796326
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"fixture_id": "edit-compose-four",
|
||||
"seed": 2304,
|
||||
"n_references": 4,
|
||||
"steps": 20,
|
||||
"true_cfg_scale": 1.0,
|
||||
"size": "1344x768",
|
||||
"wall_s": 46.48,
|
||||
"http": 200,
|
||||
"bytes": 4130915,
|
||||
"sha256": "5481e029af2cb8ef3140565c615a5abc97ed755ff180b1970badf3a997c60ac4",
|
||||
"structure": {
|
||||
"opaque_fraction": 0.0195,
|
||||
"alpha_mean": 131.7,
|
||||
"rgb_std": [
|
||||
41.0,
|
||||
46.1,
|
||||
44.5
|
||||
]
|
||||
},
|
||||
"usage": null,
|
||||
"metrics": {
|
||||
"stage_metrics": {
|
||||
"0": {
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"finish_reason": null,
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_gen_time_ms": 46385.611057281494,
|
||||
"postprocess_time_ms": 0.0,
|
||||
"audio_frames": 0,
|
||||
"audio_sample_rate": 0,
|
||||
"audio_duration_s": 0.0,
|
||||
"image_pixels": 1032192,
|
||||
"denoise_step_latency_ms": 2319.2805528640747,
|
||||
"output_unit_type": "image",
|
||||
"output_unit_count": 1,
|
||||
"serving_time_to_first_output_ms": 46385.305404663086,
|
||||
"image_time_to_first_output_ms": 46385.130643844604,
|
||||
"time_per_output_unit_ms": 0.0,
|
||||
"inter_output_latency_ms": 0.0,
|
||||
"inter_output_latencies_ms": [],
|
||||
"vllm_ttft_ms": 0.0,
|
||||
"vllm_tpot_ms": 0.0,
|
||||
"vllm_itl_ms": 0.0,
|
||||
"vllm_itls_ms": [],
|
||||
"stage_name": "diffusion"
|
||||
}
|
||||
},
|
||||
"stage_id": 0,
|
||||
"final_output_type": "image",
|
||||
"num_tokens_in": 0,
|
||||
"num_tokens_out": 0,
|
||||
"stage_durations": {
|
||||
"queue_wait_ms": 0.14768500113859773,
|
||||
"stage_0_gen_ms": 46385.611057281494
|
||||
},
|
||||
"peak_memory_mb": 23524.0
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"schema": "issue5-runs-summary-v1",
|
||||
"probed_utc": "2026-10-08",
|
||||
"host": "cachyos-alexander (RTX 3090 24 GB, sm_86, driver 615.71.09, CUDA 13.4)",
|
||||
"stack": {
|
||||
"vllm_omni": "ISSUE-5 branch e778d21f..67e17786 over main 8484d85d1",
|
||||
"vllm": "0.31.0",
|
||||
"torch": "2.13.0+cu130",
|
||||
"compressed_tensors": "0.17.0",
|
||||
"transformers": "5.14.1",
|
||||
"diffusers": "0.41.0.dev0 (git 122b1e11)"
|
||||
},
|
||||
"artifact": {
|
||||
"path": "~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo",
|
||||
"quantized_params": 6979321856,
|
||||
"bf16_boundary_params": 135802880,
|
||||
"weights_file_bytes": 7256783616,
|
||||
"mean_rel_l2_quant_error": 0.010959,
|
||||
"max_rel_l2_quant_error": 0.017284,
|
||||
"roundtrip_verification": "unpack+dequant vs sealed BF16: rel_l2 0.0098-0.0117 per tensor (see artifact-verify.txt)"
|
||||
},
|
||||
"serve_run": {
|
||||
"label": "int8wo",
|
||||
"command": "vllm serve <artifact> --omni --port 8091 --served-model-name Qwen/Qwen-Image-2.1 --enforce-eager --enable-cpu-offload",
|
||||
"init": "healthy in ~40 s; per-component quantization auto-detected from transformer/config.json (transformer -> compressed-tensors, encoder/VAE BF16); CompressedTensorsWNA16 + MarlinLinearKernel selected on sm_86",
|
||||
"failed_precursor_runs": [
|
||||
"int8wo (first): init OOM - loader treated the pre-quantized checkpoint as online quantization and loaded all components on the GPU (encoder 17 GiB + VAE + packed DiT skeleton > 24 GiB); fixed by the offline-detection loader change on this branch",
|
||||
"int8wo-expandable: same OOM with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (genuine capacity, not fragmentation); also polluted by a leftover 586 MiB DiffusionWorker from an interrupted test run"
|
||||
]
|
||||
},
|
||||
"capacity_matrix": {
|
||||
"fixtures": {"1-ref": "edit-expression-ysolde", "2-ref": "edit-compose-pair", "3-ref": "edit-compose-three", "4-ref": "edit-compose-four"},
|
||||
"seeds": [2301, 2302, 2303, 2304],
|
||||
"requests": 16,
|
||||
"http_200": 16,
|
||||
"wall_s_mean": {"1-ref": 31.0, "2-ref": 36.6, "3-ref": 41.4, "4-ref": 46.4},
|
||||
"nvidia_smi_peak_mib": 23935,
|
||||
"bf16_reference_walls": {"1-ref": 34.2, "2-ref": 36.7, "4-ref": "OOM at BF16 (issue 2)"},
|
||||
"fp8_reference_walls": {"1-ref": 29.9, "2-ref": 34.1, "3-ref": 38.9, "4-ref": 44.3}
|
||||
},
|
||||
"structure_gate": {
|
||||
"rule": "opaque_fraction >= 0.99 AND mean(rgb_std) >= 10",
|
||||
"result": "FAIL 16/16",
|
||||
"signature": "opaque 0.0-0.20 (vs 1.0 at BF16), alpha mean 114-137, rgb_std 3-93; outputs are horizontal-stripe noise, not rendered scenes",
|
||||
"detail": "gate-and-psnr-summary.txt; per-request records in matrix.json"
|
||||
},
|
||||
"psnr_vs_sealed_diffusers_baseline": {
|
||||
"edit-expression-ysolde_1ref_mean_db": 7.14,
|
||||
"edit-compose-pair_2ref_mean_db": 7.46,
|
||||
"edit-compose-four_4ref_mean_db": 7.11,
|
||||
"bf16_pair_reference_mean_db": 27.57,
|
||||
"fp8_broken_reference_mean_db": 7.11
|
||||
},
|
||||
"marlin_isolation": {
|
||||
"method": "marlin_isolate.py: vLLM QKVParallelLinear/RowParallelLinear/ColumnParallelLinear built exactly as the qwen_image_21 transformer does; artifact tensors loaded through the production weight-loaders; Marlin repack on GPU; layer(x) compared against float32 x @ dequant(W)^T",
|
||||
"results_block0_artifact": "NaN output, all four layer kinds, bf16 AND fp16 activations; loaded weight_packed bitwise-equal to artifact (incl. fused q/k/v stacked shards); scales finite pre and post repack (marlin-isolate-block0.json)",
|
||||
"results_synthetic_random_int8": "NaN identically - the checkpoint and its quantization are exonerated (marlin-isolate-synthetic.json)",
|
||||
"conclusion": "the 8-bit Marlin W8A16 path on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) produces NaN for int8 weight-only layers although query_marlin_supported_quant_types accepts uint8b128 on the card; choose_mp_linear_kernel offers no non-Marlin fallback for W8A16 on sm_86 (Machete is sm90+, ExLlama int4-only)"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env python
|
||||
"""Issue 5 diagnostic: isolate the compressed-tensors W8A16 serve path per layer.
|
||||
|
||||
Builds vLLM parallel linear layers exactly as the qwen_image_21 transformer
|
||||
does (fused QKV, row-parallel to_out, img_mlp proj/out), loads the int8
|
||||
artifact's packed tensors through the production weight-loader machinery,
|
||||
runs the Marlin repack (process_weights_after_loading) on the GPU, and
|
||||
compares each layer's forward output against a float32 reference computed
|
||||
from the artifact's own unpacked+dequantized weights.
|
||||
|
||||
Expected rel-L2 when the load->repack->apply chain is correct: ~1-2%
|
||||
(int8 quantization error plus bf16 activation rounding). A large error
|
||||
pins corruption inside the serve path with the checkpoint already
|
||||
verified correct (verify_artifact.py).
|
||||
|
||||
Usage: marlin_isolate.py --artifact ARTIFACT_DIR [--block N]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
|
||||
import torch
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
|
||||
from safetensors.torch import load_file
|
||||
|
||||
|
||||
def dequantized_weight(art: dict, base: str) -> torch.Tensor:
|
||||
shape = tuple(art[f"{base}_shape"].tolist())
|
||||
q = unpack_from_int32(art[f"{base}_packed"].cpu(), 8, shape)
|
||||
return q.to(torch.float32) * art[f"{base}_scale"].cpu().to(torch.float32)
|
||||
|
||||
|
||||
def load_tensor(layer: torch.nn.Module, param_name: str, tensor: torch.Tensor, shard: str | None) -> None:
|
||||
param = getattr(layer, param_name)
|
||||
if shard is None:
|
||||
param.weight_loader(param, tensor)
|
||||
else:
|
||||
param.weight_loader(param, tensor, shard)
|
||||
|
||||
|
||||
def check(name: str, layer: torch.nn.Module, art: dict, base: str, shards: list[tuple[str, str]], in_features: int, dtype: torch.dtype) -> dict:
|
||||
device = torch.device("cuda")
|
||||
for suffix in ("weight_packed", "weight_scale", "weight_shape"):
|
||||
for ckpt_base, shard in shards:
|
||||
load_tensor(layer, suffix, art[f"{ckpt_base}.{suffix}"].to(device), shard)
|
||||
|
||||
# Bitwise check of the loaded packed weights against the artifact.
|
||||
loaded = layer.weight_packed.detach().cpu()
|
||||
if len(shards) > 1:
|
||||
ref_packed = torch.cat([art[f"{b}.weight_packed"] for b, _ in shards], dim=0)
|
||||
else:
|
||||
ref_packed = art[f"{base}.weight_packed"]
|
||||
packed_ok = bool(torch.equal(loaded, ref_packed))
|
||||
scale_loaded = layer.weight_scale.detach().float()
|
||||
scale_ok = bool(torch.isfinite(scale_loaded).all())
|
||||
|
||||
layer.quant_method.process_weights_after_loading(layer)
|
||||
repacked_scale = getattr(layer, "weight_scale", None)
|
||||
scale_post = (
|
||||
{
|
||||
"finite": bool(torch.isfinite(repacked_scale.detach().float()).all()),
|
||||
"min": float(repacked_scale.detach().float().min()),
|
||||
"max": float(repacked_scale.detach().float().max()),
|
||||
}
|
||||
if repacked_scale is not None
|
||||
else {}
|
||||
)
|
||||
|
||||
torch.manual_seed(2301)
|
||||
ref_w = torch.cat([dequantized_weight(art, f"{b}.weight") for b, _ in shards], dim=0) if len(shards) > 1 else dequantized_weight(art, f"{base}.weight")
|
||||
outs = {}
|
||||
for act_dtype in (torch.bfloat16, torch.float16):
|
||||
x = torch.randn(512, in_features, dtype=act_dtype, device=device)
|
||||
result = layer(x)
|
||||
out = (result[0] if isinstance(result, tuple) else result).float()
|
||||
ref = x.float() @ ref_w.T.to(device)
|
||||
finite = bool(torch.isfinite(out).all())
|
||||
rel = float((out - ref).norm() / ref.norm())
|
||||
outs[str(act_dtype).replace("torch.", "")] = rel if finite else "NaN"
|
||||
|
||||
return {
|
||||
"layer": name,
|
||||
"rel_l2_by_act_dtype": outs,
|
||||
"packed_bitwise_ok": packed_ok,
|
||||
"scale_finite_pre_repack": scale_ok,
|
||||
"scale_post_repack": scale_post,
|
||||
}
|
||||
|
||||
|
||||
def synthetic_artifact(block: int) -> dict[str, torch.Tensor]:
|
||||
"""Random int8-quantized tensors in the artifact's layout, for stack-only isolation."""
|
||||
from compressed_tensors.compressors.pack_quantized.helpers import pack_to_int32
|
||||
|
||||
def quant(name: str, out: int, inp: int) -> None:
|
||||
w = torch.randn(out, inp, dtype=torch.float32)
|
||||
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||
art[f"{name}.weight_packed"] = pack_to_int32(q.contiguous(), 8).contiguous()
|
||||
art[f"{name}.weight_scale"] = scale.to(torch.float32).contiguous()
|
||||
art[f"{name}.weight_shape"] = torch.tensor([out, inp], dtype=torch.int64)
|
||||
|
||||
art: dict[str, torch.Tensor] = {}
|
||||
p = f"transformer_blocks.{block}"
|
||||
for part in ("to_q", "to_k", "to_v"):
|
||||
quant(f"{p}.attn.{part}", 4096, 4096)
|
||||
quant(f"{p}.attn.to_out.0", 4096, 4096)
|
||||
quant(f"{p}.img_mlp.proj", 12288, 4096)
|
||||
quant(f"{p}.img_mlp.out", 4096, 12288)
|
||||
return art
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--artifact", type=pathlib.Path, required=True)
|
||||
parser.add_argument("--block", type=int, default=0)
|
||||
parser.add_argument(
|
||||
"--synthetic",
|
||||
action="store_true",
|
||||
help="ignore --artifact; use random int8 weights (isolates the stack from the checkpoint)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
import vllm.distributed
|
||||
from vllm.config import VllmConfig, set_current_vllm_config
|
||||
from vllm.distributed import (
|
||||
destroy_distributed_environment,
|
||||
destroy_model_parallel,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
from vllm.model_executor.layers.linear import (
|
||||
ColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
|
||||
vllm.distributed.init_distributed_environment(
|
||||
backend="gloo",
|
||||
world_size=1,
|
||||
rank=0,
|
||||
local_rank=0,
|
||||
distributed_init_method="tcp://127.0.0.1:29601",
|
||||
)
|
||||
with set_current_vllm_config(VllmConfig()):
|
||||
initialize_model_parallel(tensor_model_parallel_size=1)
|
||||
|
||||
config = json.loads((args.artifact / "transformer" / "config.json").read_text())
|
||||
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||
|
||||
quant_config = resolve_component_quant_config(
|
||||
build_quant_config(config["quantization_config"]), "transformer"
|
||||
)
|
||||
if args.synthetic:
|
||||
art = synthetic_artifact(args.block)
|
||||
else:
|
||||
art = load_file(str(args.artifact / "transformer" / "diffusion_pytorch_model.safetensors"))
|
||||
b = args.block
|
||||
p = f"transformer.transformer_blocks.{b}"
|
||||
|
||||
results = []
|
||||
with torch.device("cuda"):
|
||||
to_qkv = QKVParallelLinear(
|
||||
hidden_size=4096, head_size=128, total_num_heads=32, bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_qkv",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"attn.to_qkv (fused q/k/v)",
|
||||
to_qkv,
|
||||
art,
|
||||
f"transformer_blocks.{b}.attn.to_qkv",
|
||||
[
|
||||
(f"transformer_blocks.{b}.attn.to_q", "q"),
|
||||
(f"transformer_blocks.{b}.attn.to_k", "k"),
|
||||
(f"transformer_blocks.{b}.attn.to_v", "v"),
|
||||
],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
to_out = RowParallelLinear(
|
||||
4096, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.attn.to_out.0",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"attn.to_out.0 (row)",
|
||||
to_out,
|
||||
art,
|
||||
f"transformer_blocks.{b}.attn.to_out.0",
|
||||
[(f"transformer_blocks.{b}.attn.to_out.0", None)],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
proj = ColumnParallelLinear(
|
||||
4096, 12288, bias=False, gather_output=False, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.img_mlp.proj",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"img_mlp.proj (column)",
|
||||
proj,
|
||||
art,
|
||||
f"transformer_blocks.{b}.img_mlp.proj",
|
||||
[(f"transformer_blocks.{b}.img_mlp.proj", None)],
|
||||
4096,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
mlp_out = RowParallelLinear(
|
||||
12288, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||
quant_config=quant_config, prefix=f"{p}.img_mlp.out",
|
||||
)
|
||||
results.append(
|
||||
check(
|
||||
"img_mlp.out (row, wide-in)",
|
||||
mlp_out,
|
||||
art,
|
||||
f"transformer_blocks.{b}.img_mlp.out",
|
||||
[(f"transformer_blocks.{b}.img_mlp.out", None)],
|
||||
12288,
|
||||
torch.bfloat16,
|
||||
)
|
||||
)
|
||||
|
||||
destroy_model_parallel()
|
||||
destroy_distributed_environment()
|
||||
print(json.dumps(results, indent=1))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user