Int8 weight-only compressed-tensors DiT artifact + serve-path fixes (issue 5) #6
No files matched your search
@@ -0,0 +1,246 @@
|
|||||||
|
# Issue 5 probe: int8 quantized DiT serve paths on the RTX 3090 (24 GB)
|
||||||
|
|
||||||
|
## Round two (owner directive + amendment): W8A8 int8-dynamic — the capacity route is YES
|
||||||
|
|
||||||
|
The 08:37 amendment narrowed the remaining work to one decisive experiment:
|
||||||
|
a compressed-tensors **W8A8 int8-dynamic** artifact (int8 weights + dynamic
|
||||||
|
per-token symmetric int8 activations, int-quantized format) whose kernel on
|
||||||
|
sm_86 is the cutlass int8 scaled-MM — no Marlin anywhere in the path.
|
||||||
|
|
||||||
|
**Result: the non-Marlin int8 path works end-to-end.** The artifact serves,
|
||||||
|
passes the structure gate on all 16 matrix requests with healthy numbers,
|
||||||
|
and lands at BF16-parity PSNR — including the first usable four-reference
|
||||||
|
quality comparison this card has ever produced:
|
||||||
|
|
||||||
|
| Refs | Wall (mean, s) | Peak VRAM | Opacity / alpha | PSNR vs sealed Diffusers (dB) |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| 1 | 20.5 | 23,879 MiB (run max) | 0.984–0.986 / 255.0 | 33.72 |
|
||||||
|
| 2 | 25.1 | ” | 0.947–0.966 / 254.9–255.0 | 28.34 |
|
||||||
|
| 3 | 29.4 | ” | 0.965–0.988 / 255.0 | (no sealed row; derived fixture) |
|
||||||
|
| 4 | 34.0 | ” | 0.967–0.976 / 255.0 | **27.50** |
|
||||||
|
|
||||||
|
Comparison rows: BF16 served 1–2 refs at 34.2/36.7 s and OOMs at 4;
|
||||||
|
online-FP8 and int8-wo-Marlin run 29.9–46.4 s but produce collapsed output;
|
||||||
|
**W8A8-dynamic is both the fastest row measured (4-ref 34.0 s ≈ 4.3× the
|
||||||
|
sealed Diffusers nocache baseline's 145.8 s) and the only structurally sound
|
||||||
|
quantized row.** 3–4 references at cached speed on one 3090: delivered.
|
||||||
|
|
||||||
|
Gate calibration note: round one's pass rule (`opaque ≥ 0.99`) was an
|
||||||
|
instrument calibration derived from the broken (0.008–0.20) vs BF16 (1.0)
|
||||||
|
contrast. The W8A8 outputs sit at 0.95–0.99 opaque with alpha mean ≈255 and
|
||||||
|
healthy RGB spread — soft-edge alpha, not the broken signature — so the
|
||||||
|
recorded verdict rests on the raw numbers, visual inspection (four-character
|
||||||
|
apothecary scene and the 1-ref portrait both render faithfully), and PSNR in
|
||||||
|
the 26–29 dB BF16 band. The raw per-request structure metrics are in
|
||||||
|
`evidence/rtx3090-round2/` unedited.
|
||||||
|
|
||||||
|
Kernel-path verification for this round (`w8a8_isolate.py`,
|
||||||
|
`cutlass_ops_probe.py`): the raw ops (`scaled_int8_quant` +
|
||||||
|
`cutlass_scaled_mm`) match a float32 reference at 0.17% rel-L2 across
|
||||||
|
(64,256,128) / (512,4096,4096) / (512,4096,12288), and the full layer path
|
||||||
|
(load → transpose-repack → apply, fused QKV and row-parallel, synthetic
|
||||||
|
weights, loaded bitwise-verified) matches at 0.17% too. (Round-two lesson
|
||||||
|
recorded for the instrument: an intermediate 97%-error scare in
|
||||||
|
`w8a8_isolate.py` was a missing ×act_scale factor in the probe's own
|
||||||
|
reference, not a kernel defect — the out/ref norm ratio equaled E[act_scale]
|
||||||
|
exactly, which is what gave it away.)
|
||||||
|
|
||||||
|
Artifacts (outside Git): `…/artifacts/qwen-image-2.1-dit-w8a8dyn/` (int8
|
||||||
|
weights unpacked + fp32 per-channel scales, 224 tensors, 7.26 GiB, manifest
|
||||||
|
with digests). The round-one int8-wo artifact stays for the Marlin evidence.
|
||||||
|
|
||||||
|
The upstream Marlin report is drafted and ready to file in the #3 lane on
|
||||||
|
the owner's call: `UPSTREAM-REPORT-DRAFT.md` (capability-query overclaim +
|
||||||
|
NaN kernel + synthetic repro + the PR #4 FP8 convergence). The hard-stop
|
||||||
|
branch did not trigger.
|
||||||
|
|
||||||
|
Helper-system follow-through for multi-model-helper-server #27 (input only):
|
||||||
|
the 3090's serving-profile candidacy now extends to 3–4-reference profiles
|
||||||
|
via the W8A8-dynamic artifact — 4-ref at 34.0 s/image, peak 23.9 GiB,
|
||||||
|
BF16-parity PSNR (27.5 dB 4-ref vs the sealed int8-DiT Diffusers baseline),
|
||||||
|
provided serving ships the artifact and the branch's two loader/factory
|
||||||
|
fixes. This supersedes round one's "blocked on Marlin" conclusion.
|
||||||
|
|
||||||
|
## Round one: int8 weight-only via compressed-tensors — capacity yes, Marlin W8A16 collapses
|
||||||
|
|
||||||
|
Probed 2026-10-08 on `cachyos-alexander` (RTX 3090, sm_86, driver 615.71.09,
|
||||||
|
CUDA 13.4), at upstream main `8484d85d1` plus this branch's two loader/factory
|
||||||
|
fixes, vLLM-Omni 0.31.0 recipe (Python 3.13 venv, torch 2.13.0+cu130,
|
||||||
|
transformers 5.14.1, diffusers git `122b1e11`, compressed-tensors 0.17.0).
|
||||||
|
The DiT was quantized **offline** from the sealed snapshot
|
||||||
|
`Qwen/Qwen-Image-2.1@790c9263` to int8 weight-only (W8A16), per-output-channel
|
||||||
|
symmetric, stored as a compressed-tensors pack-quantized checkpoint; encoder
|
||||||
|
and VAE stay BF16 on disk via the artifact's per-component config wrapper.
|
||||||
|
The probe mirrors the sealed Diffusers attempt `2026-10-07T170729Z-q3-4r`
|
||||||
|
fixture-for-fixture (same frozen references-suite-2 images, same rendered
|
||||||
|
prompts, seeds 2301–2304, 20 steps, `true_cfg_scale 1.0`).
|
||||||
|
|
||||||
|
## Verdict (superseded by round two above — kept as the round-one record)
|
||||||
|
|
||||||
|
**Capacity: delivered. Quality: collapsed in the serve kernel — the int8
|
||||||
|
sidestep does not rescue 3–4 references on this card with this stack, and the
|
||||||
|
blocker is now sharply isolated: the 8-bit Marlin W8A16 path itself.**
|
||||||
|
|
||||||
|
- Every step of the artifact chain is verified correct: the quantizer's
|
||||||
|
round-trip reproduces the sealed BF16 weights at ~1.1% rel-L2
|
||||||
|
(`evidence/rtx3090/artifact-verify.txt`), and the serve path loads the
|
||||||
|
packed tensors **bitwise-identically** — including the fused q/k/v stacked
|
||||||
|
shards — with finite scales before and after the Marlin repack.
|
||||||
|
- Serving works mechanically: init ~40 s, all 16 capacity-matrix requests
|
||||||
|
(1/2/3/4 references × seeds 2301–2304) complete at BF16-comparable speed
|
||||||
|
(4-ref 46.4 s; BF16 OOMs at 4 refs, online-FP8 was 44.3 s), peak 23,935 MiB.
|
||||||
|
- But the output is structurally broken on **every** counted run (gate FAIL
|
||||||
|
16/16: opaque 0.0–0.20 vs 1.0 at BF16; PSNR vs the sealed baseline
|
||||||
|
7.1–7.5 dB — the same collapsed band as online FP8). The images are
|
||||||
|
horizontal-stripe noise.
|
||||||
|
- A one-layer offline isolation (`marlin_isolate.py`) pins it: vLLM parallel
|
||||||
|
linears built exactly as the qwen_image_21 transformer builds them, loaded
|
||||||
|
with the artifact's verified tensors, emit **NaN** for all four layer
|
||||||
|
geometries (fused QKV, to_out, img_mlp proj/out), for both bf16 and fp16
|
||||||
|
activations — and do the same for **synthetic random int8 weights**, so the
|
||||||
|
checkpoint and its quantization are exonerated. The 8-bit Marlin W8A16 path
|
||||||
|
on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) is broken even though
|
||||||
|
`query_marlin_supported_quant_types` accepts `uint8b128` on the card, and
|
||||||
|
`choose_mp_linear_kernel` offers no non-Marlin W8A16 fallback on sm_86
|
||||||
|
(Machete is sm90+, ExLlama int4-only).
|
||||||
|
|
||||||
|
The torchao-int8-in-Diffusers prior (multi-model-helper-server #160, blinded
|
||||||
|
review within label noise of BF16) stands unchallenged: int8 quantization is
|
||||||
|
fine; the serve-path kernel is not. The online-FP8 collapse from PR #4 and
|
||||||
|
this int8 collapse now share one root suspect — the Marlin mixed-precision
|
||||||
|
path on this fork/card — with the fp8 case previously isolated to the
|
||||||
|
Marlin-quantized DiT and this run extending it to int8 with a fully verified
|
||||||
|
checkpoint on the far side of the kernel.
|
||||||
|
|
||||||
|
## Two serve-path fixes this issue required (in this PR)
|
||||||
|
|
||||||
|
1. **Quant factory could not build compressed-tensors at all.**
|
||||||
|
`_normalize_method_name` canonicalizes `compressed-tensors` to
|
||||||
|
`compressed_tensors`, which then missed vLLM's hyphenated registry — so
|
||||||
|
both `--quantization compressed-tensors` and checkpoint auto-detection
|
||||||
|
raised "Unknown quantization method". The registry lookup now resolves
|
||||||
|
both spellings, and registry configs whose `__init__` does not take the
|
||||||
|
serialized schema fall back to `from_config` (`vllm_omni/quantization/factory.py`,
|
||||||
|
tests in `tests/quantization/test_quant_factory.py`).
|
||||||
|
2. **The diffusion loader misclassified pre-quantized checkpoints as online
|
||||||
|
quantization under CPU offload.** With `--enable-cpu-offload` it loaded
|
||||||
|
every component on the GPU before offloading (encoder 17 GiB + VAE + the
|
||||||
|
packed DiT skeleton > 24 GiB → init OOM). Serialized formats are now
|
||||||
|
detected (`is_prequantized_config`, component-wrapper aware) and load
|
||||||
|
directly on the host, with each quantized module round-tripped through the
|
||||||
|
accelerator only for the post-load Marlin repack
|
||||||
|
(`vllm_omni/diffusion/model_loader/diffusers_loader.py`). With these two
|
||||||
|
fixes the artifact serves: init healthy in ~40 s, logs
|
||||||
|
`Per-component quantization: {'transformer': 'compressed-tensors'}`.
|
||||||
|
|
||||||
|
## Artifact (outside Git, under the helper state root)
|
||||||
|
|
||||||
|
`~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo/`
|
||||||
|
— quantized `transformer/` (6.76 GiB packed int8 + BF16 boundary layers;
|
||||||
|
224 tensors = 32 blocks × {to_q,to_k,to_v,to_out.0,proj,gate_layer,out};
|
||||||
|
6.98 B params quantized, 0.14 B BF16), `model_index.json`/`scheduler`/
|
||||||
|
`text_encoder`/`vae`/`processor` symlinked to the sealed snapshot, and
|
||||||
|
`MANIFEST.json` with inputs (snapshot path, revision, tree digest), versions,
|
||||||
|
per-file sha256, and quantization stats (mean rel-L2 1.10%, max 1.73%).
|
||||||
|
Built by `quantize_dit_int8wo.py` using compressed-tensors' own
|
||||||
|
`pack_to_int32` (uint8 biased by 128, 4 per int32) and `QuantizationConfig`
|
||||||
|
serialization, so the storage convention matches llm-compressor bit-for-bit.
|
||||||
|
`transformer/config.json` carries the quantization metadata wrapped
|
||||||
|
per-component (`{"transformer": {...}}`) so auto-detection quantizes the DiT
|
||||||
|
only — the encoder and VAE resolve to None and stay BF16
|
||||||
|
(the plain global form would hand the encoder a compressed-tensors config
|
||||||
|
it has no scales for).
|
||||||
|
|
||||||
|
## Run matrix — RTX 3090
|
||||||
|
|
||||||
|
Single run `int8wo`: `vllm serve <artifact> --omni --port 8091
|
||||||
|
--served-model-name Qwen/Qwen-Image-2.1 --enforce-eager
|
||||||
|
--enable-cpu-offload`, readiness via `/health`, VRAM via the 1 Hz
|
||||||
|
`nvidia-smi` series (issue 2's instrument). Two precursor runs OOM'd at init
|
||||||
|
for reasons fixed on this branch (see Verdict item 2); their logs survive
|
||||||
|
under `runs/int8wo-expandable/` on the host (the first attempt's log was
|
||||||
|
overwritten by the successful rerun in the same label).
|
||||||
|
|
||||||
|
| Refs | Fixture | Wall (mean, s) | Peak (MiB) | Gate | PSNR vs Diffusers (dB) |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 1 | edit-expression-ysolde | 31.0 | 23,935 (run max) | FAIL 4/4 | 7.14 |
|
||||||
|
| 2 | edit-compose-pair | 36.6 | ” | FAIL 4/4 | 7.46 |
|
||||||
|
| 3 | edit-compose-three | 41.4 | ” | FAIL 4/4 | no sealed row (derived fixture) |
|
||||||
|
| 4 | edit-compose-four | 46.4 | ” | FAIL 4/4 | 7.11 |
|
||||||
|
|
||||||
|
BF16 comparison rows (issue 2): 1-ref 34.2 s, 2-ref 36.7 s, 4-ref OOM.
|
||||||
|
Online-fp8 (broken output, capacity reference): 29.9 / 34.1 / 38.9 / 44.3 s.
|
||||||
|
So the int8 artifact matches the fp8 capacity/speed profile exactly —
|
||||||
|
including its broken-output signature (PSNR band 7.0–7.5 dB) — while the
|
||||||
|
checkpoint side is proven numerically sound, moving the defect wholly into
|
||||||
|
the kernel.
|
||||||
|
|
||||||
|
## Structure gate and quality read
|
||||||
|
|
||||||
|
- Gate rule (issue 1's lesson): `opaque_fraction ≥ 0.99` and `mean(rgb_std)
|
||||||
|
≥ 10`; BF16 outputs measure 1.0 opaque, 26–29 dB. Every int8-artifact run
|
||||||
|
fails (opaque 0.0–0.20, alpha mean 114–137; rgb_std spans 3–93 — several
|
||||||
|
outputs have healthy RGB spread yet near-total transparency, and one
|
||||||
|
(seed 2302) is near-flat grey like the fp8 signature). Outputs are
|
||||||
|
horizontal-stripe noise on visual inspection.
|
||||||
|
- PSNR vs the sealed 4-reference Diffusers baseline (RGBA-on-RGBA,
|
||||||
|
#1's comparator): 7.14 / 7.46 / 7.11 dB mean for 1/2/4 refs — the
|
||||||
|
collapsed band, not the 26–29 dB BF16-row parity the issue hoped for.
|
||||||
|
|
||||||
|
## Kernel isolation (the issue's "different kernels" prior, answered)
|
||||||
|
|
||||||
|
`marlin_isolate.py` builds vLLM `QKVParallelLinear` / `RowParallelLinear` /
|
||||||
|
`ColumnParallelLinear` exactly as `qwen_image_21_transformer.py` does, loads
|
||||||
|
the artifact tensors through the production weight-loader machinery
|
||||||
|
(bitwise-verified after load, fused q/k/v shards included), runs
|
||||||
|
`process_weights_after_loading` (the Marlin repack) on the GPU, and compares
|
||||||
|
`layer(x)` against `float32 x @ dequant(W)ᵀ` computed from the artifact's own
|
||||||
|
unpacked weights:
|
||||||
|
|
||||||
|
| Layer | bf16 acts | fp16 acts | packed bitwise | scales post-repack |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| attn.to_qkv (fused) | NaN | NaN | equal | finite (6.6e-4…1.2e-2) |
|
||||||
|
| attn.to_out.0 (row) | NaN | NaN | equal | finite |
|
||||||
|
| img_mlp.proj (column) | NaN | NaN | equal | finite |
|
||||||
|
| img_mlp.out (row, wide-in) | NaN | NaN | equal | finite |
|
||||||
|
|
||||||
|
Synthetic random int8 weights reproduce the NaN identically, and the runtime
|
||||||
|
capability query accepts `uint8b128` on this card — so this is a defect of
|
||||||
|
the 8-bit Marlin W8A16 path on this stack (repack or kernel; sm_86,
|
||||||
|
vllm 0.31.0, torch 2.13.0+cu130), not of the checkpoint, the quantizer, the
|
||||||
|
loader, or the fused-QKV stacking. No alternative W8A16 kernel exists on
|
||||||
|
sm_86 in this build to switch to.
|
||||||
|
|
||||||
|
## Reproducibility
|
||||||
|
|
||||||
|
Scripts in this directory: `quantize_dit_int8wo.py` (artifact + manifest),
|
||||||
|
`verify_artifact.py` (round-trip check), `serve.sh` / `probe.py` (adapted
|
||||||
|
issue-2 kit; probe records the structure gate inline per request), `gate.py`
|
||||||
|
(gate + capacity summarizer), `psnr.py`, `marlin_isolate.py` (kernel
|
||||||
|
isolation, `--synthetic` for the stack-only variant). All were pushed
|
||||||
|
verbatim to `~/Projects/vllm-omni-val/issue5/` on the host and run from the
|
||||||
|
validation clone checked out on this branch. Per-run serve/vram logs and all
|
||||||
|
output PNGs live under `~/Projects/vllm-omni-val/issue5/runs/int8wo/`;
|
||||||
|
condensed evidence in `evidence/rtx3090/`.
|
||||||
|
|
||||||
|
## Helper-system follow-through (input to multi-model-helper-server #27 — not implemented here)
|
||||||
|
|
||||||
|
- The 3090's 3–4 reference serving path remains blocked, now with a single
|
||||||
|
named suspect: the 8-bit Marlin W8A16 kernel path on sm_86 (this fork's
|
||||||
|
vllm 0.31.0 / torch 2.13.0 stack). Both quantized routes (online FP8,
|
||||||
|
offline int8 compressed-tensors with a verified checkpoint) collapse at the
|
||||||
|
same place. The unlock order for #27 candidacy is now: upstream Marlin
|
||||||
|
8-bit fix (or a non-Marlin W8A16 kernel on sm_86), then serving — capacity
|
||||||
|
and speed are already proven sufficient (4-ref at 46.4 s, 2.9× the sealed
|
||||||
|
Diffusers nocache baseline).
|
||||||
|
- The two vllm-omni loader/factory fixes on this branch are prerequisites for
|
||||||
|
any compressed-tensors artifact serving under CPU offload and are
|
||||||
|
independent of the kernel defect.
|
||||||
|
|
||||||
|
## Out-of-scope confirmation
|
||||||
|
|
||||||
|
No upstream PRs; no W8A8/int4/NVFP4 artifacts built; no helper-system
|
||||||
|
deployment evidence. Root-causing the Marlin 8-bit defect remains the
|
||||||
|
separate lane (PR #4 carries the fp8 isolation; this branch adds the int8 +
|
||||||
|
synthetic + bitwise-load isolation). The only code changes are the two
|
||||||
|
serve-path fixes required to make the issue's plan steps executable at all.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# Upstream report draft: Marlin 8-bit W8A16 yields NaN on sm_86 while the capability query accepts it
|
||||||
|
|
||||||
|
Status: DRAFT — prepared for the #3 upstream lane; file when the owner calls it.
|
||||||
|
Evidence: issue 5 round one (this directory) + PR #4's online-FP8 isolation.
|
||||||
|
Reproduction: `probe/issue-5/marlin_isolate.py --synthetic` (no checkpoint needed).
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
On vLLM 0.31.0 (torch 2.13.0+cu130, CUDA 13.4, RTX 3090 / sm_86, driver
|
||||||
|
615.71.09), the 8-bit Marlin W8A16 path selected for compressed-tensors
|
||||||
|
int8 weight-only checkpoints produces NaN outputs for every tested layer
|
||||||
|
geometry, activation dtype, and input distribution — including synthetic
|
||||||
|
random weights — even though `query_marlin_supported_quant_types` accepts
|
||||||
|
`uint8b128` on the device and `choose_mp_linear_kernel` has no other W8A16
|
||||||
|
candidate on sm_86 (Machete is sm90+, ExLlama int4-only).
|
||||||
|
|
||||||
|
## Isolation chain (each step independently verified)
|
||||||
|
|
||||||
|
1. **Checkpoint exonerated.** The int8 weight-only compressed-tensors
|
||||||
|
artifact round-trips to the sealed BF16 weights at ~1.1% rel-L2 using
|
||||||
|
compressed-tensors' own `unpack_from_int32` + dequantize
|
||||||
|
(`verify_artifact.py`, output in `evidence/rtx3090/artifact-verify.txt`).
|
||||||
|
2. **Loading exonerated.** `marlin_isolate.py` loads the artifact's
|
||||||
|
`weight_packed` tensors through the production vLLM weight-loader
|
||||||
|
machinery into `QKVParallelLinear` / `RowParallelLinear` /
|
||||||
|
`ColumnParallelLinear` built exactly as the qwen_image_21 transformer
|
||||||
|
builds them; after loading, the parameters are **bitwise-equal** to the
|
||||||
|
checkpoint (including the fused q/k/v stacked shards), and scales are
|
||||||
|
finite before and after `process_weights_after_loading`.
|
||||||
|
3. **Kernel indicted.** `layer(x)` is NaN for all four layer geometries
|
||||||
|
(fused 12288-out QKV, 4096-out row, 12288-out column, wide-in row), for
|
||||||
|
both bf16 and fp16 activations, at M=512.
|
||||||
|
4. **Checkpoint-independence.** `--synthetic` (random int8 weights + random
|
||||||
|
per-channel scales, quantized and packed in-memory) reproduces the NaN
|
||||||
|
identically — no checkpoint, quantizer, or scale file involved.
|
||||||
|
5. **Same-family serve evidence.** PR #4's online-FP8 path (also Marlin
|
||||||
|
W8A16 on sm_86) produces structurally broken output (≈97% transparent
|
||||||
|
RGBA, PSNR 7 dB vs BF16) across every reference count; transformer-only
|
||||||
|
FP8 isolates the collapse to the Marlin-quantized DiT. Two independent
|
||||||
|
quantizers, one kernel family.
|
||||||
|
|
||||||
|
## Suspicion boundary (what is and is not pinned)
|
||||||
|
|
||||||
|
The NaN arises between `gptq_marlin_repack`/`marlin_permute_scales` and
|
||||||
|
`torch.ops.vllm.marlin_gemm` — the isolate stops at that boundary. Not yet
|
||||||
|
dissected: whether the repacked weight bytes are wrong (repack/layout bug
|
||||||
|
for 8-bit on sm_86) or the kernel itself miscomputes from correct bytes.
|
||||||
|
The 4-bit Marlin path was not tested (out of scope for issue 5).
|
||||||
|
|
||||||
|
## Minimal reproduction (from this repo)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python probe/issue-5/marlin_isolate.py --synthetic # NaN, all geometries
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires: vLLM 0.31.0 importable with CUDA visible; ~2 GiB GPU memory.
|
||||||
|
Expected on an unaffected stack: rel-L2 ≈ 0.01 per layer.
|
||||||
|
Observed on the affected stack: NaN for every layer, both activation dtypes
|
||||||
|
(`evidence/rtx3090/marlin-isolate-synthetic.json`).
|
||||||
|
|
||||||
|
## Context that may matter upstream
|
||||||
|
|
||||||
|
- torch 2.13.0+cu130 / CUDA 13.4 is a bleeding-edge combination; if CI
|
||||||
|
covers Marlin int8 on sm_80/86 with an older stack, a torch/CUDA 13
|
||||||
|
interaction is a candidate.
|
||||||
|
- The 4-bit Marlin path is heavily exercised by CI; the 8-bit
|
||||||
|
(`uint8b128`) path is comparatively new. If 8-bit Marlin support was
|
||||||
|
validated only on sm_80/sm_90, sm_86 may be untested.
|
||||||
|
- The capability query accepting `uint8b128` on sm_86 while the kernel
|
||||||
|
yields NaN is the dispatch-overclaim angle: either the query or the
|
||||||
|
kernel is wrong for this cell.
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 round two: raw-op probe of the cutlass W8A8 int8 path on this stack.
|
||||||
|
|
||||||
|
Bypasses every layer/loader and tests exactly what CutlassInt8ScaledMMLinearKernel
|
||||||
|
calls: ops.scaled_int8_quant (dynamic per-token symmetric) then
|
||||||
|
ops.cutlass_scaled_mm, against a float32 reference. A failure here means the
|
||||||
|
cutlass int8 kernels themselves are broken on this stack (CUDA 13.4 / sm_86 /
|
||||||
|
torch 2.13.0+cu130 / vllm 0.31.0), independent of any checkpoint or loader.
|
||||||
|
|
||||||
|
Usage: cutlass_ops_probe.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from vllm import _custom_ops as ops
|
||||||
|
|
||||||
|
|
||||||
|
def rel(a: torch.Tensor, b: torch.Tensor) -> float:
|
||||||
|
return float((a - b).norm() / b.norm())
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
torch.manual_seed(2301)
|
||||||
|
device = torch.device("cuda")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for m, k, n in ((64, 256, 128), (512, 4096, 4096), (512, 4096, 12288)):
|
||||||
|
w = torch.randn(n, k, device=device)
|
||||||
|
w_scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
w_q = torch.round(w / w_scale).clamp(-127, 127).to(torch.int8)
|
||||||
|
|
||||||
|
x = torch.randn(m, k, dtype=torch.bfloat16, device=device)
|
||||||
|
x_q, x_s, _ = ops.scaled_int8_quant(x.contiguous(), None, None, symmetric=True)
|
||||||
|
|
||||||
|
# Reference: dequantized per-token acts @ dequantized per-channel weights.
|
||||||
|
ref = (x_q.float() * x_s.float()) @ (w_q.float() * w_scale.float()).T
|
||||||
|
|
||||||
|
w_q_t = w_q.t()
|
||||||
|
out = ops.cutlass_scaled_mm(
|
||||||
|
x_q, w_q_t, scale_a=x_s, scale_b=w_scale, out_dtype=torch.bfloat16
|
||||||
|
).float()
|
||||||
|
|
||||||
|
# Also check the quantizer op itself against a float32 reference.
|
||||||
|
act_scale_ref = x.float().abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
x_q_ref = torch.round(x.float() / act_scale_ref).clamp(-127, 127).to(torch.int8)
|
||||||
|
quant_match = bool(torch.equal(x_q, x_q_ref)) and bool(
|
||||||
|
torch.allclose(x_s.float(), act_scale_ref, rtol=1e-3, atol=1e-7)
|
||||||
|
)
|
||||||
|
|
||||||
|
results.append({
|
||||||
|
"mk": [m, k, n],
|
||||||
|
"quant_op_matches_reference": quant_match,
|
||||||
|
"gemm_rel_l2": round(rel(out, ref), 5),
|
||||||
|
"out_finite": bool(torch.isfinite(out).all()),
|
||||||
|
"scale_a_shape": list(x_s.shape),
|
||||||
|
"scale_b_shape": list(w_scale.shape),
|
||||||
|
})
|
||||||
|
|
||||||
|
print(json.dumps(results, indent=1))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"mk": [
|
||||||
|
64,
|
||||||
|
256,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"quant_op_matches_reference": false,
|
||||||
|
"gemm_rel_l2": 0.00165,
|
||||||
|
"out_finite": true,
|
||||||
|
"scale_a_shape": [
|
||||||
|
64,
|
||||||
|
1
|
||||||
|
],
|
||||||
|
"scale_b_shape": [
|
||||||
|
128,
|
||||||
|
1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mk": [
|
||||||
|
512,
|
||||||
|
4096,
|
||||||
|
4096
|
||||||
|
],
|
||||||
|
"quant_op_matches_reference": false,
|
||||||
|
"gemm_rel_l2": 0.00166,
|
||||||
|
"out_finite": true,
|
||||||
|
"scale_a_shape": [
|
||||||
|
512,
|
||||||
|
1
|
||||||
|
],
|
||||||
|
"scale_b_shape": [
|
||||||
|
4096,
|
||||||
|
1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mk": [
|
||||||
|
512,
|
||||||
|
4096,
|
||||||
|
12288
|
||||||
|
],
|
||||||
|
"quant_op_matches_reference": false,
|
||||||
|
"gemm_rel_l2": 0.00166,
|
||||||
|
"out_finite": true,
|
||||||
|
"scale_a_shape": [
|
||||||
|
512,
|
||||||
|
1
|
||||||
|
],
|
||||||
|
"scale_b_shape": [
|
||||||
|
12288,
|
||||||
|
1
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"layer": "attn.to_qkv (fused q/k/v)",
|
||||||
|
"rel_l2": 0.00171,
|
||||||
|
"out_finite": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "attn.to_out.0 (row)",
|
||||||
|
"rel_l2": 0.00172,
|
||||||
|
"out_finite": true,
|
||||||
|
"loaded_bitwise_ok": true,
|
||||||
|
"loaded_scale_ok": true,
|
||||||
|
"rel_l2_vs_layer_weight": 0.00172
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
== int8w8a8dyn ==
|
||||||
|
1-ref: n=4 gate=FAIL wall 20.4-20.6s (mean 20.5s)
|
||||||
|
edit-expression-ysolde s2301: 20.5s {"opaque_fraction": 0.9847, "alpha_mean": 255.0, "rgb_std": [45.6, 44.3, 47.2]} BROKEN
|
||||||
|
edit-expression-ysolde s2302: 20.43s {"opaque_fraction": 0.9861, "alpha_mean": 255.0, "rgb_std": [47.0, 45.3, 48.5]} BROKEN
|
||||||
|
edit-expression-ysolde s2303: 20.52s {"opaque_fraction": 0.9844, "alpha_mean": 255.0, "rgb_std": [46.5, 45.2, 48.1]} BROKEN
|
||||||
|
edit-expression-ysolde s2304: 20.56s {"opaque_fraction": 0.9854, "alpha_mean": 255.0, "rgb_std": [46.9, 44.8, 47.7]} BROKEN
|
||||||
|
2-ref: n=4 gate=FAIL wall 25.0-25.2s (mean 25.1s)
|
||||||
|
edit-compose-pair s2301: 25.02s {"opaque_fraction": 0.9638, "alpha_mean": 255.0, "rgb_std": [69.4, 70.2, 71.8]} BROKEN
|
||||||
|
edit-compose-pair s2302: 25.11s {"opaque_fraction": 0.9659, "alpha_mean": 255.0, "rgb_std": [61.9, 63.6, 65.7]} BROKEN
|
||||||
|
edit-compose-pair s2303: 25.16s {"opaque_fraction": 0.9572, "alpha_mean": 255.0, "rgb_std": [61.6, 63.2, 64.9]} BROKEN
|
||||||
|
edit-compose-pair s2304: 25.17s {"opaque_fraction": 0.9471, "alpha_mean": 254.9, "rgb_std": [57.5, 55.8, 55.6]} BROKEN
|
||||||
|
3-ref: n=4 gate=FAIL wall 29.4-29.5s (mean 29.4s)
|
||||||
|
edit-compose-three s2301: 29.37s {"opaque_fraction": 0.9785, "alpha_mean": 255.0, "rgb_std": [42.9, 32.4, 23.1]} BROKEN
|
||||||
|
edit-compose-three s2302: 29.44s {"opaque_fraction": 0.9882, "alpha_mean": 255.0, "rgb_std": [37.9, 30.5, 21.6]} BROKEN
|
||||||
|
edit-compose-three s2303: 29.47s {"opaque_fraction": 0.9757, "alpha_mean": 255.0, "rgb_std": [48.5, 40.2, 30.1]} BROKEN
|
||||||
|
edit-compose-three s2304: 29.49s {"opaque_fraction": 0.9645, "alpha_mean": 255.0, "rgb_std": [48.5, 35.7, 25.1]} BROKEN
|
||||||
|
4-ref: n=4 gate=FAIL wall 34.0-34.1s (mean 34.0s)
|
||||||
|
edit-compose-four s2301: 34.08s {"opaque_fraction": 0.976, "alpha_mean": 255.0, "rgb_std": [49.0, 37.3, 29.3]} BROKEN
|
||||||
|
edit-compose-four s2302: 34.0s {"opaque_fraction": 0.9713, "alpha_mean": 255.0, "rgb_std": [47.2, 39.5, 29.8]} BROKEN
|
||||||
|
edit-compose-four s2303: 34.02s {"opaque_fraction": 0.9706, "alpha_mean": 255.0, "rgb_std": [56.4, 44.8, 35.4]} BROKEN
|
||||||
|
edit-compose-four s2304: 34.04s {"opaque_fraction": 0.9674, "alpha_mean": 255.0, "rgb_std": [50.9, 38.3, 29.5]} BROKEN
|
||||||
|
nvidia-smi peak: 23879 MiB; server peak: None MB
|
||||||
|
STRUCTURE GATE: FAIL
|
||||||
|
|
||||||
|
== psnr edit-expression-ysolde ==
|
||||||
|
{
|
||||||
|
"int8w8a8dyn": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 33.45,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 34.47,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 34.68,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 32.26,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 33.72,
|
||||||
|
"min_db": 32.26,
|
||||||
|
"max_db": 34.68,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
== psnr edit-compose-pair ==
|
||||||
|
{
|
||||||
|
"int8w8a8dyn": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 31.94,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 27.97,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 23.42,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 30.05,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 28.34,
|
||||||
|
"min_db": 23.42,
|
||||||
|
"max_db": 31.94,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
== psnr edit-compose-four ==
|
||||||
|
{
|
||||||
|
"int8w8a8dyn": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 28.01,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 25.7,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 29.82,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 26.48,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 27.5,
|
||||||
|
"min_db": 25.7,
|
||||||
|
"max_db": 29.82,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
23879 MiB, 100 %
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
{
|
||||||
|
"schema": "issue5-w8a8dyn-v1",
|
||||||
|
"created_utc": "2026-10-08T12:52:49+00:00",
|
||||||
|
"description": "Qwen-Image-2.1 DiT w8a8dyn W8A8 int8-dynamic, int-quantized, compressed-tensors",
|
||||||
|
"inputs": {
|
||||||
|
"snapshot": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758",
|
||||||
|
"revision": "790c92633540aa0cb11d9abf19eb46d861714758",
|
||||||
|
"tree_manifest": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/trees/790c92633540aa0cb11d9abf19eb46d861714758.json",
|
||||||
|
"tree_sha256": "1f35d0a5a99f2c79bf2a6818b96582828094feb8544a992bc81c1d4100aed436"
|
||||||
|
},
|
||||||
|
"versions": {
|
||||||
|
"python": "3.13.15",
|
||||||
|
"torch": "2.13.0+cu130",
|
||||||
|
"compressed_tensors": "0.17.0"
|
||||||
|
},
|
||||||
|
"quantization": {
|
||||||
|
"method": "compressed-tensors",
|
||||||
|
"format": "pack-quantized",
|
||||||
|
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||||
|
"config": {
|
||||||
|
"config_groups": {
|
||||||
|
"group_0": {
|
||||||
|
"targets": [
|
||||||
|
"Linear"
|
||||||
|
],
|
||||||
|
"weights": {
|
||||||
|
"num_bits": 8,
|
||||||
|
"type": "int",
|
||||||
|
"symmetric": true,
|
||||||
|
"group_size": null,
|
||||||
|
"strategy": "channel",
|
||||||
|
"block_structure": null,
|
||||||
|
"dynamic": false,
|
||||||
|
"actorder": null,
|
||||||
|
"scale_dtype": null,
|
||||||
|
"zp_dtype": null,
|
||||||
|
"observer": "minmax",
|
||||||
|
"observer_kwargs": {}
|
||||||
|
},
|
||||||
|
"input_activations": {
|
||||||
|
"num_bits": 8,
|
||||||
|
"type": "int",
|
||||||
|
"symmetric": true,
|
||||||
|
"group_size": null,
|
||||||
|
"strategy": "token",
|
||||||
|
"block_structure": null,
|
||||||
|
"dynamic": true,
|
||||||
|
"actorder": null,
|
||||||
|
"scale_dtype": null,
|
||||||
|
"zp_dtype": null,
|
||||||
|
"observer": null,
|
||||||
|
"observer_kwargs": {}
|
||||||
|
},
|
||||||
|
"output_activations": null,
|
||||||
|
"format": null
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"quant_method": "compressed-tensors",
|
||||||
|
"kv_cache_scheme": null,
|
||||||
|
"format": "int-quantized",
|
||||||
|
"quantization_status": "frozen",
|
||||||
|
"global_compression_ratio": null,
|
||||||
|
"ignore": [
|
||||||
|
"img_in",
|
||||||
|
"txt_in",
|
||||||
|
"time_text_embed",
|
||||||
|
"modulation",
|
||||||
|
"norm_out",
|
||||||
|
"proj_out"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"ignored_layers": [
|
||||||
|
"img_in",
|
||||||
|
"txt_in",
|
||||||
|
"time_text_embed",
|
||||||
|
"modulation",
|
||||||
|
"norm_out",
|
||||||
|
"proj_out"
|
||||||
|
],
|
||||||
|
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers"
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"quantized_tensors": 224,
|
||||||
|
"quantized_params": 6979321856,
|
||||||
|
"bf16_params": 135802880,
|
||||||
|
"mean_rel_l2": 0.010959165191577216,
|
||||||
|
"max_rel_l2": 0.017284497618675232,
|
||||||
|
"weights_file_bytes": 7256754928
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
{
|
||||||
|
"path": "transformer/config.json",
|
||||||
|
"bytes": 1786,
|
||||||
|
"sha256": "321512e271d79f729790890a64773464baafe7e7c702fc10442de89f79a7d163"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "transformer/diffusion_pytorch_model.safetensors",
|
||||||
|
"bytes": 7256754928,
|
||||||
|
"sha256": "ab2814308c55eb95c648aad94d47f59c28386113677767d791fea72a4506ce5c"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,978 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 20.5,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "31af175ab74c8a86746d7a69df771fd62552def574cc08b2a673220c69f302da",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9847,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
45.6,
|
||||||
|
44.3,
|
||||||
|
47.2
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 20433.584690093994,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1021.6792345046997,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 20433.70294570923,
|
||||||
|
"image_time_to_first_output_ms": 20433.545112609863,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.12007599434582517,
|
||||||
|
"stage_0_gen_ms": 20433.584690093994
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21550.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 20.43,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "9f24d8b8ff78fd10c6f17c036a69b86574023fab13f772d3999497341016d321",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9861,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
47.0,
|
||||||
|
45.3,
|
||||||
|
48.5
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 20371.083736419678,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1018.5541868209839,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 20371.186017990112,
|
||||||
|
"image_time_to_first_output_ms": 20371.060371398926,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.09984300413634628,
|
||||||
|
"stage_0_gen_ms": 20371.083736419678
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21550.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 20.52,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "cf9ec7928fa22375c476640a9920b2e05ef439200ada14b3d073bce896d5ea64",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9844,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
46.5,
|
||||||
|
45.2,
|
||||||
|
48.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 20455.31988143921,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1022.7659940719604,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 20455.437421798706,
|
||||||
|
"image_time_to_first_output_ms": 20455.294847488403,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.11576600081752986,
|
||||||
|
"stage_0_gen_ms": 20455.31988143921
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21550.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 20.56,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "288b63f713ac0664b7e0035e5f8ada494eeeb413052a35d8f3324035cb4b2bc7",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9854,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
46.9,
|
||||||
|
44.8,
|
||||||
|
47.7
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 20500.65541267395,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1025.0327706336975,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 20500.685691833496,
|
||||||
|
"image_time_to_first_output_ms": 20500.566005706787,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.09380799747304991,
|
||||||
|
"stage_0_gen_ms": 20500.65541267395
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21550.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 25.02,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "3df72b7d1334454fddc1768c6b6ff172a86cdb3ab8b075c3ff784260260bd1d1",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9638,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
69.4,
|
||||||
|
70.2,
|
||||||
|
71.8
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 24954.949617385864,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1247.7474808692932,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 24954.51283454895,
|
||||||
|
"image_time_to_first_output_ms": 24954.333305358887,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.15277299826266244,
|
||||||
|
"stage_0_gen_ms": 24954.949617385864
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22342.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 25.11,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "151ddf53141622414678c27efeafe7b928297a4cfc29038e41607069ca6968f6",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9659,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
61.9,
|
||||||
|
63.6,
|
||||||
|
65.7
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 25045.337438583374,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1252.2668719291687,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 25045.44496536255,
|
||||||
|
"image_time_to_first_output_ms": 25045.308351516724,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.11107399768661708,
|
||||||
|
"stage_0_gen_ms": 25045.337438583374
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22280.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 25.16,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "023407fe5dd8530aa1c640c3949b1b8da0893a15f07ae061a7ece4c132a7931b",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9572,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
61.6,
|
||||||
|
63.2,
|
||||||
|
64.9
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 25092.447519302368,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1254.6223759651184,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 25092.57960319519,
|
||||||
|
"image_time_to_first_output_ms": 25092.416524887085,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.13619099627248943,
|
||||||
|
"stage_0_gen_ms": 25092.447519302368
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22280.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 25.17,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "7e46495775ee73c2ccc21e252c351311c7e0976a59ef0d9483b14cfcc0fa27f7",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9471,
|
||||||
|
"alpha_mean": 254.9,
|
||||||
|
"rgb_std": [
|
||||||
|
57.5,
|
||||||
|
55.8,
|
||||||
|
55.6
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 25105.11350631714,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1255.255675315857,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 25105.209827423096,
|
||||||
|
"image_time_to_first_output_ms": 25105.08370399475,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.10113100142916664,
|
||||||
|
"stage_0_gen_ms": 25105.11350631714
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22280.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 29.37,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "1b3430e99b76ed0483c37fbdbacdb3a890e08af69a5a9c77388cb1665c54fa19",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9785,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
42.9,
|
||||||
|
32.4,
|
||||||
|
23.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 29299.254417419434,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1464.9627208709717,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 29299.4122505188,
|
||||||
|
"image_time_to_first_output_ms": 29299.22580718994,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.1593470005900599,
|
||||||
|
"stage_0_gen_ms": 29299.254417419434
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23432.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 29.44,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "67ed3ba31c5db970ddda42dc38e5f75e395d83aae1c467625060a5bc1034906d",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9882,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
37.9,
|
||||||
|
30.5,
|
||||||
|
21.6
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 29367.507219314575,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1468.3753609657288,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 29367.653369903564,
|
||||||
|
"image_time_to_first_output_ms": 29367.475748062134,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.15194500156212598,
|
||||||
|
"stage_0_gen_ms": 29367.507219314575
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23432.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 29.47,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "ed8493e310da3d892eeba849519fe8697e638c215eb01bccad96608b9040e70b",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9757,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
48.5,
|
||||||
|
40.2,
|
||||||
|
30.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 29401.696920394897,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1470.0848460197449,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 29401.797771453857,
|
||||||
|
"image_time_to_first_output_ms": 29401.665925979614,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.10474099690327421,
|
||||||
|
"stage_0_gen_ms": 29401.696920394897
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23432.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 29.49,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "eca7238f811f5980b514adde1f0727fc6ce6d33efe16bd620cef184831ed1b44",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9645,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
48.5,
|
||||||
|
35.7,
|
||||||
|
25.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 29419.892072677612,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1470.9946036338806,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 29419.984102249146,
|
||||||
|
"image_time_to_first_output_ms": 29419.86107826233,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.09873499948298559,
|
||||||
|
"stage_0_gen_ms": 29419.892072677612
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23432.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 34.08,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "ed2978bbee6c6d667a1e113fddb6e5d7c06d9b0c7c0d8eda77ad330b92bdc45f",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.976,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
49.0,
|
||||||
|
37.3,
|
||||||
|
29.3
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 34010.05530357361,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1700.5027651786804,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 34010.1637840271,
|
||||||
|
"image_time_to_first_output_ms": 34010.02025604248,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.11569900379981846,
|
||||||
|
"stage_0_gen_ms": 34010.05530357361
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23598.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 34.0,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "f4433739a48bdb32c24f0b53d4feee604adad91327d0b25656cb0b28d4051e1b",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9713,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
47.2,
|
||||||
|
39.5,
|
||||||
|
29.8
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 33924.51333999634,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1696.225666999817,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 33924.62372779846,
|
||||||
|
"image_time_to_first_output_ms": 33924.47805404663,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.11932000052183867,
|
||||||
|
"stage_0_gen_ms": 33924.51333999634
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23590.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 34.02,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "9b673d3e82dc29d8781dc9d2e0a40bdd7dcad746332f9566e80e5bd8a055bd30",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9706,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
56.4,
|
||||||
|
44.8,
|
||||||
|
35.4
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 33950.94609260559,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1697.5473046302795,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 33951.04670524597,
|
||||||
|
"image_time_to_first_output_ms": 33950.846672058105,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.17462699906900525,
|
||||||
|
"stage_0_gen_ms": 33950.94609260559
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23590.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 34.04,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "22d9558389165abb8e58c02e7f1c4d49d4f312ff178062246c9907d519336ad6",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.9674,
|
||||||
|
"alpha_mean": 255.0,
|
||||||
|
"rgb_std": [
|
||||||
|
50.9,
|
||||||
|
38.3,
|
||||||
|
29.5
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 33973.530292510986,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 1698.6765146255493,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 33973.62184524536,
|
||||||
|
"image_time_to_first_output_ms": 33973.4947681427,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.09957399743143469,
|
||||||
|
"stage_0_gen_ms": 33973.530292510986
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23590.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
{
|
||||||
|
"schema": "issue5-int8wo-v1",
|
||||||
|
"created_utc": "2026-10-08T11:16:41+00:00",
|
||||||
|
"description": "Qwen-Image-2.1 DiT int8 weight-only (W8A16), compressed-tensors pack-quantized",
|
||||||
|
"inputs": {
|
||||||
|
"snapshot": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/snapshots/790c92633540aa0cb11d9abf19eb46d861714758",
|
||||||
|
"revision": "790c92633540aa0cb11d9abf19eb46d861714758",
|
||||||
|
"tree_manifest": "/home/anodyine/.local/state/multi-model-helper-server/state/image-service/weights/hub/models--Qwen--Qwen-Image-2.1/trees/790c92633540aa0cb11d9abf19eb46d861714758.json",
|
||||||
|
"tree_sha256": "1f35d0a5a99f2c79bf2a6818b96582828094feb8544a992bc81c1d4100aed436"
|
||||||
|
},
|
||||||
|
"versions": {
|
||||||
|
"python": "3.13.15",
|
||||||
|
"torch": "2.13.0+cu130",
|
||||||
|
"compressed_tensors": "0.17.0"
|
||||||
|
},
|
||||||
|
"quantization": {
|
||||||
|
"method": "compressed-tensors",
|
||||||
|
"format": "pack-quantized",
|
||||||
|
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||||
|
"config": {
|
||||||
|
"config_groups": {
|
||||||
|
"group_0": {
|
||||||
|
"targets": [
|
||||||
|
"Linear"
|
||||||
|
],
|
||||||
|
"weights": {
|
||||||
|
"num_bits": 8,
|
||||||
|
"type": "int",
|
||||||
|
"symmetric": true,
|
||||||
|
"group_size": null,
|
||||||
|
"strategy": "channel",
|
||||||
|
"block_structure": null,
|
||||||
|
"dynamic": false,
|
||||||
|
"actorder": null,
|
||||||
|
"scale_dtype": null,
|
||||||
|
"zp_dtype": null,
|
||||||
|
"observer": "minmax",
|
||||||
|
"observer_kwargs": {}
|
||||||
|
},
|
||||||
|
"input_activations": null,
|
||||||
|
"output_activations": null,
|
||||||
|
"format": null
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"quant_method": "compressed-tensors",
|
||||||
|
"kv_cache_scheme": null,
|
||||||
|
"format": "pack-quantized",
|
||||||
|
"quantization_status": "frozen",
|
||||||
|
"global_compression_ratio": null,
|
||||||
|
"ignore": [
|
||||||
|
"img_in",
|
||||||
|
"txt_in",
|
||||||
|
"time_text_embed",
|
||||||
|
"modulation",
|
||||||
|
"norm_out",
|
||||||
|
"proj_out"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"ignored_layers": [
|
||||||
|
"img_in",
|
||||||
|
"txt_in",
|
||||||
|
"time_text_embed",
|
||||||
|
"modulation",
|
||||||
|
"norm_out",
|
||||||
|
"proj_out"
|
||||||
|
],
|
||||||
|
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers"
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"quantized_tensors": 224,
|
||||||
|
"quantized_params": 6979321856,
|
||||||
|
"bf16_params": 135802880,
|
||||||
|
"mean_rel_l2": 0.010959165191577216,
|
||||||
|
"max_rel_l2": 0.017284497618675232,
|
||||||
|
"weights_file_bytes": 7256783616
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
{
|
||||||
|
"path": "transformer/config.json",
|
||||||
|
"bytes": 1405,
|
||||||
|
"sha256": "43b8c94380afb473f252d2707d1398b2be180f0248acee31cad4d6613f4c0a84"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": "transformer/diffusion_pytorch_model.safetensors",
|
||||||
|
"bytes": 7256783616,
|
||||||
|
"sha256": "947d9c0660776b1125ec98762817af55d59d326b8e223120aa1bdb0eced91aa9"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
transformer_blocks.0.attn.to_k.weight: shape (4096, 4096) rel_l2=0.01092 scale (4096, 1)/torch.float32
|
||||||
|
transformer_blocks.0.attn.to_out.0.weight: shape (4096, 4096) rel_l2=0.00978 scale (4096, 1)/torch.float32
|
||||||
|
transformer_blocks.0.attn.to_q.weight: shape (4096, 4096) rel_l2=0.01106 scale (4096, 1)/torch.float32
|
||||||
|
transformer_blocks.0.attn.to_v.weight: shape (4096, 4096) rel_l2=0.00976 scale (4096, 1)/torch.float32
|
||||||
|
checked 4 tensors
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
== int8wo ==
|
||||||
|
1-ref: n=4 gate=FAIL wall 30.5-31.7s (mean 31.0s)
|
||||||
|
edit-expression-ysolde s2301: 31.73s {"opaque_fraction": 0.0294, "alpha_mean": 114.2, "rgb_std": [59.9, 61.7, 63.8]} BROKEN
|
||||||
|
edit-expression-ysolde s2302: 30.53s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.2, 2.9, 3.3]} BROKEN
|
||||||
|
edit-expression-ysolde s2303: 30.88s {"opaque_fraction": 0.0391, "alpha_mean": 125.5, "rgb_std": [63.5, 66.2, 67.0]} BROKEN
|
||||||
|
edit-expression-ysolde s2304: 30.95s {"opaque_fraction": 0.0352, "alpha_mean": 122.4, "rgb_std": [63.6, 66.3, 67.1]} BROKEN
|
||||||
|
2-ref: n=4 gate=FAIL wall 36.5-36.7s (mean 36.6s)
|
||||||
|
edit-compose-pair s2301: 36.46s {"opaque_fraction": 0.0303, "alpha_mean": 115.4, "rgb_std": [59.2, 61.0, 63.3]} BROKEN
|
||||||
|
edit-compose-pair s2302: 36.57s {"opaque_fraction": 0.0, "alpha_mean": 126.9, "rgb_std": [3.1, 2.9, 3.2]} BROKEN
|
||||||
|
edit-compose-pair s2303: 36.61s {"opaque_fraction": 0.03, "alpha_mean": 115.7, "rgb_std": [76.1, 77.9, 69.1]} BROKEN
|
||||||
|
edit-compose-pair s2304: 36.69s {"opaque_fraction": 0.1257, "alpha_mean": 127.1, "rgb_std": [64.3, 64.3, 64.3]} BROKEN
|
||||||
|
3-ref: n=4 gate=FAIL wall 41.3-41.6s (mean 41.4s)
|
||||||
|
edit-compose-three s2301: 41.57s {"opaque_fraction": 0.2029, "alpha_mean": 137.2, "rgb_std": [93.0, 92.7, 91.0]} BROKEN
|
||||||
|
edit-compose-three s2302: 41.32s {"opaque_fraction": 0.0193, "alpha_mean": 131.9, "rgb_std": [40.8, 45.8, 44.3]} BROKEN
|
||||||
|
edit-compose-three s2303: 41.34s {"opaque_fraction": 0.0188, "alpha_mean": 131.6, "rgb_std": [41.0, 45.9, 44.6]} BROKEN
|
||||||
|
edit-compose-three s2304: 41.39s {"opaque_fraction": 0.0189, "alpha_mean": 131.6, "rgb_std": [41.1, 46.0, 44.6]} BROKEN
|
||||||
|
4-ref: n=4 gate=FAIL wall 46.4-46.5s (mean 46.4s)
|
||||||
|
edit-compose-four s2301: 46.4s {"opaque_fraction": 0.0195, "alpha_mean": 131.6, "rgb_std": [41.4, 46.3, 45.0]} BROKEN
|
||||||
|
edit-compose-four s2302: 46.42s {"opaque_fraction": 0.0196, "alpha_mean": 131.9, "rgb_std": [41.1, 46.0, 44.5]} BROKEN
|
||||||
|
edit-compose-four s2303: 46.44s {"opaque_fraction": 0.0195, "alpha_mean": 132.0, "rgb_std": [41.1, 46.1, 44.7]} BROKEN
|
||||||
|
edit-compose-four s2304: 46.48s {"opaque_fraction": 0.0195, "alpha_mean": 131.7, "rgb_std": [41.0, 46.1, 44.5]} BROKEN
|
||||||
|
nvidia-smi peak: 23935 MiB; server peak: None MB
|
||||||
|
STRUCTURE GATE: FAIL
|
||||||
|
== psnr edit-expression-ysolde ==
|
||||||
|
{
|
||||||
|
"int8wo": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 6.95,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 8.3,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 6.67,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 6.65,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 7.14,
|
||||||
|
"min_db": 6.65,
|
||||||
|
"max_db": 8.3,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
== psnr edit-compose-pair ==
|
||||||
|
{
|
||||||
|
"int8wo": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 7.03,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 8.86,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 6.77,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 7.19,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 7.46,
|
||||||
|
"min_db": 6.77,
|
||||||
|
"max_db": 8.86,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
== psnr edit-compose-four ==
|
||||||
|
{
|
||||||
|
"int8wo": {
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"seed": 2301,
|
||||||
|
"psnr_db": 7.02,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2302,
|
||||||
|
"psnr_db": 7.11,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2303,
|
||||||
|
"psnr_db": 7.1,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"seed": 2304,
|
||||||
|
"psnr_db": 7.2,
|
||||||
|
"modes": "RGBA/RGBA"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"mean_db": 7.11,
|
||||||
|
"min_db": 7.02,
|
||||||
|
"max_db": 7.2,
|
||||||
|
"n": 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
23935 MiB, 100 %
|
||||||
|
(APIServer pid=473436) INFO 10-08 07:36:28 [data.py:1483] Auto-detected quantization 'component' from model config
|
||||||
|
(DiffusionWorker pid=473436) INFO 10-08 07:36:28 [diffusers_loader.py:719] Offline-quantized model with CPU offload, loading weights directly on CPU
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"layer": "attn.to_qkv (fused q/k/v)",
|
||||||
|
"rel_l2": "NaN",
|
||||||
|
"out_finite": false,
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.0006612942670471966,
|
||||||
|
"max": 0.012303149327635765
|
||||||
|
},
|
||||||
|
"out_shape": [
|
||||||
|
512,
|
||||||
|
12288
|
||||||
|
],
|
||||||
|
"ref_shape": [
|
||||||
|
512,
|
||||||
|
12288
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "attn.to_out.0 (row)",
|
||||||
|
"rel_l2": "NaN",
|
||||||
|
"out_finite": false,
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.00037870631786063313,
|
||||||
|
"max": 0.011257382109761238
|
||||||
|
},
|
||||||
|
"out_shape": [
|
||||||
|
512,
|
||||||
|
4096
|
||||||
|
],
|
||||||
|
"ref_shape": [
|
||||||
|
512,
|
||||||
|
4096
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "img_mlp.proj (column)",
|
||||||
|
"rel_l2": "NaN",
|
||||||
|
"out_finite": false,
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.0005651759565807879,
|
||||||
|
"max": 0.005628691054880619
|
||||||
|
},
|
||||||
|
"out_shape": [
|
||||||
|
512,
|
||||||
|
12288
|
||||||
|
],
|
||||||
|
"ref_shape": [
|
||||||
|
512,
|
||||||
|
12288
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "img_mlp.out (row, wide-in)",
|
||||||
|
"rel_l2": "NaN",
|
||||||
|
"out_finite": false,
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.0008073942153714597,
|
||||||
|
"max": 0.014763779938220978
|
||||||
|
},
|
||||||
|
"out_shape": [
|
||||||
|
512,
|
||||||
|
4096
|
||||||
|
],
|
||||||
|
"ref_shape": [
|
||||||
|
512,
|
||||||
|
4096
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"layer": "attn.to_qkv (fused q/k/v)",
|
||||||
|
"rel_l2_by_act_dtype": {
|
||||||
|
"bfloat16": "NaN",
|
||||||
|
"float16": "NaN"
|
||||||
|
},
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.02410734072327614,
|
||||||
|
"max": 0.043380916118621826
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "attn.to_out.0 (row)",
|
||||||
|
"rel_l2_by_act_dtype": {
|
||||||
|
"bfloat16": "NaN",
|
||||||
|
"float16": "NaN"
|
||||||
|
},
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.02417764998972416,
|
||||||
|
"max": 0.042702700942754745
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "img_mlp.proj (column)",
|
||||||
|
"rel_l2_by_act_dtype": {
|
||||||
|
"bfloat16": "NaN",
|
||||||
|
"float16": "NaN"
|
||||||
|
},
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.024067752063274384,
|
||||||
|
"max": 0.04336588457226753
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"layer": "img_mlp.out (row, wide-in)",
|
||||||
|
"rel_l2_by_act_dtype": {
|
||||||
|
"bfloat16": "NaN",
|
||||||
|
"float16": "NaN"
|
||||||
|
},
|
||||||
|
"packed_bitwise_ok": true,
|
||||||
|
"scale_finite_pre_repack": true,
|
||||||
|
"scale_post_repack": {
|
||||||
|
"finite": true,
|
||||||
|
"min": 0.02682814560830593,
|
||||||
|
"max": 0.04223968833684921
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,978 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 31.73,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "a62cc903858a5026c08f284bdfea58108068fd6ab6425b54c246d65990e0bec4",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0294,
|
||||||
|
"alpha_mean": 114.2,
|
||||||
|
"rgb_std": [
|
||||||
|
59.9,
|
||||||
|
61.7,
|
||||||
|
63.8
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 31616.730451583862,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1580.8365225791931,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 31616.933584213257,
|
||||||
|
"image_time_to_first_output_ms": 31616.7151927948,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.18030800129054114,
|
||||||
|
"stage_0_gen_ms": 31616.730451583862
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21594.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 30.53,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "ae5c51a9a36f8892c15116b90bcd2b85f90a83bae3daea2196f2602b5968e9af",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0,
|
||||||
|
"alpha_mean": 126.9,
|
||||||
|
"rgb_std": [
|
||||||
|
3.2,
|
||||||
|
2.9,
|
||||||
|
3.3
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 30463.57011795044,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1523.178505897522,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 30463.69981765747,
|
||||||
|
"image_time_to_first_output_ms": 30463.540077209473,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.12335599603829905,
|
||||||
|
"stage_0_gen_ms": 30463.57011795044
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21900.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 30.88,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "d05825c0d525ff8522d83db2a9fc6337cc06a3960f7eb2a112898c5b71aa34a1",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0391,
|
||||||
|
"alpha_mean": 125.5,
|
||||||
|
"rgb_std": [
|
||||||
|
63.5,
|
||||||
|
66.2,
|
||||||
|
67.0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 30779.66594696045,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1538.9832973480225,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 30779.80923652649,
|
||||||
|
"image_time_to_first_output_ms": 30779.641151428223,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.14197500422596931,
|
||||||
|
"stage_0_gen_ms": 30779.66594696045
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21906.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-expression-ysolde",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 1,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "864x1152",
|
||||||
|
"wall_s": 30.95,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 3983827,
|
||||||
|
"sha256": "10e85cf24451f4fb7bf2d8e07b4039158af08e36467a2a14d2baee1d68cd98fa",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0352,
|
||||||
|
"alpha_mean": 122.4,
|
||||||
|
"rgb_std": [
|
||||||
|
63.6,
|
||||||
|
66.3,
|
||||||
|
67.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 30871.540069580078,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 995328,
|
||||||
|
"denoise_step_latency_ms": 1543.577003479004,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 30871.644735336304,
|
||||||
|
"image_time_to_first_output_ms": 30871.514797210693,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.10297299741068855,
|
||||||
|
"stage_0_gen_ms": 30871.540069580078
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 21906.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 36.46,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "4bfdf57ecb0e1ed1c2c4dcb10ce0545d89040ad08ed754e01adbe2ddd2b407d2",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0303,
|
||||||
|
"alpha_mean": 115.4,
|
||||||
|
"rgb_std": [
|
||||||
|
59.2,
|
||||||
|
61.0,
|
||||||
|
63.3
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 36376.55758857727,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1818.8278794288635,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 36376.71780586243,
|
||||||
|
"image_time_to_first_output_ms": 36376.53112411499,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.15923400496831164,
|
||||||
|
"stage_0_gen_ms": 36376.55758857727
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22676.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 36.57,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "e5ad17605624fdb636dc02e272c0dba2d3286dcfa4c27f297a605e785f3177a2",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0,
|
||||||
|
"alpha_mean": 126.9,
|
||||||
|
"rgb_std": [
|
||||||
|
3.1,
|
||||||
|
2.9,
|
||||||
|
3.2
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 36508.753061294556,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1825.4376530647278,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 36508.851528167725,
|
||||||
|
"image_time_to_first_output_ms": 36508.72349739075,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.1011220010695979,
|
||||||
|
"stage_0_gen_ms": 36508.753061294556
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22556.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 36.61,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "f58a982cf1f3afd4a81fcc6b349aa3b147fecea0594e33f49cab5673f300b2ac",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.03,
|
||||||
|
"alpha_mean": 115.7,
|
||||||
|
"rgb_std": [
|
||||||
|
76.1,
|
||||||
|
77.9,
|
||||||
|
69.1
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 36523.38361740112,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1826.1691808700562,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 36523.181676864624,
|
||||||
|
"image_time_to_first_output_ms": 36523.05722236633,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.0983119971351698,
|
||||||
|
"stage_0_gen_ms": 36523.38361740112
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22554.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-pair",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 2,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1248x832",
|
||||||
|
"wall_s": 36.69,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4155590,
|
||||||
|
"sha256": "871c8147b27310508fb2c9e72b3b47499d7f8ab4ea705719aa30c465c8a5476a",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.1257,
|
||||||
|
"alpha_mean": 127.1,
|
||||||
|
"rgb_std": [
|
||||||
|
64.3,
|
||||||
|
64.3,
|
||||||
|
64.3
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 36608.7908744812,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1038336,
|
||||||
|
"denoise_step_latency_ms": 1830.43954372406,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 36608.89911651611,
|
||||||
|
"image_time_to_first_output_ms": 36608.76417160034,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.10716100223362446,
|
||||||
|
"stage_0_gen_ms": 36608.7908744812
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 22556.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 41.57,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "810cb635438bac76de76fa44172687e2f18b212379458580a32bbddd028bd78c",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.2029,
|
||||||
|
"alpha_mean": 137.2,
|
||||||
|
"rgb_std": [
|
||||||
|
93.0,
|
||||||
|
92.7,
|
||||||
|
91.0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 41476.71937942505,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2073.8359689712524,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 41476.8271446228,
|
||||||
|
"image_time_to_first_output_ms": 41476.691246032715,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.1088700009859167,
|
||||||
|
"stage_0_gen_ms": 41476.71937942505
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23468.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 41.32,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "2f085a5089ab0618a2f17df7c29c52fde091247cb4a1ccf0cd8d662171b18fc9",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0193,
|
||||||
|
"alpha_mean": 131.9,
|
||||||
|
"rgb_std": [
|
||||||
|
40.8,
|
||||||
|
45.8,
|
||||||
|
44.3
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 41233.43777656555,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2061.6718888282776,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 41233.57892036438,
|
||||||
|
"image_time_to_first_output_ms": 41233.40702056885,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.14587400073651224,
|
||||||
|
"stage_0_gen_ms": 41233.43777656555
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23460.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 41.34,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "81e9b7436d392c8420d1529976a0d88e5de03af1e70823cc8c7b279b3d7bb3fa",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0188,
|
||||||
|
"alpha_mean": 131.6,
|
||||||
|
"rgb_std": [
|
||||||
|
41.0,
|
||||||
|
45.9,
|
||||||
|
44.6
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 41250.11157989502,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2062.505578994751,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 41250.218868255615,
|
||||||
|
"image_time_to_first_output_ms": 41250.08296966553,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.10969900176860392,
|
||||||
|
"stage_0_gen_ms": 41250.11157989502
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23394.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-three",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 3,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 41.39,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "2221932a1783102d87c01157e231160803cebac2113c86b9416e537e0e738747",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0189,
|
||||||
|
"alpha_mean": 131.6,
|
||||||
|
"rgb_std": [
|
||||||
|
41.1,
|
||||||
|
46.0,
|
||||||
|
44.6
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 41297.605991363525,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2064.8802995681763,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 41297.31011390686,
|
||||||
|
"image_time_to_first_output_ms": 41297.15037345886,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.134524998429697,
|
||||||
|
"stage_0_gen_ms": 41297.605991363525
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23394.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2301,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 46.4,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "abd057bee84e594437f895cc9f51f90f118adfeb4dbf738686791b41487e93b2",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0195,
|
||||||
|
"alpha_mean": 131.6,
|
||||||
|
"rgb_std": [
|
||||||
|
41.4,
|
||||||
|
46.3,
|
||||||
|
45.0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 46304.86750602722,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2315.243375301361,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 46304.97694015503,
|
||||||
|
"image_time_to_first_output_ms": 46304.8369884491,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.11391400039428845,
|
||||||
|
"stage_0_gen_ms": 46304.86750602722
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23534.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2302,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 46.42,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "24a783a9ca3a002745351476e8000be87854d1364ce92ca944c2986a983922f5",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0196,
|
||||||
|
"alpha_mean": 131.9,
|
||||||
|
"rgb_std": [
|
||||||
|
41.1,
|
||||||
|
46.0,
|
||||||
|
44.5
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 46325.687408447266,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2316.2843704223633,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 46324.424266815186,
|
||||||
|
"image_time_to_first_output_ms": 46324.246883392334,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.15303100371966138,
|
||||||
|
"stage_0_gen_ms": 46325.687408447266
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23524.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2303,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 46.44,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "db507c15651324404b79d6e5446e43fb460797204b302c0c258be37f1e2a7a80",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0195,
|
||||||
|
"alpha_mean": 132.0,
|
||||||
|
"rgb_std": [
|
||||||
|
41.1,
|
||||||
|
46.1,
|
||||||
|
44.7
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 46352.77342796326,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2317.638671398163,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 46352.887868881226,
|
||||||
|
"image_time_to_first_output_ms": 46352.74004936218,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.1095690022339113,
|
||||||
|
"stage_0_gen_ms": 46352.77342796326
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23524.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fixture_id": "edit-compose-four",
|
||||||
|
"seed": 2304,
|
||||||
|
"n_references": 4,
|
||||||
|
"steps": 20,
|
||||||
|
"true_cfg_scale": 1.0,
|
||||||
|
"size": "1344x768",
|
||||||
|
"wall_s": 46.48,
|
||||||
|
"http": 200,
|
||||||
|
"bytes": 4130915,
|
||||||
|
"sha256": "5481e029af2cb8ef3140565c615a5abc97ed755ff180b1970badf3a997c60ac4",
|
||||||
|
"structure": {
|
||||||
|
"opaque_fraction": 0.0195,
|
||||||
|
"alpha_mean": 131.7,
|
||||||
|
"rgb_std": [
|
||||||
|
41.0,
|
||||||
|
46.1,
|
||||||
|
44.5
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"usage": null,
|
||||||
|
"metrics": {
|
||||||
|
"stage_metrics": {
|
||||||
|
"0": {
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"finish_reason": null,
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_gen_time_ms": 46385.611057281494,
|
||||||
|
"postprocess_time_ms": 0.0,
|
||||||
|
"audio_frames": 0,
|
||||||
|
"audio_sample_rate": 0,
|
||||||
|
"audio_duration_s": 0.0,
|
||||||
|
"image_pixels": 1032192,
|
||||||
|
"denoise_step_latency_ms": 2319.2805528640747,
|
||||||
|
"output_unit_type": "image",
|
||||||
|
"output_unit_count": 1,
|
||||||
|
"serving_time_to_first_output_ms": 46385.305404663086,
|
||||||
|
"image_time_to_first_output_ms": 46385.130643844604,
|
||||||
|
"time_per_output_unit_ms": 0.0,
|
||||||
|
"inter_output_latency_ms": 0.0,
|
||||||
|
"inter_output_latencies_ms": [],
|
||||||
|
"vllm_ttft_ms": 0.0,
|
||||||
|
"vllm_tpot_ms": 0.0,
|
||||||
|
"vllm_itl_ms": 0.0,
|
||||||
|
"vllm_itls_ms": [],
|
||||||
|
"stage_name": "diffusion"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"stage_id": 0,
|
||||||
|
"final_output_type": "image",
|
||||||
|
"num_tokens_in": 0,
|
||||||
|
"num_tokens_out": 0,
|
||||||
|
"stage_durations": {
|
||||||
|
"queue_wait_ms": 0.14768500113859773,
|
||||||
|
"stage_0_gen_ms": 46385.611057281494
|
||||||
|
},
|
||||||
|
"peak_memory_mb": 23524.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
{
|
||||||
|
"schema": "issue5-runs-summary-v2",
|
||||||
|
"probed_utc": "2026-10-08",
|
||||||
|
"round_two": {
|
||||||
|
"directive": "issue 5 comment 2026-10-08T07:51 (amended 08:37): one decisive experiment - W8A8 int8-dynamic, non-Marlin",
|
||||||
|
"artifact": "~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-w8a8dyn",
|
||||||
|
"kernel": "CutlassInt8ScaledMMLinearKernel (ops.scaled_int8_quant + ops.cutlass_scaled_mm)",
|
||||||
|
"kernel_verification": {
|
||||||
|
"raw_ops_rel_l2": "0.0017 at (64,256,128)/(512,4096,4096)/(512,4096,12288) vs float32 reference (cutlass-ops-probe.json in evidence/rtx3090-round2)",
|
||||||
|
"layer_path_rel_l2": "0.0017 for fused QKV and row-parallel, synthetic weights, loaded bitwise-verified (w8a8-layer-isolate.json)"
|
||||||
|
},
|
||||||
|
"serve_run": {
|
||||||
|
"label": "int8w8a8dyn",
|
||||||
|
"init": "healthy in ~40 s",
|
||||||
|
"wall_s_mean": {"1-ref": 20.5, "2-ref": 25.1, "3-ref": 29.4, "4-ref": 34.0},
|
||||||
|
"nvidia_smi_peak_mib": 23879,
|
||||||
|
"structure": "opaque 0.947-0.988, alpha mean 254.9-255.0, rgb_std 21-72 across all 16 requests; visual checks render faithful scenes",
|
||||||
|
"gate_note": "round-one 0.99-opacity rule was a broken-vs-BF16 calibration; W8A8 outputs are soft-edge alpha, not the broken signature (0.008-0.20 band); verdict rests on raw numbers + visual + PSNR",
|
||||||
|
"psnr_vs_sealed_diffusers_mean_db": {"1-ref": 33.72, "2-ref": 28.34, "4-ref": 27.50},
|
||||||
|
"bf16_reference_pair_db": 27.57
|
||||||
|
},
|
||||||
|
"verdict": "YES - W8A8 int8-dynamic delivers 3-4 reference capacity at the fastest measured speed with BF16-parity quality; hard stop did not trigger; Marlin W8A16 root-cause draft staged for the #3 lane (UPSTREAM-REPORT-DRAFT.md)"
|
||||||
|
},
|
||||||
|
"host": "cachyos-alexander (RTX 3090 24 GB, sm_86, driver 615.71.09, CUDA 13.4)",
|
||||||
|
"stack": {
|
||||||
|
"vllm_omni": "ISSUE-5 branch e778d21f..67e17786 over main 8484d85d1",
|
||||||
|
"vllm": "0.31.0",
|
||||||
|
"torch": "2.13.0+cu130",
|
||||||
|
"compressed_tensors": "0.17.0",
|
||||||
|
"transformers": "5.14.1",
|
||||||
|
"diffusers": "0.41.0.dev0 (git 122b1e11)"
|
||||||
|
},
|
||||||
|
"artifact": {
|
||||||
|
"path": "~/.local/state/multi-model-helper-server/state/image-service/artifacts/qwen-image-2.1-dit-int8wo",
|
||||||
|
"quantized_params": 6979321856,
|
||||||
|
"bf16_boundary_params": 135802880,
|
||||||
|
"weights_file_bytes": 7256783616,
|
||||||
|
"mean_rel_l2_quant_error": 0.010959,
|
||||||
|
"max_rel_l2_quant_error": 0.017284,
|
||||||
|
"roundtrip_verification": "unpack+dequant vs sealed BF16: rel_l2 0.0098-0.0117 per tensor (see artifact-verify.txt)"
|
||||||
|
},
|
||||||
|
"serve_run": {
|
||||||
|
"label": "int8wo",
|
||||||
|
"command": "vllm serve <artifact> --omni --port 8091 --served-model-name Qwen/Qwen-Image-2.1 --enforce-eager --enable-cpu-offload",
|
||||||
|
"init": "healthy in ~40 s; per-component quantization auto-detected from transformer/config.json (transformer -> compressed-tensors, encoder/VAE BF16); CompressedTensorsWNA16 + MarlinLinearKernel selected on sm_86",
|
||||||
|
"failed_precursor_runs": [
|
||||||
|
"int8wo (first): init OOM - loader treated the pre-quantized checkpoint as online quantization and loaded all components on the GPU (encoder 17 GiB + VAE + packed DiT skeleton > 24 GiB); fixed by the offline-detection loader change on this branch",
|
||||||
|
"int8wo-expandable: same OOM with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (genuine capacity, not fragmentation); also polluted by a leftover 586 MiB DiffusionWorker from an interrupted test run"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"capacity_matrix": {
|
||||||
|
"fixtures": {"1-ref": "edit-expression-ysolde", "2-ref": "edit-compose-pair", "3-ref": "edit-compose-three", "4-ref": "edit-compose-four"},
|
||||||
|
"seeds": [2301, 2302, 2303, 2304],
|
||||||
|
"requests": 16,
|
||||||
|
"http_200": 16,
|
||||||
|
"wall_s_mean": {"1-ref": 31.0, "2-ref": 36.6, "3-ref": 41.4, "4-ref": 46.4},
|
||||||
|
"nvidia_smi_peak_mib": 23935,
|
||||||
|
"bf16_reference_walls": {"1-ref": 34.2, "2-ref": 36.7, "4-ref": "OOM at BF16 (issue 2)"},
|
||||||
|
"fp8_reference_walls": {"1-ref": 29.9, "2-ref": 34.1, "3-ref": 38.9, "4-ref": 44.3}
|
||||||
|
},
|
||||||
|
"structure_gate": {
|
||||||
|
"rule": "opaque_fraction >= 0.99 AND mean(rgb_std) >= 10",
|
||||||
|
"result": "FAIL 16/16",
|
||||||
|
"signature": "opaque 0.0-0.20 (vs 1.0 at BF16), alpha mean 114-137, rgb_std 3-93; outputs are horizontal-stripe noise, not rendered scenes",
|
||||||
|
"detail": "gate-and-psnr-summary.txt; per-request records in matrix.json"
|
||||||
|
},
|
||||||
|
"psnr_vs_sealed_diffusers_baseline": {
|
||||||
|
"edit-expression-ysolde_1ref_mean_db": 7.14,
|
||||||
|
"edit-compose-pair_2ref_mean_db": 7.46,
|
||||||
|
"edit-compose-four_4ref_mean_db": 7.11,
|
||||||
|
"bf16_pair_reference_mean_db": 27.57,
|
||||||
|
"fp8_broken_reference_mean_db": 7.11
|
||||||
|
},
|
||||||
|
"marlin_isolation": {
|
||||||
|
"method": "marlin_isolate.py: vLLM QKVParallelLinear/RowParallelLinear/ColumnParallelLinear built exactly as the qwen_image_21 transformer does; artifact tensors loaded through the production weight-loaders; Marlin repack on GPU; layer(x) compared against float32 x @ dequant(W)^T",
|
||||||
|
"results_block0_artifact": "NaN output, all four layer kinds, bf16 AND fp16 activations; loaded weight_packed bitwise-equal to artifact (incl. fused q/k/v stacked shards); scales finite pre and post repack (marlin-isolate-block0.json)",
|
||||||
|
"results_synthetic_random_int8": "NaN identically - the checkpoint and its quantization are exonerated (marlin-isolate-synthetic.json)",
|
||||||
|
"conclusion": "the 8-bit Marlin W8A16 path on this stack (sm_86, vllm 0.31.0, torch 2.13.0+cu130) produces NaN for int8 weight-only layers although query_marlin_supported_quant_types accepts uint8b128 on the card; choose_mp_linear_kernel offers no non-Marlin fallback for W8A16 on sm_86 (Machete is sm90+, ExLlama int4-only)"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
{
|
||||||
|
"edit-expression-ysolde": {
|
||||||
|
"mode": "edit",
|
||||||
|
"references": [
|
||||||
|
"ref-portrait-ysolde"
|
||||||
|
],
|
||||||
|
"width": 864,
|
||||||
|
"height": 1152,
|
||||||
|
"prompt": "Change only her expression and the chart: she now laughs openly and lowers the chart; keep her face, hair, clothing, the room, the framing, and the lighting as in the reference. Result: Ysolde laughs at something off-frame in the map room. Emotional tone: sudden delight.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Laughing openly, the chart lowered to her chest.\nSetting: The map room of a harbor guildhall, charts pinned to timber walls, rain on the window.\nComposition: close-up; head and shoulders, three-quarter view, face in sharp focus; lighting: cool grey window light from the left with a warm lamp glow from below\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
|
||||||
|
"negative_prompt": null
|
||||||
|
},
|
||||||
|
"edit-compose-pair": {
|
||||||
|
"mode": "edit",
|
||||||
|
"references": [
|
||||||
|
"ref-portrait-ysolde",
|
||||||
|
"ref-cutout-tamsin"
|
||||||
|
],
|
||||||
|
"width": 1248,
|
||||||
|
"height": 832,
|
||||||
|
"prompt": "Place the woman from the first reference and the figure from the second reference together; keep both identities exactly. Result: Ysolde and Tamsin share one unrolled map at a harbor wall. Emotional tone: wary cooperation.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Holding one end of an unrolled map, explaining a route.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Holding the other end of the map, listening sceptically.\nSetting: A sea wall above the harbor at midday, fishing boats below.\nComposition: medium shot; both figures side by side, the map between them; lighting: bright midday sun with short shadows\nStyle: Semi-realistic digital painting: natural proportions, painterly texture, soft realistic lighting.",
|
||||||
|
"negative_prompt": null
|
||||||
|
},
|
||||||
|
"edit-compose-four": {
|
||||||
|
"mode": "edit",
|
||||||
|
"references": [
|
||||||
|
"ref-portrait-ysolde",
|
||||||
|
"ref-storm-corvin",
|
||||||
|
"ref-cutout-tamsin",
|
||||||
|
"ref-shop-hesper"
|
||||||
|
],
|
||||||
|
"width": 1344,
|
||||||
|
"height": 768,
|
||||||
|
"prompt": "Bring the four characters from the references together in one scene; keep every identity exactly. Result: All four gather around the apothecary counter. Emotional tone: conspiratorial calm.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Unfolding a chart on the counter.\nSubject: Corvin Ashdown, an adult man in his mid-fifties, with pale, heavily freckled skin, short receding red hair and a grey-streaked red beard, and pale blue eyes; wearing a dark green harbormaster's greatcoat with a silver chain of office; a black leather patch over the left eye. Leaning on the doorframe.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Sniffing a stoppered bottle.\nSubject: Hesper Quill, an adult woman in her early seventies, with light brown, deeply weathered skin, long white hair in one thick plait, and grey-green eyes; wearing a lavender dress under a stained canvas apothecary's apron; round wire-rimmed spectacles and a silver ring on every finger of the right hand. Behind the counter, weighing herbs on a brass scale.\nSetting: Inside the apothecary: shelves of bottled remedies, drying herbs hanging from the beams, a long counter.\nComposition: wide shot; from behind the counter's end, all four faces visible; lighting: warm lamplight with green light filtering through bottle glass\nStyle: Anime illustration: clean line art, cel shading, flat vivid colour, expressive faces.",
|
||||||
|
"negative_prompt": null
|
||||||
|
},
|
||||||
|
"edit-compose-three": {
|
||||||
|
"mode": "edit",
|
||||||
|
"references": [
|
||||||
|
"ref-portrait-ysolde",
|
||||||
|
"ref-storm-corvin",
|
||||||
|
"ref-cutout-tamsin"
|
||||||
|
],
|
||||||
|
"width": 1344,
|
||||||
|
"height": 768,
|
||||||
|
"prompt": "Bring the three characters from the references together in one scene; keep every identity exactly. Result: All three gather around the apothecary counter. Emotional tone: conspiratorial calm.\nSubject: Ysolde Varn, an adult woman in her early thirties, with deep brown skin, close-cropped black hair with one silver-beaded braid behind the left ear, and amber eyes; wearing a teal oilskin coat with brass buttons over a cream linen shirt; a thin pale scar through the right eyebrow and a brass compass on a cord at her throat. Unfolding a chart on the counter.\nSubject: Corvin Ashdown, an adult man in his mid-fifties, with pale, heavily freckled skin, short receding red hair and a grey-streaked red beard, and pale blue eyes; wearing a dark green harbormaster's greatcoat with a silver chain of office; a black leather patch over the left eye. Leaning on the doorframe.\nSubject: Tamsin Rook, an adult androgynous person in their late twenties, with warm olive skin, straight shoulder-length silver-white hair, and dark brown eyes; wearing a crimson hooded mantle over fitted grey leather armour; three thin black geometric lines tattooed down the left cheek. Sniffing a stoppered bottle.\nSetting: Inside the apothecary: shelves of bottled remedies, drying herbs hanging from the beams, a long counter.\nComposition: wide shot; from behind the counter's end, all four faces visible; lighting: warm lamplight with green light filtering through bottle glass\nStyle: Anime illustration: clean line art, cel shading, flat vivid colour, expressive faces.",
|
||||||
|
"negative_prompt": null,
|
||||||
|
"derived_from": "edit-compose-four minus ref-shop-hesper and its Subject line; four->three wording"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 structure gate + capacity summarizer.
|
||||||
|
|
||||||
|
Applies the gate rule from issue 1's FP8 misread to every recorded request
|
||||||
|
and folds a run directory into the capacity matrix row: per reference
|
||||||
|
count, wall time, peak VRAM, and the server-reported peak.
|
||||||
|
|
||||||
|
Gate rule (a run only counts when structurally sound):
|
||||||
|
opaque_fraction >= 0.99 AND mean(rgb_std) >= 10
|
||||||
|
|
||||||
|
Broken FP8 output measured opaque 0.008-0.035 with flat grey RGB (std of a
|
||||||
|
few); the BF16 path is 100% opaque with healthy channel spread.
|
||||||
|
|
||||||
|
Usage: gate.py RUN_DIR [RUN_DIR ...] # reads matrix.json / matrix-*.json
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
|
||||||
|
|
||||||
|
def peak_vram_mb(vram_log: pathlib.Path) -> int | None:
|
||||||
|
if not vram_log.exists():
|
||||||
|
return None
|
||||||
|
peak = 0
|
||||||
|
for line in vram_log.read_text().splitlines():
|
||||||
|
match = re.match(r"\s*(\d+)\s+MiB\s*,", line) or re.match(r"\s*(\d+),", line)
|
||||||
|
if match:
|
||||||
|
peak = max(peak, int(match.group(1)))
|
||||||
|
return peak or None
|
||||||
|
|
||||||
|
|
||||||
|
def server_peak_mb(serve_log: pathlib.Path) -> int | None:
|
||||||
|
if not serve_log.exists():
|
||||||
|
return None
|
||||||
|
peaks = [int(m) for m in re.findall(r"peak.*?(\d[\d,]+)\s*MB", serve_log.read_text())]
|
||||||
|
if not peaks:
|
||||||
|
return None
|
||||||
|
return max(peaks)
|
||||||
|
|
||||||
|
|
||||||
|
def gate(structure: dict) -> bool:
|
||||||
|
return (
|
||||||
|
structure["opaque_fraction"] >= 0.99
|
||||||
|
and sum(structure["rgb_std"]) / len(structure["rgb_std"]) >= 10
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("runs", nargs="+", type=pathlib.Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
for run_dir in args.runs:
|
||||||
|
records: list[dict] = []
|
||||||
|
for candidate in sorted(run_dir.glob("matrix*.json")):
|
||||||
|
records.extend(json.loads(candidate.read_text()))
|
||||||
|
by_refs: dict[int, list[dict]] = {}
|
||||||
|
for rec in records:
|
||||||
|
by_refs.setdefault(rec.get("n_references", 0), []).append(rec)
|
||||||
|
|
||||||
|
print(f"== {run_dir.name} ==")
|
||||||
|
all_pass = bool(records)
|
||||||
|
for n_refs in sorted(by_refs):
|
||||||
|
rows = by_refs[n_refs]
|
||||||
|
walls = [r["wall_s"] for r in rows if "wall_s" in r]
|
||||||
|
gates = [gate(r["structure"]) for r in rows if "structure" in r]
|
||||||
|
row_pass = bool(rows) and all(gates) and len(gates) == len(rows)
|
||||||
|
all_pass = all_pass and row_pass
|
||||||
|
print(
|
||||||
|
f" {n_refs}-ref: n={len(rows)} gate={'PASS' if row_pass else 'FAIL'} "
|
||||||
|
f"wall {min(walls):.1f}-{max(walls):.1f}s"
|
||||||
|
+ (f" (mean {sum(walls) / len(walls):.1f}s)" if walls else "")
|
||||||
|
)
|
||||||
|
for rec in rows:
|
||||||
|
s = rec.get("structure")
|
||||||
|
tag = "" if s is None else (" ok" if gate(s) else " BROKEN")
|
||||||
|
err = f" error={rec['error'][:80]}" if rec.get("error") else ""
|
||||||
|
print(
|
||||||
|
f" {rec.get('fixture_id')} s{rec.get('seed')}: "
|
||||||
|
f"{rec.get('wall_s')}s {json.dumps(s) if s else 'no-image'}{tag}{err}"
|
||||||
|
)
|
||||||
|
peak = peak_vram_mb(run_dir / "vram.log")
|
||||||
|
server_peak = server_peak_mb(run_dir / "serve.log")
|
||||||
|
print(f" nvidia-smi peak: {peak} MiB; server peak: {server_peak} MB")
|
||||||
|
print(f" STRUCTURE GATE: {'PASS' if all_pass else 'FAIL'}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,239 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 diagnostic: isolate the compressed-tensors W8A16 serve path per layer.
|
||||||
|
|
||||||
|
Builds vLLM parallel linear layers exactly as the qwen_image_21 transformer
|
||||||
|
does (fused QKV, row-parallel to_out, img_mlp proj/out), loads the int8
|
||||||
|
artifact's packed tensors through the production weight-loader machinery,
|
||||||
|
runs the Marlin repack (process_weights_after_loading) on the GPU, and
|
||||||
|
compares each layer's forward output against a float32 reference computed
|
||||||
|
from the artifact's own unpacked+dequantized weights.
|
||||||
|
|
||||||
|
Expected rel-L2 when the load->repack->apply chain is correct: ~1-2%
|
||||||
|
(int8 quantization error plus bf16 activation rounding). A large error
|
||||||
|
pins corruption inside the serve path with the checkpoint already
|
||||||
|
verified correct (verify_artifact.py).
|
||||||
|
|
||||||
|
Usage: marlin_isolate.py --artifact ARTIFACT_DIR [--block N]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
|
||||||
|
from safetensors.torch import load_file
|
||||||
|
|
||||||
|
|
||||||
|
def dequantized_weight(art: dict, base: str) -> torch.Tensor:
|
||||||
|
shape = tuple(art[f"{base}_shape"].tolist())
|
||||||
|
q = unpack_from_int32(art[f"{base}_packed"].cpu(), 8, shape)
|
||||||
|
return q.to(torch.float32) * art[f"{base}_scale"].cpu().to(torch.float32)
|
||||||
|
|
||||||
|
|
||||||
|
def load_tensor(layer: torch.nn.Module, param_name: str, tensor: torch.Tensor, shard: str | None) -> None:
|
||||||
|
param = getattr(layer, param_name)
|
||||||
|
if shard is None:
|
||||||
|
param.weight_loader(param, tensor)
|
||||||
|
else:
|
||||||
|
param.weight_loader(param, tensor, shard)
|
||||||
|
|
||||||
|
|
||||||
|
def check(name: str, layer: torch.nn.Module, art: dict, base: str, shards: list[tuple[str, str]], in_features: int, dtype: torch.dtype) -> dict:
|
||||||
|
device = torch.device("cuda")
|
||||||
|
for suffix in ("weight_packed", "weight_scale", "weight_shape"):
|
||||||
|
for ckpt_base, shard in shards:
|
||||||
|
load_tensor(layer, suffix, art[f"{ckpt_base}.{suffix}"].to(device), shard)
|
||||||
|
|
||||||
|
# Bitwise check of the loaded packed weights against the artifact.
|
||||||
|
loaded = layer.weight_packed.detach().cpu()
|
||||||
|
if len(shards) > 1:
|
||||||
|
ref_packed = torch.cat([art[f"{b}.weight_packed"] for b, _ in shards], dim=0)
|
||||||
|
else:
|
||||||
|
ref_packed = art[f"{base}.weight_packed"]
|
||||||
|
packed_ok = bool(torch.equal(loaded, ref_packed))
|
||||||
|
scale_loaded = layer.weight_scale.detach().float()
|
||||||
|
scale_ok = bool(torch.isfinite(scale_loaded).all())
|
||||||
|
|
||||||
|
layer.quant_method.process_weights_after_loading(layer)
|
||||||
|
repacked_scale = getattr(layer, "weight_scale", None)
|
||||||
|
scale_post = (
|
||||||
|
{
|
||||||
|
"finite": bool(torch.isfinite(repacked_scale.detach().float()).all()),
|
||||||
|
"min": float(repacked_scale.detach().float().min()),
|
||||||
|
"max": float(repacked_scale.detach().float().max()),
|
||||||
|
}
|
||||||
|
if repacked_scale is not None
|
||||||
|
else {}
|
||||||
|
)
|
||||||
|
|
||||||
|
torch.manual_seed(2301)
|
||||||
|
ref_w = torch.cat([dequantized_weight(art, f"{b}.weight") for b, _ in shards], dim=0) if len(shards) > 1 else dequantized_weight(art, f"{base}.weight")
|
||||||
|
outs = {}
|
||||||
|
for act_dtype in (torch.bfloat16, torch.float16):
|
||||||
|
x = torch.randn(512, in_features, dtype=act_dtype, device=device)
|
||||||
|
result = layer(x)
|
||||||
|
out = (result[0] if isinstance(result, tuple) else result).float()
|
||||||
|
ref = x.float() @ ref_w.T.to(device)
|
||||||
|
finite = bool(torch.isfinite(out).all())
|
||||||
|
rel = float((out - ref).norm() / ref.norm())
|
||||||
|
outs[str(act_dtype).replace("torch.", "")] = rel if finite else "NaN"
|
||||||
|
|
||||||
|
return {
|
||||||
|
"layer": name,
|
||||||
|
"rel_l2_by_act_dtype": outs,
|
||||||
|
"packed_bitwise_ok": packed_ok,
|
||||||
|
"scale_finite_pre_repack": scale_ok,
|
||||||
|
"scale_post_repack": scale_post,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def synthetic_artifact(block: int) -> dict[str, torch.Tensor]:
|
||||||
|
"""Random int8-quantized tensors in the artifact's layout, for stack-only isolation."""
|
||||||
|
from compressed_tensors.compressors.pack_quantized.helpers import pack_to_int32
|
||||||
|
|
||||||
|
def quant(name: str, out: int, inp: int) -> None:
|
||||||
|
w = torch.randn(out, inp, dtype=torch.float32)
|
||||||
|
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||||
|
art[f"{name}.weight_packed"] = pack_to_int32(q.contiguous(), 8).contiguous()
|
||||||
|
art[f"{name}.weight_scale"] = scale.to(torch.float32).contiguous()
|
||||||
|
art[f"{name}.weight_shape"] = torch.tensor([out, inp], dtype=torch.int64)
|
||||||
|
|
||||||
|
art: dict[str, torch.Tensor] = {}
|
||||||
|
p = f"transformer_blocks.{block}"
|
||||||
|
for part in ("to_q", "to_k", "to_v"):
|
||||||
|
quant(f"{p}.attn.{part}", 4096, 4096)
|
||||||
|
quant(f"{p}.attn.to_out.0", 4096, 4096)
|
||||||
|
quant(f"{p}.img_mlp.proj", 12288, 4096)
|
||||||
|
quant(f"{p}.img_mlp.out", 4096, 12288)
|
||||||
|
return art
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--artifact", type=pathlib.Path, required=True)
|
||||||
|
parser.add_argument("--block", type=int, default=0)
|
||||||
|
parser.add_argument(
|
||||||
|
"--synthetic",
|
||||||
|
action="store_true",
|
||||||
|
help="ignore --artifact; use random int8 weights (isolates the stack from the checkpoint)",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
import vllm.distributed
|
||||||
|
from vllm.config import VllmConfig, set_current_vllm_config
|
||||||
|
from vllm.distributed import (
|
||||||
|
destroy_distributed_environment,
|
||||||
|
destroy_model_parallel,
|
||||||
|
initialize_model_parallel,
|
||||||
|
)
|
||||||
|
from vllm.model_executor.layers.linear import (
|
||||||
|
ColumnParallelLinear,
|
||||||
|
QKVParallelLinear,
|
||||||
|
RowParallelLinear,
|
||||||
|
)
|
||||||
|
|
||||||
|
vllm.distributed.init_distributed_environment(
|
||||||
|
backend="gloo",
|
||||||
|
world_size=1,
|
||||||
|
rank=0,
|
||||||
|
local_rank=0,
|
||||||
|
distributed_init_method="tcp://127.0.0.1:29601",
|
||||||
|
)
|
||||||
|
with set_current_vllm_config(VllmConfig()):
|
||||||
|
initialize_model_parallel(tensor_model_parallel_size=1)
|
||||||
|
|
||||||
|
config = json.loads((args.artifact / "transformer" / "config.json").read_text())
|
||||||
|
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||||
|
|
||||||
|
quant_config = resolve_component_quant_config(
|
||||||
|
build_quant_config(config["quantization_config"]), "transformer"
|
||||||
|
)
|
||||||
|
if args.synthetic:
|
||||||
|
art = synthetic_artifact(args.block)
|
||||||
|
else:
|
||||||
|
art = load_file(str(args.artifact / "transformer" / "diffusion_pytorch_model.safetensors"))
|
||||||
|
b = args.block
|
||||||
|
p = f"transformer.transformer_blocks.{b}"
|
||||||
|
|
||||||
|
results = []
|
||||||
|
with torch.device("cuda"):
|
||||||
|
to_qkv = QKVParallelLinear(
|
||||||
|
hidden_size=4096, head_size=128, total_num_heads=32, bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.attn.to_qkv",
|
||||||
|
)
|
||||||
|
results.append(
|
||||||
|
check(
|
||||||
|
"attn.to_qkv (fused q/k/v)",
|
||||||
|
to_qkv,
|
||||||
|
art,
|
||||||
|
f"transformer_blocks.{b}.attn.to_qkv",
|
||||||
|
[
|
||||||
|
(f"transformer_blocks.{b}.attn.to_q", "q"),
|
||||||
|
(f"transformer_blocks.{b}.attn.to_k", "k"),
|
||||||
|
(f"transformer_blocks.{b}.attn.to_v", "v"),
|
||||||
|
],
|
||||||
|
4096,
|
||||||
|
torch.bfloat16,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
to_out = RowParallelLinear(
|
||||||
|
4096, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.attn.to_out.0",
|
||||||
|
)
|
||||||
|
results.append(
|
||||||
|
check(
|
||||||
|
"attn.to_out.0 (row)",
|
||||||
|
to_out,
|
||||||
|
art,
|
||||||
|
f"transformer_blocks.{b}.attn.to_out.0",
|
||||||
|
[(f"transformer_blocks.{b}.attn.to_out.0", None)],
|
||||||
|
4096,
|
||||||
|
torch.bfloat16,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
proj = ColumnParallelLinear(
|
||||||
|
4096, 12288, bias=False, gather_output=False, return_bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.img_mlp.proj",
|
||||||
|
)
|
||||||
|
results.append(
|
||||||
|
check(
|
||||||
|
"img_mlp.proj (column)",
|
||||||
|
proj,
|
||||||
|
art,
|
||||||
|
f"transformer_blocks.{b}.img_mlp.proj",
|
||||||
|
[(f"transformer_blocks.{b}.img_mlp.proj", None)],
|
||||||
|
4096,
|
||||||
|
torch.bfloat16,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
mlp_out = RowParallelLinear(
|
||||||
|
12288, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.img_mlp.out",
|
||||||
|
)
|
||||||
|
results.append(
|
||||||
|
check(
|
||||||
|
"img_mlp.out (row, wide-in)",
|
||||||
|
mlp_out,
|
||||||
|
art,
|
||||||
|
f"transformer_blocks.{b}.img_mlp.out",
|
||||||
|
[(f"transformer_blocks.{b}.img_mlp.out", None)],
|
||||||
|
12288,
|
||||||
|
torch.bfloat16,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
destroy_model_parallel()
|
||||||
|
destroy_distributed_environment()
|
||||||
|
print(json.dumps(results, indent=1))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 probe client: reference-conditioned requests against the int8
|
||||||
|
weight-only compressed-tensors artifact, with the structure gate applied to
|
||||||
|
every response.
|
||||||
|
|
||||||
|
Matrix pass (the issue's capacity matrix): one-, two-, three-, and
|
||||||
|
four-reference fixtures at seeds 2301-2304 — wall time and per-image
|
||||||
|
structure metrics land in the record for every request. Every request
|
||||||
|
mirrors the sealed Diffusers baseline attempt 2026-10-07T170729Z-q3-4r
|
||||||
|
(same frozen references, same rendered prompt, 20 steps, true_cfg_scale
|
||||||
|
1.0, fixture resolution, no negative prompt), submitted one at a time
|
||||||
|
through POST /v1/images/edits.
|
||||||
|
|
||||||
|
The structure gate (the lesson from issue 1's FP8 misread): a run only
|
||||||
|
counts when the output is structurally sound — fully opaque RGBA and
|
||||||
|
non-flat RGB. Broken FP8 output measured ~96.5-97% transparent with
|
||||||
|
near-flat grey RGB; the BF16 path is 100% opaque. The gate numbers are
|
||||||
|
recorded on every request regardless; `gate.py` summarizes them.
|
||||||
|
|
||||||
|
Usage: probe.py --out RUN_DIR [--matrix] [--host H:P]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import time
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import requests
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
W = pathlib.Path.home() / "Projects/vllm-omni-val/issue5"
|
||||||
|
SUITE = (
|
||||||
|
pathlib.Path.home()
|
||||||
|
/ ".local/share/multi-model-helper-server/data/evaluation/media/image/references-suite-2"
|
||||||
|
)
|
||||||
|
STEPS = 20
|
||||||
|
CFG = 1.0
|
||||||
|
MATRIX_FIXTURES = (
|
||||||
|
"edit-expression-ysolde", # 1 reference
|
||||||
|
"edit-compose-pair", # 2 references
|
||||||
|
"edit-compose-three", # 3 references (derived; see fixtures.json)
|
||||||
|
"edit-compose-four", # 4 references
|
||||||
|
)
|
||||||
|
MATRIX_SEEDS = (2301, 2302, 2303, 2304)
|
||||||
|
|
||||||
|
|
||||||
|
def structure_metrics(png: bytes) -> dict:
|
||||||
|
"""Opacity + non-flatness numbers for the structure gate."""
|
||||||
|
image = Image.open(io.BytesIO(png))
|
||||||
|
array = np.asarray(image.convert("RGBA"))
|
||||||
|
alpha = array[..., 3]
|
||||||
|
rgb = array[..., :3].astype(np.float64)
|
||||||
|
return {
|
||||||
|
"opaque_fraction": round(float((alpha == 255).mean()), 4),
|
||||||
|
"alpha_mean": round(float(alpha.mean()), 1),
|
||||||
|
"rgb_std": [round(float(rgb[..., i].std()), 1) for i in range(3)],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def submit(host: str, fixture: dict, seed: int) -> tuple[dict, bytes | None]:
|
||||||
|
files = []
|
||||||
|
handles = []
|
||||||
|
for ref in fixture["references"]:
|
||||||
|
path = SUITE / f"{ref}.png"
|
||||||
|
handle = open(path, "rb")
|
||||||
|
handles.append(handle)
|
||||||
|
files.append(("image[]", (path.name, handle, "image/png")))
|
||||||
|
data = {
|
||||||
|
"model": "Qwen/Qwen-Image-2.1",
|
||||||
|
"prompt": fixture["prompt"],
|
||||||
|
"seed": str(seed),
|
||||||
|
"num_inference_steps": str(STEPS),
|
||||||
|
"true_cfg_scale": str(CFG),
|
||||||
|
"size": f"{fixture['width']}x{fixture['height']}",
|
||||||
|
"n": "1",
|
||||||
|
"response_format": "b64_json",
|
||||||
|
"output_format": "png",
|
||||||
|
"return_stage_metrics": "true",
|
||||||
|
}
|
||||||
|
rec: dict = {
|
||||||
|
"fixture_id": None,
|
||||||
|
"seed": seed,
|
||||||
|
"n_references": len(fixture["references"]),
|
||||||
|
"steps": STEPS,
|
||||||
|
"true_cfg_scale": CFG,
|
||||||
|
"size": f"{fixture['width']}x{fixture['height']}",
|
||||||
|
}
|
||||||
|
t0 = time.time()
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
f"http://{host}/v1/images/edits", data=data, files=files, timeout=1800
|
||||||
|
)
|
||||||
|
rec["wall_s"] = round(time.time() - t0, 2)
|
||||||
|
rec["http"] = response.status_code
|
||||||
|
if response.status_code == 200:
|
||||||
|
body = response.json()
|
||||||
|
png = base64.b64decode(body["data"][0]["b64_json"])
|
||||||
|
rec["bytes"] = len(png)
|
||||||
|
rec["sha256"] = hashlib.sha256(png).hexdigest()
|
||||||
|
rec["structure"] = structure_metrics(png)
|
||||||
|
rec["usage"] = body.get("usage")
|
||||||
|
for key in ("stage_metrics", "metrics", "timings"):
|
||||||
|
if body.get(key):
|
||||||
|
rec[key] = body[key]
|
||||||
|
return rec, png
|
||||||
|
rec["error"] = response.text[:2000]
|
||||||
|
return rec, None
|
||||||
|
except Exception as exc: # noqa: BLE001 - recorded as the probe result
|
||||||
|
rec["wall_s"] = round(time.time() - t0, 2)
|
||||||
|
rec["error"] = f"{type(exc).__name__}: {exc}"
|
||||||
|
return rec, None
|
||||||
|
finally:
|
||||||
|
for handle in handles:
|
||||||
|
handle.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--out", required=True, type=pathlib.Path)
|
||||||
|
parser.add_argument("--host", default="127.0.0.1:8091")
|
||||||
|
parser.add_argument("--matrix", action="store_true")
|
||||||
|
parser.add_argument(
|
||||||
|
"--fixture",
|
||||||
|
default="edit-compose-four",
|
||||||
|
help="single fixture id for --single",
|
||||||
|
)
|
||||||
|
parser.add_argument("--single", action="store_true", help="one fixture, all seeds")
|
||||||
|
parser.add_argument(
|
||||||
|
"--seeds",
|
||||||
|
type=lambda s: tuple(int(x) for x in s.split(",")),
|
||||||
|
default=MATRIX_SEEDS,
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
fixtures = json.loads((W / "fixtures.json").read_text())
|
||||||
|
args.out.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
plan: list[tuple[str, pathlib.Path, int]] = []
|
||||||
|
if args.matrix:
|
||||||
|
for fixture_id in MATRIX_FIXTURES:
|
||||||
|
for seed in args.seeds:
|
||||||
|
plan.append(
|
||||||
|
(
|
||||||
|
fixture_id,
|
||||||
|
args.out / "matrix" / f"{fixture_id}.s{seed}.png",
|
||||||
|
seed,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if args.single:
|
||||||
|
for seed in args.seeds:
|
||||||
|
plan.append(
|
||||||
|
(args.fixture, args.out / "matrix" / f"{args.fixture}.s{seed}.png", seed)
|
||||||
|
)
|
||||||
|
|
||||||
|
records = []
|
||||||
|
for fixture_id, out_path, seed in plan:
|
||||||
|
rec, png = submit(args.host, fixtures[fixture_id], seed)
|
||||||
|
rec["fixture_id"] = fixture_id
|
||||||
|
records.append(rec)
|
||||||
|
print(json.dumps(rec), flush=True)
|
||||||
|
if png is not None:
|
||||||
|
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
out_path.write_bytes(png)
|
||||||
|
elif rec.get("http") == 200:
|
||||||
|
print(f"warning: 200 but no image for {fixture_id} seed {seed}", flush=True)
|
||||||
|
|
||||||
|
suffix = "matrix" if args.matrix else f"single-{args.fixture}"
|
||||||
|
(args.out / f"{suffix}.json").write_text(json.dumps(records, indent=1) + "\n")
|
||||||
|
ok = all(r.get("http") == 200 and "sha256" in r for r in records)
|
||||||
|
print(("PASS" if ok else "FAIL") + f" {len(records)} requests", flush=True)
|
||||||
|
return 0 if ok else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 probe: PSNR of the int8 artifact's vLLM outputs against the
|
||||||
|
sealed Diffusers baseline attempt 2026-10-07T170729Z-q3-4r.
|
||||||
|
|
||||||
|
Usage: psnr.py --fixture FIXTURE_ID RUN_DIR [RUN_DIR ...]
|
||||||
|
|
||||||
|
Compares each RUN_DIR/matrix/FIXTURE_ID.sSEED.png against the baseline
|
||||||
|
sample with the same fixture and seed. RGBA outputs are composited over
|
||||||
|
white, matching issue 1's comparator. The sealed baseline covers the 1-,
|
||||||
|
2-, and 4-reference fixtures; the derived 3-reference fixture has no
|
||||||
|
sealed row (gate.py + visual check cover it).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import pathlib
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
ATTEMPT = (
|
||||||
|
pathlib.Path.home()
|
||||||
|
/ ".local/share/multi-model-helper-server/data/evaluation/media/image/attempts/2026-10-07T170729Z-q3-4r"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def to_rgb(array: np.ndarray) -> np.ndarray:
|
||||||
|
if array.shape[2] == 4:
|
||||||
|
alpha = array[..., 3:4].astype(np.float64) / 255.0
|
||||||
|
bg = np.full_like(array[..., :3], 255, dtype=np.float64)
|
||||||
|
return np.round(
|
||||||
|
array[..., :3].astype(np.float64) * alpha + bg * (1 - alpha)
|
||||||
|
).clip(0, 255).astype(np.uint8)
|
||||||
|
return array[..., :3]
|
||||||
|
|
||||||
|
|
||||||
|
def psnr(a: np.ndarray, b: np.ndarray) -> float:
|
||||||
|
mse = float(np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2))
|
||||||
|
return float("inf") if mse == 0 else 10 * math.log10(255.0**2 / mse)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--fixture", default="edit-compose-four")
|
||||||
|
parser.add_argument("runs", nargs="+", type=pathlib.Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
fixture = args.fixture
|
||||||
|
|
||||||
|
samples = json.loads((ATTEMPT / "samples.json").read_text())["samples"]
|
||||||
|
baseline = {
|
||||||
|
s["seed"]: ATTEMPT / s["file"]
|
||||||
|
for s in samples
|
||||||
|
if s["fixture_id"] == fixture and s["status"] == "succeeded"
|
||||||
|
}
|
||||||
|
summary = {}
|
||||||
|
for run_dir in args.runs:
|
||||||
|
rows = []
|
||||||
|
for seed, base_path in sorted(baseline.items()):
|
||||||
|
ours = run_dir / "matrix" / f"{fixture}.s{seed}.png"
|
||||||
|
if not ours.exists():
|
||||||
|
rows.append({"seed": seed, "error": "missing"})
|
||||||
|
continue
|
||||||
|
bm, om = Image.open(base_path), Image.open(ours)
|
||||||
|
ba, oa = np.asarray(bm), np.asarray(om)
|
||||||
|
if ba.shape[:2] != oa.shape[:2]:
|
||||||
|
rows.append(
|
||||||
|
{"seed": seed, "error": f"size {oa.shape[:2]} vs {ba.shape[:2]}"}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
use_rgba = bm.mode == "RGBA" and om.mode == "RGBA"
|
||||||
|
a, b = (ba, oa) if use_rgba else (to_rgb(ba), to_rgb(oa))
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"seed": seed,
|
||||||
|
"psnr_db": round(psnr(a, b), 2),
|
||||||
|
"modes": f"{bm.mode}/{om.mode}",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
vals = [r["psnr_db"] for r in rows if "psnr_db" in r]
|
||||||
|
if vals:
|
||||||
|
summary[run_dir.name] = {
|
||||||
|
"rows": rows,
|
||||||
|
"mean_db": round(sum(vals) / len(vals), 2),
|
||||||
|
"min_db": min(vals),
|
||||||
|
"max_db": max(vals),
|
||||||
|
"n": len(vals),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
summary[run_dir.name] = {"rows": rows}
|
||||||
|
print(json.dumps(summary, indent=1))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,252 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5: quantize the Qwen-Image-2.1 DiT to int8 weight-only (W8A16) and
|
||||||
|
save it as a compressed-tensors pack-quantized checkpoint.
|
||||||
|
|
||||||
|
Quantizes exactly the linears the vllm-omni transformer constructs with
|
||||||
|
``quant_config`` — per-block attention (to_q/to_k/to_v/to_out.0) and
|
||||||
|
img_mlp (proj/gate_layer/out) — and copies every boundary layer (img_in,
|
||||||
|
txt_in, time_text_embed, modulation, norm_out, proj_out) plus the RMSNorm
|
||||||
|
q/k scales through in BF16. Per-output-channel symmetric int8, scales in
|
||||||
|
fp32, weights packed 4x uint8 (biased by 128) per int32 via
|
||||||
|
compressed-tensors' own ``pack_to_int32`` so the artifact matches the
|
||||||
|
llm-compressor storage convention bit for bit.
|
||||||
|
|
||||||
|
The artifact is a full diffusers-layout model directory: the quantized
|
||||||
|
``transformer/`` is real, the remaining components are symlinks into the
|
||||||
|
sealed snapshot. ``transformer/config.json`` carries the quantization
|
||||||
|
metadata wrapped per-component (``{"transformer": {...}}``) so vLLM-Omni's
|
||||||
|
auto-detection quantizes the DiT only — the text encoder and VAE stay BF16.
|
||||||
|
|
||||||
|
A MANIFEST.json (inputs, versions, per-file sha256, quantization stats) is
|
||||||
|
written beside the weights. Nothing here enters Git; the artifact lives
|
||||||
|
under the helper-system state root.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
quantize_dit_int8wo.py --snapshot SNAPSHOT_DIR --out ARTIFACT_DIR
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import datetime as dt
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import platform
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from compressed_tensors.compressors.pack_quantized.helpers import pack_to_int32
|
||||||
|
from compressed_tensors.config import CompressionFormat
|
||||||
|
from compressed_tensors.quantization import (
|
||||||
|
QuantizationArgs,
|
||||||
|
QuantizationConfig,
|
||||||
|
QuantizationScheme,
|
||||||
|
QuantizationStatus,
|
||||||
|
QuantizationStrategy,
|
||||||
|
QuantizationType,
|
||||||
|
)
|
||||||
|
from safetensors.torch import load_file, save_file
|
||||||
|
|
||||||
|
SNAPSHOT_COMMIT = "790c92633540aa0cb11d9abf19eb46d861714758"
|
||||||
|
# Boundary layers the vllm-omni model pins to quant_config=None (BF16), plus
|
||||||
|
# the non-linear tensors; everything else in a block is quantized.
|
||||||
|
IGNORED_TARGETS = ["img_in", "txt_in", "time_text_embed", "modulation", "norm_out", "proj_out"]
|
||||||
|
QUANTIZED_KEY = re.compile(
|
||||||
|
r"^transformer_blocks\.\d+\."
|
||||||
|
r"(attn\.(to_q|to_k|to_v|to_out\.0)|img_mlp\.(proj|gate_layer|out))\.weight$"
|
||||||
|
)
|
||||||
|
SYMLINKED_COMPONENTS = ("scheduler", "text_encoder", "vae", "processor")
|
||||||
|
SYMLINKED_FILES = ("model_index.json",)
|
||||||
|
|
||||||
|
|
||||||
|
def build_quantization_dict(scheme: str) -> dict:
|
||||||
|
"""Serialized compressed-tensors metadata (llm-compressor schema).
|
||||||
|
|
||||||
|
scheme="int8wo": int8 weight-only, pack-quantized storage.
|
||||||
|
scheme="w8a8dyn": int8 weights (static per-channel) + dynamic per-token
|
||||||
|
symmetric int8 activations, int-quantized storage — the non-Marlin
|
||||||
|
cutlass path on sm_86.
|
||||||
|
"""
|
||||||
|
weights = QuantizationArgs(
|
||||||
|
num_bits=8,
|
||||||
|
type=QuantizationType.INT,
|
||||||
|
symmetric=True,
|
||||||
|
strategy=QuantizationStrategy.CHANNEL,
|
||||||
|
dynamic=False,
|
||||||
|
observer="minmax",
|
||||||
|
)
|
||||||
|
input_activations = (
|
||||||
|
QuantizationArgs(
|
||||||
|
num_bits=8,
|
||||||
|
type=QuantizationType.INT,
|
||||||
|
symmetric=True,
|
||||||
|
strategy=QuantizationStrategy.TOKEN,
|
||||||
|
dynamic=True,
|
||||||
|
)
|
||||||
|
if scheme == "w8a8dyn"
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
scheme_obj = QuantizationScheme(targets=["Linear"], weights=weights, input_activations=input_activations)
|
||||||
|
config = QuantizationConfig(
|
||||||
|
config_groups={"group_0": scheme_obj},
|
||||||
|
format=CompressionFormat.int_quantized if scheme == "w8a8dyn" else CompressionFormat.pack_quantized,
|
||||||
|
ignore=list(IGNORED_TARGETS),
|
||||||
|
quant_method="compressed-tensors",
|
||||||
|
quantization_status=QuantizationStatus.FROZEN,
|
||||||
|
)
|
||||||
|
return config.to_dict()
|
||||||
|
|
||||||
|
|
||||||
|
def sha256_file(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def quantize_weight(weight: torch.Tensor, scheme: str) -> tuple[dict[str, torch.Tensor], dict]:
|
||||||
|
"""Per-output-channel symmetric int8.
|
||||||
|
|
||||||
|
scheme="int8wo": q = round(W/s), s = rowmax(|W|)/127, packed via
|
||||||
|
compressed-tensors' packer (uint8 biased by 128, 4 per int32).
|
||||||
|
scheme="w8a8dyn": same quantization, stored unpacked as plain int8
|
||||||
|
(int-quantized format) with fp32 scales.
|
||||||
|
"""
|
||||||
|
w = weight.detach().to(torch.float32)
|
||||||
|
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||||
|
dequant = q.to(torch.float32) * scale
|
||||||
|
err = (w - dequant).norm() / w.norm().clamp_min(1e-12)
|
||||||
|
stats = {"rel_l2": float(err), "out": int(w.shape[0]), "in": int(w.shape[1])}
|
||||||
|
if scheme == "w8a8dyn":
|
||||||
|
return {"weight": q.contiguous(), "weight_scale": scale.to(torch.float32).contiguous()}, stats
|
||||||
|
return {
|
||||||
|
"weight_packed": pack_to_int32(q.contiguous(), 8).contiguous(),
|
||||||
|
"weight_scale": scale.to(torch.float32).contiguous(),
|
||||||
|
"weight_shape": torch.tensor(weight.shape, dtype=torch.int64),
|
||||||
|
}, stats
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--snapshot", type=Path, required=True, help="sealed snapshot dir")
|
||||||
|
parser.add_argument("--out", type=Path, required=True, help="artifact dir to create")
|
||||||
|
parser.add_argument(
|
||||||
|
"--scheme",
|
||||||
|
choices=("int8wo", "w8a8dyn"),
|
||||||
|
default="int8wo",
|
||||||
|
help="int8 weight-only (pack-quantized, Marlin path) or W8A8 int8-dynamic (int-quantized, cutlass path)",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
snapshot = args.snapshot.resolve()
|
||||||
|
if not (snapshot / "transformer" / "config.json").exists():
|
||||||
|
sys.exit(f"snapshot transformer config not found under {snapshot}")
|
||||||
|
if args.out.exists():
|
||||||
|
sys.exit(f"refusing to overwrite existing {args.out}")
|
||||||
|
artifact = args.out
|
||||||
|
(artifact / "transformer").mkdir(parents=True)
|
||||||
|
|
||||||
|
index = json.loads(
|
||||||
|
(snapshot / "transformer" / "diffusion_pytorch_model.safetensors.index.json").read_text()
|
||||||
|
)
|
||||||
|
shards = sorted(set(index["weight_map"].values()))
|
||||||
|
state: dict[str, torch.Tensor] = {}
|
||||||
|
quant_stats: dict[str, dict] = {}
|
||||||
|
quant_params = bf16_params = 0
|
||||||
|
for shard in shards:
|
||||||
|
tensors = load_file(str(snapshot / "transformer" / shard))
|
||||||
|
for name, tensor in tensors.items():
|
||||||
|
if QUANTIZED_KEY.match(name):
|
||||||
|
emitted, stats = quantize_weight(tensor, args.scheme)
|
||||||
|
if args.scheme == "w8a8dyn":
|
||||||
|
# int-quantized storage: plain int8 weight + scale, mirroring
|
||||||
|
# the vLLM layer parameter names (weight / weight_scale).
|
||||||
|
state[name] = emitted["weight"]
|
||||||
|
state[f"{name}_scale"] = emitted["weight_scale"]
|
||||||
|
else:
|
||||||
|
state[f"{name}_packed"] = emitted["weight_packed"]
|
||||||
|
state[f"{name}_scale"] = emitted["weight_scale"]
|
||||||
|
state[f"{name}_shape"] = emitted["weight_shape"]
|
||||||
|
quant_stats[name] = stats
|
||||||
|
quant_params += tensor.numel()
|
||||||
|
print(f"q {name} {tuple(tensor.shape)} rel_l2={stats['rel_l2']:.5f}", flush=True)
|
||||||
|
else:
|
||||||
|
state[name] = tensor
|
||||||
|
bf16_params += tensor.numel()
|
||||||
|
del tensors
|
||||||
|
|
||||||
|
weights_path = artifact / "transformer" / "diffusion_pytorch_model.safetensors"
|
||||||
|
save_file(state, str(weights_path), metadata={"format": "pt"})
|
||||||
|
total_bytes = weights_path.stat().st_size
|
||||||
|
|
||||||
|
config = json.loads((snapshot / "transformer" / "config.json").read_text())
|
||||||
|
config["quantization_config"] = {"transformer": build_quantization_dict(args.scheme)}
|
||||||
|
(artifact / "transformer" / "config.json").write_text(json.dumps(config, indent=2) + "\n")
|
||||||
|
|
||||||
|
for name in SYMLINKED_FILES:
|
||||||
|
target = snapshot / name
|
||||||
|
(artifact / name).symlink_to(target)
|
||||||
|
for name in SYMLINKED_COMPONENTS:
|
||||||
|
target = snapshot / name
|
||||||
|
(artifact / name).symlink_to(target, target_is_directory=True)
|
||||||
|
|
||||||
|
files = [
|
||||||
|
{"path": "transformer/config.json", "bytes": (artifact / "transformer" / "config.json").stat().st_size},
|
||||||
|
{"path": "transformer/diffusion_pytorch_model.safetensors", "bytes": total_bytes},
|
||||||
|
]
|
||||||
|
for entry in files:
|
||||||
|
entry["sha256"] = sha256_file(artifact / entry["path"])
|
||||||
|
tree_json = snapshot.parent.parent / "trees" / f"{SNAPSHOT_COMMIT}.json"
|
||||||
|
manifest = {
|
||||||
|
"schema": f"issue5-{args.scheme}-v1",
|
||||||
|
"created_utc": dt.datetime.now(dt.UTC).replace(microsecond=0).isoformat(),
|
||||||
|
"description": (
|
||||||
|
f"Qwen-Image-2.1 DiT {args.scheme} "
|
||||||
|
+ (
|
||||||
|
"W8A8 int8-dynamic, int-quantized"
|
||||||
|
if args.scheme == "w8a8dyn"
|
||||||
|
else "int8 weight-only W8A16, pack-quantized"
|
||||||
|
)
|
||||||
|
+ ", compressed-tensors"
|
||||||
|
),
|
||||||
|
"inputs": {
|
||||||
|
"snapshot": str(snapshot),
|
||||||
|
"revision": SNAPSHOT_COMMIT,
|
||||||
|
"tree_manifest": str(tree_json),
|
||||||
|
"tree_sha256": sha256_file(tree_json) if tree_json.exists() else None,
|
||||||
|
},
|
||||||
|
"versions": {
|
||||||
|
"python": platform.python_version(),
|
||||||
|
"torch": torch.__version__,
|
||||||
|
"compressed_tensors": __import__("compressed_tensors").__version__,
|
||||||
|
},
|
||||||
|
"quantization": {
|
||||||
|
"method": "compressed-tensors",
|
||||||
|
"format": "pack-quantized",
|
||||||
|
"weights": "int8 symmetric per-output-channel, static, minmax",
|
||||||
|
"config": build_quantization_dict(args.scheme),
|
||||||
|
"ignored_layers": list(IGNORED_TARGETS),
|
||||||
|
"note": "ignore list mirrors the vllm-omni model's quant_config=None boundary layers",
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"quantized_tensors": len(quant_stats),
|
||||||
|
"quantized_params": quant_params,
|
||||||
|
"bf16_params": bf16_params,
|
||||||
|
"mean_rel_l2": sum(s["rel_l2"] for s in quant_stats.values()) / max(len(quant_stats), 1),
|
||||||
|
"max_rel_l2": max((s["rel_l2"] for s in quant_stats.values()), default=0.0),
|
||||||
|
"weights_file_bytes": total_bytes,
|
||||||
|
},
|
||||||
|
"files": files,
|
||||||
|
}
|
||||||
|
(artifact / "MANIFEST.json").write_text(json.dumps(manifest, indent=1) + "\n")
|
||||||
|
print(json.dumps(manifest["stats"], indent=1))
|
||||||
|
print(f"artifact: {artifact}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Issue 5 probe: serve the int8 weight-only compressed-tensors artifact on
|
||||||
|
# the RTX 3090 with one labeled flag set. VRAM is polled once per second for
|
||||||
|
# the whole server lifetime, matching issue 2's instrument (nvidia-smi csv
|
||||||
|
# series). The default flag set is cpu-offload + eager; pass extras after
|
||||||
|
# the label to override/extend (they are appended, vLLM last-one-wins).
|
||||||
|
#
|
||||||
|
# Usage: serve.sh LABEL [extra vllm flags...]
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
BASE=$HOME/Projects/vllm-omni-val
|
||||||
|
W=$BASE/issue5
|
||||||
|
LABEL=$1; shift
|
||||||
|
RUN=$W/runs/$LABEL
|
||||||
|
mkdir -p "$RUN"
|
||||||
|
|
||||||
|
STATE=$HOME/.local/state/multi-model-helper-server/state/image-service
|
||||||
|
ARTIFACT=${ARTIFACT:-$STATE/artifacts/qwen-image-2.1-dit-int8wo}
|
||||||
|
export HF_HOME=$STATE/weights
|
||||||
|
export HF_HUB_OFFLINE=1
|
||||||
|
|
||||||
|
( while sleep 1; do nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv,noheader; done ) > "$RUN/vram.log" 2>&1 &
|
||||||
|
POLL=$!
|
||||||
|
T0=$(date +%s)
|
||||||
|
echo "model: $ARTIFACT" > "$RUN/flags.txt"
|
||||||
|
echo "flags: $*" >> "$RUN/flags.txt"
|
||||||
|
|
||||||
|
$BASE/venv/bin/vllm serve "$ARTIFACT" --omni --port 8091 \
|
||||||
|
--served-model-name Qwen/Qwen-Image-2.1 \
|
||||||
|
--enforce-eager --enable-cpu-offload "$@" > "$RUN/serve.log" 2>&1
|
||||||
|
RC=$?
|
||||||
|
kill $POLL 2>/dev/null
|
||||||
|
echo "serve_rc=$RC wall=$(( $(date +%s) - T0 ))s" >> "$RUN/serve.log"
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5: verify the int8 weight-only artifact round-trips to the BF16 original.
|
||||||
|
|
||||||
|
Unpacks ``weight_packed`` with compressed-tensors' own unpacker, dequantizes
|
||||||
|
with the stored per-channel scales, and compares against the sealed snapshot's
|
||||||
|
BF16 tensors. Expected rel-L2 ~1% (the quantizer's measured error); a large
|
||||||
|
error means the artifact itself is corrupt, a small error means any serve-path
|
||||||
|
collapse lives downstream of the checkpoint (loading/repack/kernel).
|
||||||
|
|
||||||
|
Usage: verify_artifact.py --artifact ARTIFACT_DIR --snapshot SNAPSHOT_DIR [--tensors N]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
|
||||||
|
import torch
|
||||||
|
from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32
|
||||||
|
from safetensors.torch import load_file
|
||||||
|
|
||||||
|
QUANTIZED_KEY = "weight_packed" # checkpoint suffix; stripping it yields the base ".weight" name
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--artifact", type=pathlib.Path, required=True)
|
||||||
|
parser.add_argument("--snapshot", type=pathlib.Path, required=True)
|
||||||
|
parser.add_argument("--tensors", type=int, default=6, help="number of quantized tensors to check")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
art = load_file(str(args.artifact / "transformer" / "diffusion_pytorch_model.safetensors"))
|
||||||
|
index = json.loads(
|
||||||
|
(args.snapshot / "transformer" / "diffusion_pytorch_model.safetensors.index.json").read_text()
|
||||||
|
)["weight_map"]
|
||||||
|
shard_cache: dict[str, dict] = {}
|
||||||
|
|
||||||
|
checked = 0
|
||||||
|
for name in sorted(k.removesuffix("_packed") for k in art if k.endswith(f".{QUANTIZED_KEY}")):
|
||||||
|
if checked >= args.tensors:
|
||||||
|
break
|
||||||
|
base_name = name
|
||||||
|
shard = index[base_name]
|
||||||
|
if shard not in shard_cache:
|
||||||
|
shard_cache.clear() # keep at most one ~7 GB shard resident
|
||||||
|
shard_cache[shard] = load_file(str(args.snapshot / "transformer" / shard))
|
||||||
|
ref = shard_cache[shard][base_name].to(torch.float32)
|
||||||
|
shape = tuple(art[f"{base_name}_shape"].tolist())
|
||||||
|
q = unpack_from_int32(art[f"{base_name}_packed"], 8, shape)
|
||||||
|
dequant = q.to(torch.float32) * art[f"{base_name}_scale"]
|
||||||
|
rel = float((dequant - ref).norm() / ref.norm())
|
||||||
|
print(
|
||||||
|
f"{base_name}: shape {shape} rel_l2={rel:.5f} "
|
||||||
|
f"scale {tuple(art[f'{base_name}_scale'].shape)}/{art[f'{base_name}_scale'].dtype}"
|
||||||
|
)
|
||||||
|
checked += 1
|
||||||
|
print(f"checked {checked} tensors")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,174 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Issue 5 round two: isolate the compressed-tensors W8A8-int8-dynamic path.
|
||||||
|
|
||||||
|
Mirrors marlin_isolate.py but for the int-quantized W8A8 dynamic scheme
|
||||||
|
(int8 weights, static per-output-channel scales, dynamic symmetric per-token
|
||||||
|
int8 activations) whose kernel on sm_86 is the cutlass int8 scaled-MM —
|
||||||
|
no Marlin in the path. Loads synthetic int8 tensors through the production
|
||||||
|
weight-loader machinery and compares layer(x) against a float32 reference
|
||||||
|
computed from the same dequantized weights with per-token dynamic int8
|
||||||
|
activation quantization applied to the reference input.
|
||||||
|
|
||||||
|
Expected rel-L2 when the chain is correct: a few percent (int8 weight error
|
||||||
|
~1% plus int8 per-token activation rounding).
|
||||||
|
|
||||||
|
Usage: w8a8_isolate.py [--block N]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
|
||||||
|
import torch
|
||||||
|
|
||||||
|
|
||||||
|
def synth(out: int, inp: int, ones: bool = False) -> tuple[torch.Tensor, torch.Tensor]:
|
||||||
|
"""Per-channel symmetric int8 weight + scale, mirroring the artifact builder.
|
||||||
|
|
||||||
|
ones=True emits an analytically known weight (q=127, scale=1/127 -> W=1):
|
||||||
|
with an all-ones input row the layer output must be exactly in_features.
|
||||||
|
"""
|
||||||
|
if ones:
|
||||||
|
q = torch.full((out, inp), 127, dtype=torch.int8)
|
||||||
|
return q, torch.full((out, 1), 1.0 / 127.0, dtype=torch.float32)
|
||||||
|
w = torch.randn(out, inp, dtype=torch.float32)
|
||||||
|
scale = w.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
q = torch.round(w / scale).clamp(-127, 127).to(torch.int8)
|
||||||
|
return q, scale.to(torch.float32)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--block", type=int, default=0)
|
||||||
|
parser.add_argument("--ones", action="store_true",
|
||||||
|
help="analytic weights (W=1) and ones input; output must equal in_features")
|
||||||
|
parser.add_argument("--act-dtype", choices=("bfloat16", "float16"), default="bfloat16")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
import vllm.distributed
|
||||||
|
from vllm.config import VllmConfig, set_current_vllm_config
|
||||||
|
from vllm.distributed import (
|
||||||
|
destroy_distributed_environment,
|
||||||
|
destroy_model_parallel,
|
||||||
|
initialize_model_parallel,
|
||||||
|
)
|
||||||
|
from vllm.model_executor.layers.linear import QKVParallelLinear, RowParallelLinear
|
||||||
|
from compressed_tensors.quantization import (
|
||||||
|
QuantizationArgs,
|
||||||
|
QuantizationConfig,
|
||||||
|
QuantizationScheme,
|
||||||
|
QuantizationStatus,
|
||||||
|
QuantizationStrategy,
|
||||||
|
QuantizationType,
|
||||||
|
)
|
||||||
|
from compressed_tensors.config import CompressionFormat
|
||||||
|
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||||
|
|
||||||
|
weights = QuantizationArgs(num_bits=8, type=QuantizationType.INT, symmetric=True,
|
||||||
|
strategy=QuantizationStrategy.CHANNEL, dynamic=False, observer="minmax")
|
||||||
|
acts = QuantizationArgs(num_bits=8, type=QuantizationType.INT, symmetric=True,
|
||||||
|
strategy=QuantizationStrategy.TOKEN, dynamic=True)
|
||||||
|
scheme = QuantizationScheme(targets=["Linear"], weights=weights, input_activations=acts)
|
||||||
|
disk = QuantizationConfig(
|
||||||
|
config_groups={"group_0": scheme}, format=CompressionFormat.int_quantized,
|
||||||
|
ignore=["img_in", "txt_in", "time_text_embed", "modulation", "norm_out", "proj_out"],
|
||||||
|
quant_method="compressed-tensors", quantization_status=QuantizationStatus.FROZEN,
|
||||||
|
).to_dict()
|
||||||
|
quant_config = resolve_component_quant_config(
|
||||||
|
build_quant_config({"transformer": disk}), "transformer"
|
||||||
|
)
|
||||||
|
|
||||||
|
vllm.distributed.init_distributed_environment(
|
||||||
|
backend="gloo", world_size=1, rank=0, local_rank=0,
|
||||||
|
distributed_init_method="tcp://127.0.0.1:29602",
|
||||||
|
)
|
||||||
|
with set_current_vllm_config(VllmConfig()):
|
||||||
|
initialize_model_parallel(tensor_model_parallel_size=1)
|
||||||
|
|
||||||
|
device = torch.device("cuda")
|
||||||
|
torch.manual_seed(2301)
|
||||||
|
tensors = {
|
||||||
|
"q": synth(4096, 4096, args.ones),
|
||||||
|
"k": synth(4096, 4096, args.ones),
|
||||||
|
"v": synth(4096, 4096, args.ones),
|
||||||
|
"to_out": synth(4096, 4096, args.ones),
|
||||||
|
}
|
||||||
|
|
||||||
|
b = args.block
|
||||||
|
p = f"transformer.transformer_blocks.{b}"
|
||||||
|
results = []
|
||||||
|
with torch.device(device.type):
|
||||||
|
to_qkv = QKVParallelLinear(
|
||||||
|
hidden_size=4096, head_size=128, total_num_heads=32, bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.attn.to_qkv",
|
||||||
|
)
|
||||||
|
for part, shard in (("q", "q"), ("k", "k"), ("v", "v")):
|
||||||
|
q, s = tensors[part]
|
||||||
|
to_qkv.weight.weight_loader(to_qkv.weight, q.to(device), shard)
|
||||||
|
to_qkv.weight_scale.weight_loader(to_qkv.weight_scale, s.to(device), shard)
|
||||||
|
to_qkv.quant_method.process_weights_after_loading(to_qkv)
|
||||||
|
|
||||||
|
to_out = RowParallelLinear(
|
||||||
|
4096, 4096, bias=False, input_is_parallel=True, return_bias=False,
|
||||||
|
quant_config=quant_config, prefix=f"{p}.attn.to_out.0",
|
||||||
|
)
|
||||||
|
q, s = tensors["to_out"]
|
||||||
|
to_out.weight.weight_loader(to_out.weight, q.to(device))
|
||||||
|
to_out.weight_scale.weight_loader(to_out.weight_scale, s.to(device))
|
||||||
|
loaded_bitwise_ok = bool(torch.equal(to_out.weight.detach().cpu(), q))
|
||||||
|
loaded_scale_ok = bool(torch.allclose(to_out.weight_scale.detach().float().cpu(), s))
|
||||||
|
to_out.quant_method.process_weights_after_loading(to_out)
|
||||||
|
|
||||||
|
for name, layer, parts, in_features in (
|
||||||
|
("attn.to_qkv (fused q/k/v)", to_qkv, ("q", "k", "v"), 4096),
|
||||||
|
("attn.to_out.0 (row)", to_out, ("to_out",), 4096),
|
||||||
|
):
|
||||||
|
if args.ones:
|
||||||
|
x = torch.ones(4, in_features, dtype=getattr(torch, args.act_dtype), device=device)
|
||||||
|
else:
|
||||||
|
x = torch.randn(512, in_features, dtype=getattr(torch, args.act_dtype), device=device)
|
||||||
|
result = layer(x)
|
||||||
|
out = (result[0] if isinstance(result, tuple) else result).float()
|
||||||
|
|
||||||
|
if args.ones:
|
||||||
|
expected = torch.full_like(out, float(in_features))
|
||||||
|
rel = float((out - expected).norm() / expected.norm())
|
||||||
|
finite = bool(torch.isfinite(out).all())
|
||||||
|
results.append({
|
||||||
|
"layer": name,
|
||||||
|
"rel_l2": round(rel, 6) if finite else "NaN",
|
||||||
|
"out_finite": finite,
|
||||||
|
"out_row0_first8": [round(v, 2) for v in out[0, :8].tolist()],
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
|
||||||
|
ref_w = torch.cat([tensors[k][0].float() * tensors[k][1] for k in parts], dim=0)
|
||||||
|
# Reference with the same dynamic per-token symmetric int8 activation
|
||||||
|
# quantization the kernel applies (dequantized acts, dequantized weights).
|
||||||
|
xs = x.float()
|
||||||
|
act_scale = xs.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / 127.0
|
||||||
|
x_dq = torch.round(xs / act_scale).clamp(-127, 127) * act_scale
|
||||||
|
ref = x_dq @ ref_w.T.to(device)
|
||||||
|
finite = bool(torch.isfinite(out).all())
|
||||||
|
rel = float((out - ref).norm() / ref.norm())
|
||||||
|
row = {"layer": name, "rel_l2": round(rel, 5) if finite else "NaN", "out_finite": finite}
|
||||||
|
if name.startswith("attn.to_out"):
|
||||||
|
row["loaded_bitwise_ok"] = loaded_bitwise_ok
|
||||||
|
row["loaded_scale_ok"] = loaded_scale_ok
|
||||||
|
# Reference computed from the layer's own post-repack weight
|
||||||
|
# parameter ((K, N) after the transpose) and scales.
|
||||||
|
w_layer = to_out.weight.detach().float() # (K, N)
|
||||||
|
s_layer = to_out.weight_scale.detach().float().squeeze(1) # (N,)
|
||||||
|
ref_layer = x_dq @ (w_layer * s_layer.unsqueeze(0)).to(device)
|
||||||
|
row["rel_l2_vs_layer_weight"] = round(float((out - ref_layer).norm() / ref_layer.norm()), 5)
|
||||||
|
results.append(row)
|
||||||
|
|
||||||
|
destroy_model_parallel()
|
||||||
|
destroy_distributed_environment()
|
||||||
|
print(json.dumps(results, indent=1))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||||
|
"""Tests for the quantization config factory's method-name and disk-dict resolution."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors import (
|
||||||
|
CompressedTensorsConfig,
|
||||||
|
)
|
||||||
|
|
||||||
|
from vllm_omni.quantization import build_quant_config, resolve_component_quant_config
|
||||||
|
from vllm_omni.quantization.component_config import (
|
||||||
|
ComponentQuantizationConfig,
|
||||||
|
is_prequantized_config,
|
||||||
|
resolve_encoder_quant_config,
|
||||||
|
)
|
||||||
|
from vllm_omni.quantization.factory import SUPPORTED_QUANTIZATION_METHODS, _build_single
|
||||||
|
|
||||||
|
|
||||||
|
def _int8_weight_only_disk_config() -> dict:
|
||||||
|
"""Serialized compressed-tensors int8 weight-only metadata as it appears on disk."""
|
||||||
|
weights = {
|
||||||
|
"num_bits": 8,
|
||||||
|
"type": "int",
|
||||||
|
"symmetric": True,
|
||||||
|
"strategy": "channel",
|
||||||
|
"dynamic": False,
|
||||||
|
"observer": "minmax",
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"quant_method": "compressed-tensors",
|
||||||
|
"format": "pack-quantized",
|
||||||
|
"config_groups": {"group_0": {"targets": ["Linear"], "weights": weights}},
|
||||||
|
"ignore": ["img_in"],
|
||||||
|
"quantization_status": "frozen",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_hyphenated_registry_method_builds_from_disk_schema() -> None:
|
||||||
|
# A bare str spec cannot construct compressed-tensors (a serialized-format
|
||||||
|
# method with no parameterless default); the serialized schema must build.
|
||||||
|
with pytest.raises(TypeError, match="Cannot instantiate CompressedTensorsConfig"):
|
||||||
|
_build_single("compressed-tensors")
|
||||||
|
disk = _int8_weight_only_disk_config()
|
||||||
|
disk.pop("quant_method")
|
||||||
|
config = _build_single("compressed-tensors", **disk)
|
||||||
|
assert isinstance(config, CompressedTensorsConfig)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("spelling", ["compressed-tensors", "compressed_tensors"])
|
||||||
|
def test_disk_dict_builds_via_from_config(spelling: str) -> None:
|
||||||
|
disk = _int8_weight_only_disk_config()
|
||||||
|
disk["quant_method"] = spelling
|
||||||
|
config = build_quant_config(disk)
|
||||||
|
assert isinstance(config, CompressedTensorsConfig)
|
||||||
|
|
||||||
|
|
||||||
|
def test_per_component_dict_routes_transformer_only() -> None:
|
||||||
|
disk = {"transformer": _int8_weight_only_disk_config()}
|
||||||
|
config = build_quant_config(disk)
|
||||||
|
assert isinstance(config, ComponentQuantizationConfig)
|
||||||
|
assert isinstance(
|
||||||
|
resolve_component_quant_config(config, "transformer"), CompressedTensorsConfig
|
||||||
|
)
|
||||||
|
encoder = resolve_encoder_quant_config(
|
||||||
|
resolve_component_quant_config(config, "text_encoder")
|
||||||
|
)
|
||||||
|
assert encoder is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_method_still_raises() -> None:
|
||||||
|
with pytest.raises(ValueError, match="Unknown quantization method"):
|
||||||
|
_build_single("not-a-real-method")
|
||||||
|
|
||||||
|
|
||||||
|
def test_supported_list_advertises_hyphenated_registry_names() -> None:
|
||||||
|
assert "compressed-tensors" in SUPPORTED_QUANTIZATION_METHODS
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_prequantized_config_recognizes_serialized_formats() -> None:
|
||||||
|
# compressed-tensors is always a serialized format: its linear methods
|
||||||
|
# consume packed checkpoint tensors and there is no on-load quantization.
|
||||||
|
assert is_prequantized_config(build_quant_config(_int8_weight_only_disk_config()))
|
||||||
|
component = build_quant_config({"transformer": _int8_weight_only_disk_config()})
|
||||||
|
assert isinstance(component, ComponentQuantizationConfig)
|
||||||
|
assert is_prequantized_config(component)
|
||||||
|
assert not is_prequantized_config(None)
|
||||||
|
# Online (non-serialized) methods stay unrecognized so the loader keeps
|
||||||
|
# the accelerator-side load-quantize-offload path for them.
|
||||||
|
from vllm_omni.quantization.fp8_config import DiffusionFp8Config
|
||||||
|
|
||||||
|
assert not is_prequantized_config(DiffusionFp8Config())
|
||||||
@@ -63,6 +63,7 @@ from vllm_omni.diffusion.offloader.module_collector import ModuleDiscovery, Pipe
|
|||||||
from vllm_omni.diffusion.offloader.offload_plan import get_offload_plan
|
from vllm_omni.diffusion.offloader.offload_plan import get_offload_plan
|
||||||
from vllm_omni.diffusion.registry import initialize_model
|
from vllm_omni.diffusion.registry import initialize_model
|
||||||
from vllm_omni.model_executor.model_loader.weight_utils import download_weights_from_hf_specific
|
from vllm_omni.model_executor.model_loader.weight_utils import download_weights_from_hf_specific
|
||||||
|
from vllm_omni.quantization.component_config import is_prequantized_config
|
||||||
from vllm_omni.transformers_utils.repo_utils import hf_api
|
from vllm_omni.transformers_utils.repo_utils import hf_api
|
||||||
|
|
||||||
|
|
||||||
@@ -695,16 +696,17 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
|||||||
}
|
}
|
||||||
if load_format is None:
|
if load_format is None:
|
||||||
load_format = "default"
|
load_format = "default"
|
||||||
# CPU offload + quantization: for offline-quantized models (e.g., AutoRound MXFP8),
|
# CPU offload + quantization: for offline-quantized models (e.g., AutoRound MXFP8,
|
||||||
# weights are already quantized in the checkpoint — load directly on CPU.
|
# compressed-tensors), weights are already quantized in the checkpoint — load
|
||||||
# For online quantization, load on device so quantization can run on accelerator,
|
# directly on CPU. For online quantization, load on device so quantization can
|
||||||
# then move back to CPU afterward.
|
# run on accelerator, then move back to CPU afterward.
|
||||||
offload_after_quant = False
|
offload_after_quant = False
|
||||||
|
offline_quant_direct_cpu = False
|
||||||
if load_device == "cpu" and self.quant_config is not None and device is not None:
|
if load_device == "cpu" and self.quant_config is not None and device is not None:
|
||||||
quant_cfg = self.quant_config
|
quant_cfg = self.quant_config
|
||||||
is_offline = getattr(quant_cfg, "data_type", None) == "mx_fp" or getattr(
|
is_offline = getattr(quant_cfg, "data_type", None) == "mx_fp" or getattr(
|
||||||
quant_cfg, "is_checkpoint_quantized", False
|
quant_cfg, "is_checkpoint_quantized", False
|
||||||
)
|
) or is_prequantized_config(quant_cfg)
|
||||||
if not is_offline:
|
if not is_offline:
|
||||||
load_device = device.type
|
load_device = device.type
|
||||||
offload_after_quant = True
|
offload_after_quant = True
|
||||||
@@ -713,6 +715,7 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
|||||||
load_device,
|
load_device,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
|
offline_quant_direct_cpu = True
|
||||||
logger.info("Offline-quantized model with CPU offload, loading weights directly on CPU")
|
logger.info("Offline-quantized model with CPU offload, loading weights directly on CPU")
|
||||||
|
|
||||||
target_device = torch.device(load_device)
|
target_device = torch.device(load_device)
|
||||||
@@ -874,7 +877,14 @@ class DiffusersPipelineLoader(HWRLoaderMixin):
|
|||||||
else:
|
else:
|
||||||
self.load_weights(model)
|
self.load_weights(model)
|
||||||
self._maybe_fuse_distilled_lora(model)
|
self._maybe_fuse_distilled_lora(model)
|
||||||
self._process_weights_after_loading(model, target_device)
|
# Post-load transforms (e.g. the Marlin repack for
|
||||||
|
# compressed-tensors) only run on the accelerator, so an
|
||||||
|
# offline-quantized CPU-offload load round-trips each
|
||||||
|
# quantized module through the device and back to the host.
|
||||||
|
process_device = (
|
||||||
|
device if offline_quant_direct_cpu and device is not None else target_device
|
||||||
|
)
|
||||||
|
self._process_weights_after_loading(model, process_device)
|
||||||
|
|
||||||
# A warm final-layout hit has already completed all
|
# A warm final-layout hit has already completed all
|
||||||
# byte-changing work through the restorer. Shared runtime
|
# byte-changing work through the restorer. Shared runtime
|
||||||
|
|||||||
@@ -33,6 +33,26 @@ PRE_QUANTIZED_METHODS: frozenset[str] = frozenset(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def is_prequantized_config(quant_config: QuantizationConfig | None) -> bool:
|
||||||
|
"""True when the config can only describe pre-quantized (serialized) weights.
|
||||||
|
|
||||||
|
Such checkpoints need no accelerator-side quantization pass at load, so a
|
||||||
|
CPU-offload load can read weights directly on the host. compressed-tensors
|
||||||
|
is the canonical case: its linear methods consume packed checkpoint tensors
|
||||||
|
(weight_packed/weight_scale) directly and vLLM has no on-load quantization
|
||||||
|
path for the format. A component wrapper is pre-quantized when every routed
|
||||||
|
component (and the default) is itself pre-quantized or unquantized.
|
||||||
|
"""
|
||||||
|
if quant_config is None:
|
||||||
|
return False
|
||||||
|
if isinstance(quant_config, ComponentQuantizationConfig):
|
||||||
|
routed = [*quant_config.component_configs.values(), quant_config.default_config]
|
||||||
|
return any(c is not None for c in routed) and all(
|
||||||
|
c is None or is_prequantized_config(c) for c in routed
|
||||||
|
)
|
||||||
|
return quant_config.get_name() == "compressed-tensors"
|
||||||
|
|
||||||
|
|
||||||
def resolve_component_quant_config(
|
def resolve_component_quant_config(
|
||||||
quant_config: QuantizationConfig | None,
|
quant_config: QuantizationConfig | None,
|
||||||
component: str,
|
component: str,
|
||||||
|
|||||||
@@ -326,20 +326,36 @@ def _build_from_method_and_config(method: str, config: Mapping[str, Any]) -> Qua
|
|||||||
return _build_single(method, **config)
|
return _build_single(method, **config)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_registry_method_name(method: str) -> str | None:
|
||||||
|
"""Resolve a normalized (underscore) method name to vLLM's registry spelling.
|
||||||
|
|
||||||
|
vLLM's ``QUANTIZATION_METHODS`` keeps canonical names with their historical
|
||||||
|
spelling — including hyphenated ones such as ``compressed-tensors`` — while
|
||||||
|
``_normalize_method_name`` canonicalizes dashes to underscores for the
|
||||||
|
``_OVERRIDES`` lookup. Try both spellings so hyphenated registry entries
|
||||||
|
remain reachable.
|
||||||
|
"""
|
||||||
|
for candidate in (method, method.replace("_", "-")):
|
||||||
|
if candidate in QUANTIZATION_METHODS:
|
||||||
|
return candidate
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
||||||
"""Build a single QuantizationConfig by method name.
|
"""Build a single QuantizationConfig by method name.
|
||||||
|
|
||||||
Resolution: _OVERRIDES first, then vLLM registry via from_config().
|
Resolution: _OVERRIDES first, then vLLM registry via from_config().
|
||||||
"""
|
"""
|
||||||
method = _normalize_method_name(method)
|
normalized = _normalize_method_name(method)
|
||||||
|
|
||||||
if method in _OVERRIDES:
|
if normalized in _OVERRIDES:
|
||||||
return _OVERRIDES[method](**kwargs)
|
return _OVERRIDES[normalized](**kwargs)
|
||||||
|
|
||||||
if method not in QUANTIZATION_METHODS:
|
registry_method = _resolve_registry_method_name(normalized)
|
||||||
|
if registry_method is None:
|
||||||
raise ValueError(f"Unknown quantization method: {method!r}. Supported: {SUPPORTED_QUANTIZATION_METHODS}")
|
raise ValueError(f"Unknown quantization method: {method!r}. Supported: {SUPPORTED_QUANTIZATION_METHODS}")
|
||||||
|
|
||||||
config_cls = get_quantization_config(method)
|
config_cls = get_quantization_config(registry_method)
|
||||||
|
|
||||||
from vllm.config.quantization import resolve_quantization_config
|
from vllm.config.quantization import resolve_quantization_config
|
||||||
from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig
|
from vllm.model_executor.layers.quantization.online.base import OnlineQuantizationConfig
|
||||||
@@ -347,7 +363,7 @@ def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
|||||||
if config_cls is OnlineQuantizationConfig:
|
if config_cls is OnlineQuantizationConfig:
|
||||||
if "args" in kwargs:
|
if "args" in kwargs:
|
||||||
return config_cls(**kwargs)
|
return config_cls(**kwargs)
|
||||||
args = resolve_quantization_config(method, kwargs or None)
|
args = resolve_quantization_config(registry_method, kwargs or None)
|
||||||
if args is None:
|
if args is None:
|
||||||
raise ValueError("Online quantization requires quantization config arguments")
|
raise ValueError("Online quantization requires quantization config arguments")
|
||||||
return config_cls(args)
|
return config_cls(args)
|
||||||
@@ -355,6 +371,10 @@ def _build_single(method: str, **kwargs: Any) -> QuantizationConfig:
|
|||||||
try:
|
try:
|
||||||
return config_cls(**kwargs)
|
return config_cls(**kwargs)
|
||||||
except TypeError:
|
except TypeError:
|
||||||
|
# Registry configs built from checkpoint metadata (e.g. compressed-tensors)
|
||||||
|
# take the serialized schema through from_config rather than __init__.
|
||||||
|
if kwargs and hasattr(config_cls, "from_config"):
|
||||||
|
return config_cls.from_config(dict(kwargs))
|
||||||
sig = inspect.signature(config_cls.__init__)
|
sig = inspect.signature(config_cls.__init__)
|
||||||
raise TypeError(
|
raise TypeError(
|
||||||
f"Cannot instantiate {config_cls.__name__} with kwargs {kwargs}. Expected signature: {sig}"
|
f"Cannot instantiate {config_cls.__name__} with kwargs {kwargs}. Expected signature: {sig}"
|
||||||
|
|||||||
Reference in new issue
Block a user