[Quantization][tools] Add diffusion quantization output comparison tool (#3175)

Signed-off-by: david6666666 <530634352@qq.com>
Signed-off-by: David Chen <530634352@qq.com>

docs/user_guide/quantization/overview.md

@@ -138,3 +138,98 @@ build_quant_config({"method": "gguf", "gguf_model": "/path/to/model.gguf"})
 build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
 build_quant_config(None)
 ```
+
+## Output Similarity Comparison Tool
+
+Use `vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity`
+to compare a reference diffusion run with a quantized candidate run using the
+same prompt, seed, resolution, scheduler settings, and inference steps. The
+tool compares final decoded images or video frames, and also reports generation
+latency and worker-reported peak memory when available.
+
+This is useful when validating whether online quantization, an offline
+pre-quantized checkpoint, or a new `ignored_layers` choice keeps generation
+quality close to the BF16 reference.
+
+### Online Quantization Example
+
+```bash
+python -m vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity \
+  --task t2i \
+  --model Qwen/Qwen-Image \
+  --candidate-quantization fp8 \
+  --ignored-layers img_mlp \
+  --prompt "a cup of coffee on the table" \
+  --height 512 --width 512 \
+  --num-inference-steps 20 \
+  --seed 142 \
+  --output-json /tmp/qwen_image_fp8_similarity/result.json \
+  --save-output-dir /tmp/qwen_image_fp8_similarity/images \
+  --enforce-eager
+```
+
+### Offline Checkpoint Example
+
+Use `--candidate-model` when the candidate is already quantized or lives at a
+different model path:
+
+```bash
+python -m vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity \
+  --task t2i \
+  --reference-model Qwen/Qwen-Image \
+  --candidate-model /path/to/qwen-image-fp8-checkpoint \
+  --prompt "a cup of coffee on the table" \
+  --height 512 --width 512 \
+  --num-inference-steps 20 \
+  --seed 142 \
+  --output-json /tmp/qwen_image_fp8_checkpoint_similarity/result.json
+```
+
+If the checkpoint does not include a loadable quantization config, pass one
+explicitly:
+
+```bash
+--candidate-quantization-config-json '{"method":"fp8"}'
+```
+
+### Output Metrics
+
+The output JSON includes `output_metrics`, `reference_generation`, and
+`candidate_generation`.
+
+| Metric | Direction | Meaning |
+|--------|-----------|---------|
+| `cosine_similarity` | Higher is better | Vector direction similarity between output pixels or frames. Useful as a broad sanity check. |
+| `mae` | Lower is better | Mean absolute pixel or frame error. For decoded outputs, values are in uint8 pixel units. |
+| `mse` / `rmse` | Lower is better | Squared error and its square root. These penalize localized large differences more than `mae`. |
+| `max_abs` | Lower is better | Worst single-element absolute error. Treat it as an outlier/debug signal, not as a release gate. |
+| `l2` / `relative_l2` | Lower is better | Absolute and reference-normalized L2 distance. `relative_l2` is easier to compare across resolutions. |
+| `psnr_db` | Higher is better | Pixel-space signal-to-noise ratio in dB for uint8 images or frames. |
+| `avg_generation_time_s` | Lower is better | Average wall-clock generation time across measured runs. |
+| `max_peak_memory_mb` | Lower is better | Maximum worker-reported peak device memory across measured runs, when the worker reports it. |
+
+Recommended starting thresholds for same-seed diffusion comparisons:
+
+| Metric | Smoke threshold | Stricter target | Notes |
+|--------|-----------------|-----------------|-------|
+| `psnr_db` | `>= 20.0` | `>= 25.0` | Good for quick image or frame regression checks. |
+| `mae` | `<= 12.0` | `<= 6.0` | Interpreted in decoded uint8 pixel units. |
+| `cosine_similarity` | `>= 0.98` | `>= 0.995` | Less sensitive to global scale than L2-style metrics. |
+| `relative_l2` | `<= 0.20` | `<= 0.08` | Useful when comparing across prompts or resolutions. |
+
+These thresholds are heuristics. Tune them by model family, task, resolution,
+quantization method, and deployment tolerance. For release gating, pair the
+numeric report with visual inspection of saved reference and candidate outputs.
+
+The tool intentionally reports separate quality, latency, and memory metrics
+instead of a single consolidated similarity score. A single score can hide
+important tradeoffs, for example a candidate with good PSNR but a meaningful
+memory regression, or a candidate with low average error but localized visual
+artifacts. If you need a project-specific pass/fail gate, define it as an
+explicit policy over the individual metrics.
+
+Pixel-level metrics do not measure semantic consistency. For higher-cost
+evaluation, you can complement this report with a vision-language judge that
+describes the reference and candidate outputs and compares those descriptions.
+Keep that semantic check separate from this lightweight tool so users can
+choose whether the additional model cost and latency are appropriate.