vllm-omni/docs/design/module/benchmarking.md

title, kind, status, owners, primary_code_paths, related_code_paths, depends_on, validation_paths, upstream_refs, last_reviewed

title	kind	status	owners	primary_code_paths	related_code_paths	depends_on	validation_paths	upstream_refs	last_reviewed
Benchmarking	module	draft	@alex-jw-brooks @Bounty-hunter	vllm_omni/benchmarks/** vllm_omni/entrypoints/cli/benchmark/**	benchmarks/**	entrypoints.md profiling.md	tests/benchmarks/** tests/dfx/perf/**	vllm.benchmarks	2026-07-16

Benchmarking

Benchmarking defines reproducible workloads, metrics, warm-up, measurement, and result metadata for comparing vLLM-Omni performance.

Candidate invariants

BENCH-INV-001: Results are reproducible

Rule: Results MUST identify the model, revision, commit, hardware, configuration, workload, concurrency, warm-up, and measurement window.

BENCH-INV-002: Correctness precedes performance

Rule: A benchmark MUST validate successful and semantically acceptable outputs before using timings as performance evidence.

BENCH-INV-003: Metric definitions are explicit

Rule: Metrics MUST state their unit, population, and aggregation.

Safe-change guide

Test workload parsing and metric calculations with deterministic fixtures. Do not compare materially different workloads or runtime settings.