5e4b9f8c6e
Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
49 lines
1.2 KiB
Markdown
49 lines
1.2 KiB
Markdown
---
|
|
title: Benchmarking
|
|
kind: module
|
|
status: draft
|
|
owners:
|
|
- "@alex-jw-brooks"
|
|
- "@Bounty-hunter"
|
|
primary_code_paths:
|
|
- vllm_omni/benchmarks/**
|
|
- vllm_omni/entrypoints/cli/benchmark/**
|
|
related_code_paths:
|
|
- benchmarks/**
|
|
depends_on:
|
|
- entrypoints.md
|
|
- profiling.md
|
|
validation_paths:
|
|
- tests/benchmarks/**
|
|
- tests/dfx/perf/**
|
|
upstream_refs:
|
|
- vllm.benchmarks
|
|
last_reviewed: 2026-07-16
|
|
---
|
|
|
|
# Benchmarking
|
|
|
|
Benchmarking defines reproducible workloads, metrics, warm-up, measurement,
|
|
and result metadata for comparing vLLM-Omni performance.
|
|
|
|
## Candidate invariants
|
|
|
|
### BENCH-INV-001: Results are reproducible
|
|
|
|
**Rule:** Results MUST identify the model, revision, commit, hardware,
|
|
configuration, workload, concurrency, warm-up, and measurement window.
|
|
|
|
### BENCH-INV-002: Correctness precedes performance
|
|
|
|
**Rule:** A benchmark MUST validate successful and semantically acceptable
|
|
outputs before using timings as performance evidence.
|
|
|
|
### BENCH-INV-003: Metric definitions are explicit
|
|
|
|
**Rule:** Metrics MUST state their unit, population, and aggregation.
|
|
|
|
## Safe-change guide
|
|
|
|
Test workload parsing and metric calculations with deterministic fixtures. Do
|
|
not compare materially different workloads or runtime settings.
|