vllm-omni/benchmarks/edge_harness/PROFILING.md

# Local model qualification and profiling

These runners exercise existing Omni implementations with real weights. They
do not add device or model support. A completed timing run is distinct from
reference-quality, streaming, fault-recovery and release qualification.

## Matrix and prerequisites

The [60-cell audit](results/model_device_matrix_20260922/README.md) covers twelve
PC/mobile/embedded configurations and five model families. Its raw evidence
distinguishes complete pipelines from components and backend-only probes.
Unavailable complete pipelines retain a reason and next repair step; a missing
measurement is never represented as zero latency.

Use the actual installed runtime and exact checkpoint identified in the result,
including quantization, rather than substituting another family member. The
current Spark CPU runner uses 1.7B INT8 and the CUDA runner uses 4B BF16. Their
results do not establish same-model acceleration. Qwen3-TTS uses the pinned
0.6B CustomVoice checkpoint, English/Vivian, through both Omni stages.

Windows and WSL are distinct configurations on the same physical laptop.
Run benchmarks serially: they share CPU, physical RAM, GPU and power limits.
GPU board power is not whole-device energy. Windows used RAM includes WSL;
do not add Windows and WSL counters as independent memory pools.

## Run the workload

Run from the repository root using the model's validated Python environment:

```bash
PYTHONPATH=. HF_HUB_OFFLINE=1 VLLM_USE_FLASHINFER_SAMPLER=0 \
  python benchmarks/edge_harness/profile_local_text.py \
  --model /absolute/path/to/checkpoint --out /absolute/path/to/new-run

PYTHONPATH=. HF_HUB_OFFLINE=1 VLLM_USE_FLASHINFER_SAMPLER=0 \
  python benchmarks/edge_harness/profile_local_tts.py \
  --model /absolute/path/to/customvoice-checkpoint --out /absolute/path/to/new-run
```

Set the validated CPU thread/KV-cache settings or CUDA compiler environment
before launching. Native Windows requires UTF-8 Python mode before interpreter
startup (`PYTHONUTF8=1`, `PYTHONIOENCODING=utf-8`). The common benchmark helper
uses the active environment's `torch/lib/cudart64_13.dll` when present and a
short `~/c29` vLLM cache path unless explicitly overridden. These reproduce
settings required by the tested Windows installation; they are not a general
claim about other CUDA/runtime versions.

Both runners require a new output directory and default to:

- Three input lengths × concurrency 1/2/4 × 20 measured requests per group.
- Separate warmup requests, excluded from group percentiles.
- At least 1800 seconds of continuous medium-input requests.
- Slow-consumer, cancellation and subsequent-request checks.

Text timing forces 128 output tokens with greedy sampling and records actual
input lengths, token IDs, sequence/epoch metadata and queue high-water marks.
It does not recommend those sampling settings for applications. TTS requires
DELTA output, records every audio chunk including terminal audio, verifies
finite PCM, and archives warmup float32 chunks with hashes. Playback underruns
are simulated from chunk arrivals; no sound-device playback or perceptual
quality is implied.

`profile_host_command.py` wraps an explicit command on Windows to collect
one-second host-memory, AC/battery and whole-NVIDIA-device telemetry. Each host
directory retains the exact command, exit status and combined log. Existing
disk/JIT caches are used. Constructor time is not a cold-disk measurement and
must remain separate from request latency. Sampled memory peaks are lower
bounds, not proof that every transient allocation was observed.

On Windows, a live reader can deny replacement of a report snapshot. The
writer defers such running updates, records the permission error, and requires
the final snapshot to persist. Request JSONL is the primary sample record.
Read live reports through Linux when they reside in WSL, or use a Windows
reader that permits file deletion/replacement. Failed profile/trace reports
produce a nonzero exit code in the current runners; historical attempts may
require inspection of the report rather than relying on the process exit code.

## Separate diagnostics

After timing runs finish, use `profile_trace_diagnostic.py --kind spark|tts`
with the same `--model` and a fresh `--out`. It invokes Omni's profiler RPCs and
measures three requests before, during and after tracing. Current text traces
use 32 generated tokens and disable expensive shape/memory instrumentation.
Compare paired diagnostic requests, not these 32-token times against the
128-token baseline. Profiler start/stop/export costs are recorded separately
when supported. The initial archived CPU diagnostic used full instrumentation
and 128 tokens; its configuration and limitations are retained in its report.

Run `check_text_reliability.py` or `check_tts_reliability.py` separately. They
test real-model interruption/recovery and deliberately terminate a worker
belonging to their own engine after verifying process ancestry. Text also
checks actual consumer-queue saturation, stale handles and oversized-plan
rejection. TTS's slow consumer does not prove saturation of every internal
buffer. Inspect individual outcomes: `status=completed` means the probe ran,
not that all checks passed. Any forced cleanup remains a finding.

## Analyze and qualify

For a separate TTS stage-attribution diagnostic, enable the existing
`VLLM_OMNI_STEP_STATS_DIR=/absolute/path/to/counters` hook and use
`VLLM_OMNI_STEP_STATS_SYNC=0`. Run `profile_local_tts.py` with
`--length-band medium --concurrency 1 --repeats 20 --sustained-seconds 0`
and a fresh output directory. This is not a full baseline run. The counters
include warmup, slow-consumer, abort and recovery activity, and nested timers
overlap. Preserve per-process summaries rather than pooling their percentiles.
These are host-side timers without accelerator synchronization. Periodic
flushing can omit the final counter tail after forced process termination.
Comparisons with an uninstrumented baseline must retain cache/thermal and
stochastic speech-length caveats.
Core-step counts include idle/no-work iterations. The current orchestrator
dispatch hook is only in the optional event-driven loop; the default loop
does not produce that counter. Absent counters are not zero-time measurements.

```bash
python benchmarks/edge_harness/summarize_e2e_profiles.py \
  --matrix benchmarks/edge_harness/results/model_device_matrix_20260922/matrix.json \
  --runs /absolute/path/to/run-root --out /absolute/path/to/summary

python benchmarks/edge_harness/analyze_profile_resources.py \
  --runs /absolute/path/to/run-root --out /absolute/path/to/resource-summary.json
```

Use run directory names `spark-cpu-wsl`, `spark-cuda-wsl`,
`spark-cuda-windows`, `tts-cuda-wsl` and `tts-cuda-windows`, with `-host`,
`-trace` and `-reliability` suffixes for their corresponding records. The matrix
summarizer verifies all sixty pairings and the original evidence hashes. It
reports nearest-rank p50/p95 and sample counts, checks per-group coverage,
flags recorded output-contract violations, and retains unavailable metrics.
Batch-dependent greedy outputs are a numerical/reproducibility finding;
timings alone cannot establish whether the cause is numerical or state-related.
Delivery-spacing statistics pool adjacent intervals within each request; their
sample count is the number of intervals. Text output updates may coalesce
tokens, so this is delivery timing rather than isolated kernel inter-token
latency. Effective prefill throughput includes all work before the first output.

Parse large traces only after inference timing has finished. The resource
analyzer streams trace JSON, separates summed event duration from interval
union, and excludes telemetry gaps over five seconds from energy integration.
Actual device placement, fallback, transfer and synchronization conclusions
must be supported by the trace/runtime records, not merely the intended plan.
The TTS diagnostic starts each stage with a distinct profiler prefix because
both workers have rank 0 and the current runtime otherwise overwrites their
shared trace filename. Both stages are active during traced requests. Sequential
stop/export RPCs can extend later-stage trace windows with idle/export time;
trace-window length is not model execution time or GPU utilization.
Startup summaries retain both constructor time and inclusive host-launch to
first warmup submission/output. The latter includes artifact hashing, plan
construction and imports; constructor timing alone excludes that preparation.

Keep compile/load/run failures and their exact configurations. Before declaring
a pairing qualified, review model-specific reference quality, complete declared
modalities, state/stream correctness, memory admission, executed placement and
the performance protocol. Missing artifacts, backends, device access, sensors
or reference suites require explicit reasons and repair actions.

After every writer exits, archive evidence with reversible compression of large
uncompressed files and an original/stored byte-hash manifest:

```bash
python benchmarks/edge_harness/archive_profiles.py create /path/to/run-root /path/to/new-archive
python benchmarks/edge_harness/archive_profiles.py verify /path/to/new-archive
python benchmarks/edge_harness/archive_profiles.py restore /path/to/new-archive /path/to/new-run-root
```

Restore before using original report-relative paths for compressed artifacts.
The utility does not rewrite raw reports. Integrity verification establishes
that archived bytes match, not that a model passes its qualification gates.