tzhouamandClaude Fable 5.1
37c0ea9a6b
- vllm_omni/edge/step_stats.py: process-local counters enabled by VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped per process and merged into an attribution table (share of the 80 ms frame). Hooks: stage_engine_core_proc (instrument after construction, dump at exit), chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime), orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats. - vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and the no-preemption KV admission rule (exact bytes/token from the talker KV geometry); adapt.derive_overrides now sizes max_num_seqs with it and auto_profile passes the geometry read from the model config. - Engine re-reads parallel_stage_init after apply_deploy_profile so the edge profile's orchestrator default takes effect (was read before the profile). - benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor) with per-step attribution. - Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py, tests/benchmarks/test_cpu_thread_sweep.py, profile default test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
14 lines
1.4 KiB
Markdown
14 lines
1.4 KiB
Markdown
# Edge benchmark results
|
|
|
|
Committed JSON produced by the edge branch benchmarks (kept small; WAVs are never committed):
|
|
|
|
- `stream_latency/<model>/<deploy>/<timestamp>[_tag].json` — `stream_latency_bench.py` output (schema 1): `meta` (model, deploy config, git SHA, GPU), `init` (`init_s`, optional init timeline), `resources` (peak RSS, peak GPU MiB, sampling mode), `requests[]` (per-chunk arrivals, TTFA, playback-start, stall at TTFA, RTF streamed / total) and `summary` (p50/p90/max).
|
|
- `init_profile/<timestamp>/summary.json` — `init_profile.py` comparison of initialization phases per configuration.
|
|
- `hw_emulation/<timestamp>/summary.json` — `hw_emulation.py` matrix (emulated hardware class x TTFA / playback-start / RTF / RSS; per-case bench JSON, `hw_profile.json` and init timeline next to it).
|
|
- `cpu_sweep/<timestamp>/summary.json` — `cpu_thread_sweep.py` (CPU-only 8/16/32 threads + inductor variant, with per-step attribution).
|
|
- `stream_latency/step_stats/<timestamp>/*.json` — raw per-process counters from `--step-stats` (merged into the bench JSON under `step_stats`).
|
|
- `codec_stream/` — `codec_stream_e2e.py` parity (server PCM vs client-decoded codec tokens, SNR) and client latency.
|
|
- `stream_latency/timelines/` — init timelines (`VLLM_OMNI_INIT_TIMELINE`) of the committed runs.
|
|
|
|
Metric definitions live in `benchmarks/tts/stream_latency_metrics.py`.
|