Files
vllm-omni/benchmarks/edge_harness/results/profile_lm_head_w8a16.json
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

1 line
1.9 KiB
JSON

{"execution_summary": {"estimated_inference_time": 3862, "estimated_inference_peak_memory": 138051584, "first_load_time": 139684, "first_load_peak_memory": 113541120, "warm_load_time": 146509, "warm_load_peak_memory": 125521920, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [2583936, 12322976], "first_load_memory_peak_range": [11759616, 21498656], "warm_load_memory_increase_range": [0, 1511168], "warm_load_memory_peak_range": [983040, 11197440], "inference_memory_increase_range": [0, 24624], "inference_memory_peak_range": [20480, 9138416], "all_compile_times": [], "all_first_load_times": [139684], "all_warm_load_times": [146509], "all_inference_times": [5092, 4632, 4065, 3881, 3887, 3880, 3894, 3879, 3888, 3888, 3876, 4173, 4120, 3950, 3914, 3967, 3883, 3897, 3912, 3978, 3984, 3885, 3951, 3901, 3879, 3948, 3880, 3891, 3880, 3890, 3965, 3891, 3884, 3891, 3897, 3885, 3881, 3882, 3905, 3880, 3895, 3898, 3909, 3862, 3877, 3877, 3872, 3872, 3923, 3928, 4000, 3953, 3980, 3959, 3962, 4915, 4755, 4061, 4097, 4072, 4171, 4131, 4070, 4054, 3944, 3908, 3892, 3887, 3877, 3918, 3982, 3874, 3873, 3874, 3893, 4553, 3871, 3876, 3888, 4111, 3866, 3893, 3870, 3877, 4307, 4161, 4048, 3968, 3883, 3936, 4038, 3969, 3958, 4774, 4849, 4110, 4140, 4342, 4381, 4277]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14198}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 151909}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1307}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9681411}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35501}]}