Files
vllm-omni/benchmarks/edge_harness/results/profile_lm_head_w4a16.json
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

1 line
1.9 KiB
JSON

{"execution_summary": {"estimated_inference_time": 3868, "estimated_inference_peak_memory": 137539584, "first_load_time": 140576, "first_load_peak_memory": 113246208, "warm_load_time": 149076, "warm_load_peak_memory": 125939712, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 1046944], "first_load_memory_peak_range": [753664, 9164752], "warm_load_memory_increase_range": [0, 2845344], "warm_load_memory_peak_range": [2326528, 11678224], "inference_memory_increase_range": [0, 75920], "inference_memory_peak_range": [24576, 8852352], "all_compile_times": [], "all_first_load_times": [140576], "all_warm_load_times": [149076], "all_inference_times": [5046, 4616, 4128, 4077, 4103, 4020, 3940, 3894, 4265, 4737, 4008, 4543, 4147, 4109, 4144, 4048, 3980, 3885, 3903, 3879, 3882, 3901, 3951, 3895, 3926, 3923, 3963, 3895, 3873, 4657, 3889, 3969, 3880, 3892, 4748, 3893, 3895, 3891, 3887, 3876, 3868, 3903, 3879, 3932, 4737, 3915, 3874, 3892, 3889, 3908, 3887, 3898, 3880, 3897, 3877, 3889, 3887, 3890, 3877, 4039, 3941, 3964, 3876, 3892, 3871, 3900, 3887, 3928, 3992, 3956, 3974, 5026, 4694, 4079, 4146, 4183, 4420, 4226, 4090, 4034, 3879, 3956, 3985, 3897, 3879, 3885, 3874, 3906, 3875, 3893, 3881, 4595, 3884, 3885, 3876, 3890, 3877, 3880, 3882, 3875]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11685}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 86165}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1495}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9206236}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35181}]}