The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
1 line
1.9 KiB
JSON
1 line
1.9 KiB
JSON
{"execution_summary": {"estimated_inference_time": 3862, "estimated_inference_peak_memory": 138051584, "first_load_time": 139684, "first_load_peak_memory": 113541120, "warm_load_time": 146509, "warm_load_peak_memory": 125521920, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [2583936, 12322976], "first_load_memory_peak_range": [11759616, 21498656], "warm_load_memory_increase_range": [0, 1511168], "warm_load_memory_peak_range": [983040, 11197440], "inference_memory_increase_range": [0, 24624], "inference_memory_peak_range": [20480, 9138416], "all_compile_times": [], "all_first_load_times": [139684], "all_warm_load_times": [146509], "all_inference_times": [5092, 4632, 4065, 3881, 3887, 3880, 3894, 3879, 3888, 3888, 3876, 4173, 4120, 3950, 3914, 3967, 3883, 3897, 3912, 3978, 3984, 3885, 3951, 3901, 3879, 3948, 3880, 3891, 3880, 3890, 3965, 3891, 3884, 3891, 3897, 3885, 3881, 3882, 3905, 3880, 3895, 3898, 3909, 3862, 3877, 3877, 3872, 3872, 3923, 3928, 4000, 3953, 3980, 3959, 3962, 4915, 4755, 4061, 4097, 4072, 4171, 4131, 4070, 4054, 3944, 3908, 3892, 3887, 3877, 3918, 3982, 3874, 3873, 3874, 3893, 4553, 3871, 3876, 3888, 4111, 3866, 3893, 3870, 3877, 4307, 4161, 4048, 3968, 3883, 3936, 4038, 3969, 3958, 4774, 4849, 4110, 4140, 4342, 4381, 4277]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14198}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 151909}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1307}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9681411}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35501}]} |