tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
15 lines
8.7 KiB
JSON
15 lines
8.7 KiB
JSON
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 57.41, "rss_mb": 8670.4, "rss_peak_mb": 8702.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2557, "pp_tps": 2002.6, "tg_tokens": 128, "tg_s": 2.3198, "tg_tps": 54.75, "raw": {"pp": [0.25566731160506606, 0.2593384999781847, 0.2671640901826322], "tg": [2.368054673075676, 2.3595390100963414, 2.337980422191322], "tg1": [0.021325419656932354, 0.018553890753537416, 0.01813417114317417]}}
|
|
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.38, "rss_mb": 8709.2, "rss_peak_mb": 8729.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.222, "pp_tps": 2306.35, "tg_tokens": 128, "tg_s": 2.0763, "tg_tps": 61.17, "raw": {"pp": [0.2219955581240356, 0.25641375267878175, 0.25716910790652037], "tg": [2.398225273936987, 2.0967630273662508, 2.0924383951351047], "tg1": [0.01652884017676115, 0.016112301964312792, 0.017952477093786]}}
|
|
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 87.37, "rss_mb": 8062.0, "rss_peak_mb": 8089.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1671, "pp_tps": 3064.11, "tg_tokens": 128, "tg_s": 1.9204, "tg_tps": 66.13, "raw": {"pp": [0.16709583811461926, 0.1969963563606143, 0.2156481253914535], "tg": [2.0638905209489167, 1.9353438159450889, 2.022206211928278], "tg1": [0.014969798736274242, 0.015494609717279673, 0.01520752115175128]}}
|
|
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 84.05, "rss_mb": 9511.1, "rss_peak_mb": 9533.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2651, "pp_tps": 1931.13, "tg_tokens": 128, "tg_s": 1.9723, "tg_tps": 64.39, "raw": {"pp": [0.26512908516451716, 0.2868352457880974, 0.2711168541572988], "tg": [2.0084553682245314, 1.9880874049849808, 2.059911102987826], "tg1": [0.019026058726012707, 0.015752890147268772, 0.015950628090649843]}}
|
|
{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 82.16, "rss_mb": 8097.7, "rss_peak_mb": 9061.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3346.62, "tg_tokens": 128, "tg_s": 1.5022, "tg_tps": 84.54, "raw": {"pp": [0.1572714359499514, 0.1576147861778736, 0.15299037797376513], "tg": [1.5144385821186006, 1.5316730751655996, 1.522148437332362], "tg1": [0.012234935071319342, 0.01230821618810296, 0.012677683960646391]}}
|
|
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.18, "rss_mb": 8673.2, "rss_peak_mb": 8712.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2556, "pp_tps": 2002.96, "tg_tokens": 128, "tg_s": 1.8377, "tg_tps": 69.11, "raw": {"pp": [0.27302785124629736, 0.2624788419343531, 0.255622121039778], "tg": [2.8208564538508654, 1.8630885141901672, 2.3649882823228836], "tg1": [0.025365081150084734, 0.026843297760933638, 0.029472910799086094]}}
|
|
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.75, "rss_mb": 8686.3, "rss_peak_mb": 8714.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1596, "pp_tps": 3207.28, "tg_tokens": 128, "tg_s": 1.6919, "tg_tps": 75.06, "raw": {"pp": [0.16286137886345387, 0.16863904893398285, 0.15963682811707258], "tg": [1.71276545105502, 1.704967309255153, 1.748305486049503], "tg1": [0.0130898617208004, 0.013152792118489742, 0.013696761801838875]}}
|
|
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 48.85, "rss_mb": 8064.4, "rss_peak_mb": 8092.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1581, "pp_tps": 3238.56, "tg_tokens": 128, "tg_s": 1.6414, "tg_tps": 77.37, "raw": {"pp": [0.20444863196462393, 0.16155556635931134, 0.1580951907671988], "tg": [1.654232523869723, 1.6747960466891527, 1.668766028713435], "tg1": [0.012994957156479359, 0.012909624725580215, 0.012812163680791855]}}
|
|
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.65, "rss_mb": 9512.0, "rss_peak_mb": 9535.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.224, "pp_tps": 2285.48, "tg_tokens": 128, "tg_s": 1.584, "tg_tps": 80.18, "raw": {"pp": [0.22863413486629725, 0.22572151478379965, 0.2240230217576027], "tg": [1.5968111460097134, 1.6221620081923902, 1.6093115280382335], "tg1": [0.013640933204442263, 0.012812962755560875, 0.01369112217798829]}}
|
|
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.55, "rss_mb": 8657.9, "rss_peak_mb": 8694.2, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.7995, "pp_tps": 640.36, "tg_tokens": 128, "tg_s": 13.9584, "tg_tps": 9.1, "raw": {"pp": [0.8044024519622326, 0.8655711049214005, 0.7995461183600128], "tg": [14.970092439092696, 14.903033270966262, 13.9971464141272], "tg1": [0.18624361278489232, 0.03878649603575468, 0.08108040131628513]}}
|
|
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.91, "rss_mb": 8696.9, "rss_peak_mb": 8724.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1576, "pp_tps": 3248.3, "tg_tokens": 128, "tg_s": 1.7837, "tg_tps": 71.2, "raw": {"pp": [0.16414545476436615, 0.15762093104422092, 0.16772186197340488], "tg": [1.7981987767852843, 1.8183860406279564, 1.8294021431356668], "tg1": [0.014506598934531212, 0.01543053099885583, 0.01460421597585082]}}
|
|
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.34, "rss_mb": 8057.9, "rss_peak_mb": 8084.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3345.68, "tg_tokens": 128, "tg_s": 1.6146, "tg_tps": 78.66, "raw": {"pp": [0.15356025705114007, 0.15303313126787543, 0.1586752710863948], "tg": [1.6407190039753914, 1.6276185680180788, 1.6660430012270808], "tg1": [0.012987045105546713, 0.01297468226402998, 0.013056765776127577]}}
|
|
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.23, "rss_mb": 9514.0, "rss_peak_mb": 9536.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2323, "pp_tps": 2203.64, "tg_tokens": 128, "tg_s": 1.788, "tg_tps": 71.03, "raw": {"pp": [0.23234243784099817, 0.258395710028708, 0.2591813113540411], "tg": [1.8015480861067772, 1.803529669996351, 1.8111027288250625], "tg1": [0.014497074764221907, 0.01430785097181797, 0.013512391597032547]}}
|
|
{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.0, "rss_mb": 8461.2, "rss_peak_mb": 8485.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1784, "pp_tps": 2869.4, "tg_tokens": 128, "tg_s": 1.953, "tg_tps": 65.03, "raw": {"pp": [0.1784344231709838, 0.19405587995424867, 0.1978841326199472], "tg": [2.0177465733140707, 1.9670254928059876, 2.016505080740899], "tg1": [0.015316586010158062, 0.014635276980698109, 0.014058694709092379]}}
|