Files
vllm-omni/benchmarks/edge_harness/results/vllm_cpu_gptq.jsonl
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

7 lines
3.8 KiB
JSON

{"tag": "gptq-w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.36, "rss_mb": 8450.5, "rss_peak_mb": 8476.7, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1191, "pp_tps": 4299.46, "tg_tokens": 128, "tg_s": 1.6702, "tg_tps": 76.04, "raw": {"pp": [0.11972225410863757, 0.11908483225852251, 0.12134581292048097], "tg": [1.732507862150669, 1.6979356687515974, 1.6837419890798628], "tg1": [0.01368768885731697, 0.013497897889465094, 0.013701466843485832]}}
{"tag": "gptq-w4a8-g128-head", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.6, "rss_mb": 8104.2, "rss_peak_mb": 9060.8, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1185, "pp_tps": 4320.16, "tg_tokens": 128, "tg_s": 1.4336, "tg_tps": 88.59, "raw": {"pp": [0.14377897279337049, 0.11851405678316951, 0.14887535758316517], "tg": [1.4482398289255798, 1.904409646987915, 2.0828713290393353], "tg1": [0.015112247318029404, 0.016326773911714554, 0.014672661200165749]}}
{"tag": "gptq-w4a16-g128-head", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.86, "rss_mb": 8617.9, "rss_peak_mb": 8620.6, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1099, "pp_tps": 4658.85, "tg_tokens": 128, "tg_s": 3.3844, "tg_tps": 37.53, "raw": {"pp": [0.11926615796983242, 0.115292611066252, 0.10989844426512718], "tg": [3.4093596399761736, 3.5208795364014804, 3.4581055720336735], "tg1": [0.02689351700246334, 0.025302201975136995, 0.02496027434244752]}}
{"tag": "gptq-w4a8-g128-head-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 52.79, "rss_mb": 8921.1, "rss_peak_mb": 8951.6, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1638, "pp_tps": 3126.36, "tg_tokens": 128, "tg_s": 1.5445, "tg_tps": 82.23, "raw": {"pp": [0.17734218668192625, 0.16376852383837104, 0.16509774001315236], "tg": [1.8541209190152586, 1.5617477837949991, 1.5574598712846637], "tg1": [0.013258503284305334, 0.012924029026180506, 0.014902695082128048]}}
{"tag": "gptq-w4a8-confirm1", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.82, "rss_mb": 8098.6, "rss_peak_mb": 9056.3, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1176, "pp_tps": 4353.18, "tg_tokens": 128, "tg_s": 1.4289, "tg_tps": 88.88, "raw": {"pp": [0.11795889306813478, 0.11761508556082845, 0.11978365201503038], "tg": [1.4589618942700326, 1.4405048349872231, 1.4417059477418661], "tg1": [0.011758381966501474, 0.011617015115916729, 0.011557874735444784]}}
{"tag": "gptq-w4a8-confirm2", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.91, "rss_mb": 8119.8, "rss_peak_mb": 9060.8, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1175, "pp_tps": 4357.17, "tg_tokens": 128, "tg_s": 1.4345, "tg_tps": 88.54, "raw": {"pp": [0.125939070712775, 0.12151174107566476, 0.1175074027851224], "tg": [1.4461753987707198, 1.4628022378310561, 1.4459114633500576], "tg1": [0.011875604279339314, 0.011563725769519806, 0.01145239407196641]}}