anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
e0142926b66b9b8a98e2f9fc59b25b4d7a472450
vllm-omni/baselines/llama_cpp/llamacpp_bench.json
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork 
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

1083 lines
32 KiB
JSON
Raw Blame History

[
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:48:28Z",
"avg_ns": 5826194444,
"stddev_ns": 18499227,
"avg_ts": 87.879684,
"stddev_ts": 0.278992,
"samples_ns": [ 5805110833, 5811473500, 5847614162, 5841912112, 5824861616 ],
"samples_ts": [ 88.1981, 88.1016, 87.5571, 87.6425, 87.8991 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:49:03Z",
"avg_ns": 14610334471,
"stddev_ns": 123024936,
"avg_ts": 8.761414,
"stddev_ts": 0.072994,
"samples_ns": [ 14565270602, 14580640193, 14828037521, 14541752491, 14535971551 ],
"samples_ts": [ 8.78803, 8.77876, 8.6323, 8.80224, 8.80574 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:50:16Z",
"avg_ns": 2918362099,
"stddev_ns": 12286131,
"avg_ts": 175.443371,
"stddev_ts": 0.741092,
"samples_ns": [ 2925259382, 2929673132, 2898545311, 2923305305, 2915027369 ],
"samples_ts": [ 175.027, 174.764, 176.64, 175.144, 175.642 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:50:34Z",
"avg_ns": 7865646966,
"stddev_ns": 53771152,
"avg_ts": 16.273905,
"stddev_ts": 0.111300,
"samples_ns": [ 7791699373, 7865327825, 7937201692, 7889061755, 7844944189 ],
"samples_ts": [ 16.4277, 16.274, 16.1266, 16.225, 16.3162 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:51:13Z",
"avg_ns": 1491348299,
"stddev_ns": 5631768,
"avg_ts": 343.317407,
"stddev_ts": 1.294111,
"samples_ns": [ 1489819485, 1499678994, 1485448813, 1494052451, 1487741754 ],
"samples_ts": [ 343.666, 341.406, 344.677, 342.692, 344.146 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:51:22Z",
"avg_ns": 5197057974,
"stddev_ns": 84695512,
"avg_ts": 24.634512,
"stddev_ts": 0.398355,
"samples_ns": [ 5165558444, 5161453873, 5102450401, 5324858098, 5230969056 ],
"samples_ts": [ 24.7795, 24.7992, 25.086, 24.0382, 24.4697 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:51:48Z",
"avg_ns": 825028648,
"stddev_ns": 6429234,
"avg_ts": 620.614565,
"stddev_ts": 4.820950,
"samples_ns": [ 819600650, 819801745, 830087549, 822025580, 833627717 ],
"samples_ts": [ 624.694, 624.541, 616.802, 622.852, 614.183 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
"model_type": "spark2_5 1.7B BF16",
"model_size": 3415547904,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:51:53Z",
"avg_ns": 4431906380,
"stddev_ns": 123688081,
"avg_ts": 28.899578,
"stddev_ts": 0.810937,
"samples_ns": [ 4392305744, 4530404684, 4573176188, 4261345178, 4402300109 ],
"samples_ts": [ 29.1419, 28.2535, 27.9893, 30.0375, 29.0757 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:52:20Z",
"avg_ns": 2086923625,
"stddev_ns": 25271587,
"avg_ts": 245.366053,
"stddev_ts": 2.978325,
"samples_ns": [ 2055651221, 2066285152, 2110647408, 2110779760, 2091254587 ],
"samples_ts": [ 249.069, 247.788, 242.58, 242.564, 244.829 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:52:33Z",
"avg_ns": 5013971032,
"stddev_ns": 41743502,
"avg_ts": 25.530080,
"stddev_ts": 0.212086,
"samples_ns": [ 5039794469, 5070632551, 5009766482, 4977188073, 4972473585 ],
"samples_ts": [ 25.3979, 25.2434, 25.5501, 25.7173, 25.7417 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:52:58Z",
"avg_ns": 1060499778,
"stddev_ns": 5400430,
"avg_ts": 482.801252,
"stddev_ts": 2.458365,
"samples_ns": [ 1053815970, 1059128196, 1066100805, 1065952731, 1057501191 ],
"samples_ts": [ 485.853, 483.416, 480.255, 480.321, 484.16 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:53:04Z",
"avg_ns": 2908476289,
"stddev_ns": 47792262,
"avg_ts": 44.018841,
"stddev_ts": 0.725951,
"samples_ns": [ 2953468801, 2917824550, 2869732967, 2848706503, 2952648627 ],
"samples_ts": [ 43.3389, 43.8683, 44.6035, 44.9327, 43.3509 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:53:19Z",
"avg_ns": 576466351,
"stddev_ns": 8725348,
"avg_ts": 888.330924,
"stddev_ts": 13.307556,
"samples_ns": [ 568632667, 572234118, 571232829, 580205764, 590026378 ],
"samples_ts": [ 900.406, 894.739, 896.307, 882.446, 867.758 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:53:22Z",
"avg_ns": 2455402468,
"stddev_ns": 22724145,
"avg_ts": 52.133526,
"stddev_ts": 0.483419,
"samples_ns": [ 2466040684, 2466684755, 2481092179, 2427613738, 2435580988 ],
"samples_ts": [ 51.9051, 51.8915, 51.5902, 52.7267, 52.5542 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:53:35Z",
"avg_ns": 351336879,
"stddev_ns": 6538663,
"avg_ts": 1457.702717,
"stddev_ts": 27.668807,
"samples_ns": [ 354177005, 353254854, 340174283, 351878376, 357199877 ],
"samples_ts": [ 1445.6, 1449.38, 1505.11, 1455.05, 1433.37 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
"model_type": "spark2_5 1.7B Q8_0",
"model_size": 1814728704,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:53:37Z",
"avg_ns": 1930300252,
"stddev_ns": 109214314,
"avg_ts": 66.480943,
"stddev_ts": 3.761762,
"samples_ns": [ 1947853181, 1955129848, 2082878516, 1880261716, 1785378002 ],
"samples_ts": [ 65.7134, 65.4688, 61.4534, 68.0756, 71.6935 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:53:50Z",
"avg_ns": 2313133293,
"stddev_ns": 36186266,
"avg_ts": 221.387418,
"stddev_ts": 3.406370,
"samples_ns": [ 2375000633, 2314305249, 2299530128, 2289367141, 2287463318 ],
"samples_ts": [ 215.579, 221.233, 222.654, 223.643, 223.829 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 4,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:54:04Z",
"avg_ns": 5179171846,
"stddev_ns": 35680419,
"avg_ts": 24.715311,
"stddev_ts": 0.169597,
"samples_ns": [ 5234609005, 5153997992, 5178746130, 5142573279, 5185932826 ],
"samples_ts": [ 24.4526, 24.8351, 24.7164, 24.8903, 24.6822 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:54:30Z",
"avg_ns": 1188981105,
"stddev_ns": 6871140,
"avg_ts": 430.632306,
"stddev_ts": 2.487893,
"samples_ns": [ 1185615908, 1186572492, 1196193805, 1180559099, 1195964224 ],
"samples_ts": [ 431.843, 431.495, 428.024, 433.693, 428.106 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 8,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:54:37Z",
"avg_ns": 2872538766,
"stddev_ns": 17359014,
"avg_ts": 44.561187,
"stddev_ts": 0.269161,
"samples_ns": [ 2848511671, 2872323411, 2872557128, 2871716659, 2897584962 ],
"samples_ts": [ 44.9357, 44.5632, 44.5596, 44.5726, 44.1747 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:54:52Z",
"avg_ns": 660202254,
"stddev_ns": 14648576,
"avg_ts": 775.825680,
"stddev_ts": 17.226725,
"samples_ns": [ 646606112, 661641326, 644173392, 671358896, 677231548 ],
"samples_ts": [ 791.827, 773.833, 794.817, 762.632, 756.019 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 16,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:54:56Z",
"avg_ns": 2314691648,
"stddev_ns": 51135411,
"avg_ts": 55.320440,
"stddev_ts": 1.216515,
"samples_ns": [ 2312103918, 2383338578, 2262024591, 2345921300, 2270069855 ],
"samples_ts": [ 55.3608, 53.7062, 56.5865, 54.5628, 56.3859 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 512,
"n_gen": 0,
"n_depth": 0,
"test_time": "2026-09-11T05:55:07Z",
"avg_ns": 391029137,
"stddev_ns": 14214073,
"avg_ts": 1310.735568,
"stddev_ts": 47.148189,
"samples_ns": [ 394859789, 386788168, 387958028, 373276159, 412263541 ],
"samples_ts": [ 1296.66, 1323.72, 1319.73, 1371.64, 1241.92 ]
},
{
"build_commit": "e71b80510",
"build_number": 10849,
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
"gpu_info": "",
"backends": "CPU",
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
"model_type": "spark2_5 1.7B Q4_K - Medium",
"model_size": 1102073856,
"model_n_params": 1707657216,
"n_batch": 2048,
"n_ubatch": 512,
"n_threads": 32,
"cpu_mask": "0x0",
"cpu_strict": false,
"poll": 50,
"type_k": "f16",
"type_v": "f16",
"n_gpu_layers": -1,
"n_cpu_moe": 0,
"split_mode": "layer",
"main_gpu": 0,
"no_kv_offload": false,
"flash_attn": -1,
"devices": "auto",
"tensor_split": "0.00",
"tensor_buft_overrides": "none",
"load_mode": "auto",
"lazy_mode": "auto",
"embeddings": false,
"no_op_offload": 0,
"no_host": false,
"fit_target": 0,
"fit_min_ctx": 0,
"n_prompt": 0,
"n_gen": 128,
"n_depth": 0,
"test_time": "2026-09-11T05:55:10Z",
"avg_ns": 1485928332,
"stddev_ns": 22400321,
"avg_ts": 86.157092,
"stddev_ts": 1.298278,
"samples_ns": [ 1514864818, 1485917904, 1471521773, 1499366549, 1457970619 ],
"samples_ts": [ 84.496, 86.142, 86.9848, 85.3694, 87.7933 ]
}
]
Reference in new issue View Git Blame Copy Permalink