Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
tzhouamandClaude Opus 5 committed
1 parent
b36a3431bd
commit
9264ff81ab
162 files changed
+22640
-9
No files matched your search
+14
@@ -281,3 +281,17 @@ lance_compare/
|
||||
|
||||
# Linting binaries downloaded by local pre-commit hooks
|
||||
shellcheck*/
|
||||
|
||||
# --- vllm-embedding ---------------------------------------------------------
|
||||
# Measurement artifacts are the substance of this fork, not build output: every
|
||||
# published figure is traceable to a file below. They are tracked despite the
|
||||
# blanket `*.json` and `results/` rules above. Binary captures (.npz, .onnx,
|
||||
# .wav, .npy) stay ignored.
|
||||
!benchmarks/edge_harness/results/
|
||||
!benchmarks/edge_harness/results/**
|
||||
!baselines/
|
||||
!baselines/**
|
||||
benchmarks/edge_harness/results/**/*.npz
|
||||
benchmarks/edge_harness/results/**/*.onnx
|
||||
benchmarks/edge_harness/results/**/*.npy
|
||||
benchmarks/edge_harness/results/**/*.wav
|
||||
@@ -18,6 +18,17 @@ README.md; the substantive ones are:
|
||||
vllm_omni/platforms/cpu/, vllm_omni/patch.py
|
||||
CPU platform and CPU-backend patches
|
||||
benchmarks/edge_harness/
|
||||
measurement tooling (original to this fork)
|
||||
measurement tooling and results (original to this fork)
|
||||
baselines/, docs/engines/, docs/analysis/
|
||||
benchmark results and written analyses (original to this fork)
|
||||
|
||||
vLLM itself is not modified and no vLLM source is redistributed here. This fork
|
||||
changes vLLM's CPU behaviour through its own extension points at runtime
|
||||
(vllm_omni/patch.py); the pinned upstream versions are recorded in
|
||||
docs/edge/environment.md.
|
||||
|
||||
Baseline measurements reference llama.cpp (https://github.com/ggml-org/llama.cpp,
|
||||
MIT). No llama.cpp source is redistributed; only benchmark output produced with
|
||||
it, under baselines/llama_cpp/.
|
||||
|
||||
Upstream's README is preserved at docs/README.vllm-omni.md.
|
||||
@@ -12,14 +12,19 @@ measured came from the code path you thought it did.
|
||||
|
||||
Upstream's own README is preserved at [docs/README.vllm-omni.md](docs/README.vllm-omni.md).
|
||||
|
||||
## What this fork adds
|
||||
## What's here
|
||||
|
||||
| area | change |
|
||||
| area | where |
|
||||
|---|---|
|
||||
| 4-bit CPU quantization | `cpu_int4` — a W4A16 path built on PyTorch's tinygemm kernel, with the prefill path reading the kernel's own weight layout instead of a duplicate |
|
||||
| Memory | RoPE tables sized to the servable context; weights repacked in blocks; KV budget that is actually honoured on CPU |
|
||||
| 4-bit CPU quantization | `vllm_omni/model_executor/layers/quantization/cpu_int4.py` — W4A16 on PyTorch's tinygemm kernel, prefill reading the kernel's own weight layout instead of a duplicate |
|
||||
| Memory | RoPE tables sized to the servable context; weights repacked in blocks; a KV budget that is actually honoured on CPU |
|
||||
| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM itself is **unmodified**; see [docs/edge/environment.md](docs/edge/environment.md) |
|
||||
| Device routing | `vllm_omni/edge/weight_path.py` picks the weight format per device class, from measurements rather than assumption |
|
||||
| Measurement | `benchmarks/edge_harness/` — the benchmark, profiling, memory-attribution and fidelity tooling every number here was produced with |
|
||||
| Measurement harness | `benchmarks/edge_harness/` — the grid runner, benchmarks, memory ledger, fidelity metric and agent suite every number here came from |
|
||||
| Raw results | `benchmarks/edge_harness/results/` — 100+ artifacts backing the published figures, each with provenance |
|
||||
| Baselines | `baselines/` — llama.cpp runs and the comparison scripts |
|
||||
| Engine analyses | `docs/engines/` — llama.cpp, ExecuTorch, MNN, ncnn, vLLM, vLLM-Omni |
|
||||
| Study documents | `docs/analysis/` — the full reports these summaries distil |
|
||||
|
||||
## Headline results
|
||||
|
||||
@@ -47,6 +52,11 @@ These overlap and **must not be added** — see [docs/edge/results.md](docs/edge
|
||||
| `cpu_gemm_wna16` fallback | 32.3 tok/s | — |
|
||||
| llama.cpp Q4_K_M (reference) | 90.0 tok/s | 1121 tok/s |
|
||||
|
||||
Decode is llama.cpp's; prefill is ours by ~2×. Neither engine is
|
||||
bandwidth-bound — both sit at 33–39% of the measured 222 GB/s ceiling — so the
|
||||
decode gap is the execution model, not the kernels. See
|
||||
[docs/edge/comparisons.md](docs/edge/comparisons.md).
|
||||
|
||||
## Why the numbers here are worth reading
|
||||
|
||||
This project has retracted five conclusions, every one of them a measurement
|
||||
@@ -78,9 +88,17 @@ print(llm.generate(['Hello'], SamplingParams(max_tokens=32))[0].outputs[0].text)
|
||||
"
|
||||
```
|
||||
|
||||
See [docs/edge/cpu-4bit.md](docs/edge/cpu-4bit.md) for building a checkpoint and
|
||||
choosing between the 4-bit paths, and [ROADMAP.md](ROADMAP.md) for what is
|
||||
measured, what is next, and what was tried and abandoned.
|
||||
## Documentation
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [docs/edge/results.md](docs/edge/results.md) | every measured figure, with spread and baseline |
|
||||
| [docs/edge/methodology.md](docs/edge/methodology.md) | how to measure on a shared machine without fooling yourself — the five retractions and the rules they left |
|
||||
| [docs/edge/cpu-4bit.md](docs/edge/cpu-4bit.md) | the two 4-bit paths, the packed layout as measured, building a checkpoint |
|
||||
| [docs/edge/comparisons.md](docs/edge/comparisons.md) | against llama.cpp and the other engines considered |
|
||||
| [docs/edge/environment.md](docs/edge/environment.md) | pins, machine, and why there is no vLLM patch series |
|
||||
| [docs/edge/harness.md](docs/edge/harness.md) | running a comparison |
|
||||
| [ROADMAP.md](ROADMAP.md) | what is next, with expected gains and how each would be verified — and what was rejected |
|
||||
|
||||
## Licence
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Baselines
|
||||
|
||||
Measurements of other engines and of unquantized references, kept so the
|
||||
comparisons in [docs/edge/comparisons.md](../docs/edge/comparisons.md) can be
|
||||
checked rather than taken on trust.
|
||||
|
||||
## `llama_cpp/`
|
||||
|
||||
llama.cpp is the reference for CPU inference of a quantized model. Pinned
|
||||
commit and build flags are in
|
||||
[docs/edge/environment.md](../docs/edge/environment.md); no llama.cpp source is
|
||||
redistributed here, only output produced with it.
|
||||
|
||||
| file | what |
|
||||
|---|---|
|
||||
| `llamacpp_q4km_t{4,8,16,32}.json` | Q4_K_M across thread counts |
|
||||
| `llamacpp_grid_t*.json`, `llamacpp_node1_t*.json` | thread and NUMA-node sweeps |
|
||||
| `ab_llamacpp_*.json` | interleaved A/B against vLLM on the same cores |
|
||||
| `il_llamacpp_p{1,2,3}.json` | the interleaved passes behind the headline decode figure |
|
||||
| `cmp_llamacpp.py` | the comparison driver |
|
||||
|
||||
Both engines are pinned with `taskset` to the same physical cores and run
|
||||
strictly one at a time; vLLM's prefix caching is disabled, because `llama-bench`
|
||||
has no such cache and leaving it on would not be a comparison.
|
||||
|
||||
## Quality references
|
||||
|
||||
`quality_*.json` and `quality_cmp_*.json` hold the greedy-divergence comparison:
|
||||
each build against its **own** bf16 output over 12 prompts, plus the
|
||||
cross-engine roll-up. Read the caveats in
|
||||
[docs/edge/comparisons.md](../docs/edge/comparisons.md) before quoting these —
|
||||
the bit budgets are not matched, and the metric saturates.
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:59:20Z",
|
||||
"avg_ns": 580079289,
|
||||
"stddev_ns": 22445145,
|
||||
"avg_ts": 883.695051,
|
||||
"stddev_ts": 34.173372,
|
||||
"samples_ns": [ 550140411, 579685176, 611186497, 588372185, 571012179 ],
|
||||
"samples_ts": [ 930.671, 883.238, 837.715, 870.197, 896.653 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:59:24Z",
|
||||
"avg_ns": 2118395683,
|
||||
"stddev_ns": 31881177,
|
||||
"avg_ts": 60.434035,
|
||||
"stddev_ts": 0.909662,
|
||||
"samples_ns": [ 2110731151, 2152277901, 2078510078, 2149464130, 2100995155 ],
|
||||
"samples_ts": [ 60.6425, 59.4719, 61.5826, 59.5497, 60.9235 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:50:11Z",
|
||||
"avg_ns": 453009837,
|
||||
"stddev_ns": 4617262,
|
||||
"avg_ts": 1130.311918,
|
||||
"stddev_ts": 11.483962,
|
||||
"samples_ns": [ 459248279, 451575469, 456242077, 449875010, 448108352 ],
|
||||
"samples_ts": [ 1114.87, 1133.81, 1122.21, 1138.09, 1142.58 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:50:14Z",
|
||||
"avg_ns": 1475847924,
|
||||
"stddev_ns": 9833345,
|
||||
"avg_ts": 86.732884,
|
||||
"stddev_ts": 0.578067,
|
||||
"samples_ns": [ 1489621433, 1475111972, 1476264136, 1476396077, 1461846002 ],
|
||||
"samples_ts": [ 85.9279, 86.7731, 86.7054, 86.6976, 87.5605 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:44:52Z",
|
||||
"avg_ns": 487743656,
|
||||
"stddev_ns": 25108865,
|
||||
"avg_ts": 1051.969118,
|
||||
"stddev_ts": 54.409083,
|
||||
"samples_ns": [ 484545979, 492513415, 522973237, 486232711, 452452942 ],
|
||||
"samples_ts": [ 1056.66, 1039.57, 979.018, 1052.99, 1131.61 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T18:44:55Z",
|
||||
"avg_ns": 1589326576,
|
||||
"stddev_ns": 47473222,
|
||||
"avg_ts": 80.594928,
|
||||
"stddev_ts": 2.413871,
|
||||
"samples_ns": [ 1529730792, 1557930160, 1636596682, 1585818829, 1636556420 ],
|
||||
"samples_ts": [ 83.6749, 82.1603, 78.2111, 80.7154, 78.213 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Greedy-decode parity for llama.cpp against the same fp32 reference vLLM is
|
||||
scored on.
|
||||
|
||||
The point of the comparison is that "4-bit" is not one thing: llama.cpp's
|
||||
Q4_K_M and our `cpu_int4` spend a similar number of bits per weight but pick
|
||||
different grids, so the speed comparison is only meaningful next to a quality
|
||||
comparison run on identical prompts. llama-server takes a prompt as raw token
|
||||
ids, so both engines see exactly the token sequence the reference did.
|
||||
"""
|
||||
import argparse, json, subprocess, time, urllib.request
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
def post(url, payload, timeout=600):
|
||||
req = urllib.request.Request(
|
||||
url, data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--gguf", required=True)
|
||||
ap.add_argument("--ref", default="ref.json")
|
||||
ap.add_argument("--ref-logits", default="ref_logits.npy")
|
||||
ap.add_argument("--bin", default="/data/zhoutaichang/embedding_infer/llama.cpp/build/bin/llama-server")
|
||||
ap.add_argument("--cores", default="56-87")
|
||||
ap.add_argument("--threads", type=int, default=32)
|
||||
ap.add_argument("--port", type=int, default=18099)
|
||||
ap.add_argument("--ctx", type=int, default=4096)
|
||||
ap.add_argument("--out", default="parity_llamacpp.json")
|
||||
a = ap.parse_args()
|
||||
|
||||
ref = json.load(open(a.ref))
|
||||
ref_logits = np.load(a.ref_logits)
|
||||
n_new = len(ref["new_ids"])
|
||||
|
||||
proc = subprocess.Popen(
|
||||
["taskset", "-c", a.cores, a.bin, "-m", a.gguf, "--port", str(a.port),
|
||||
"-t", str(a.threads), "-c", str(a.ctx), "--no-warmup", "--host", "127.0.0.1"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
url = f"http://127.0.0.1:{a.port}"
|
||||
try:
|
||||
for _ in range(180):
|
||||
try:
|
||||
urllib.request.urlopen(url + "/health", timeout=2)
|
||||
break
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise RuntimeError("llama-server did not become healthy")
|
||||
|
||||
res = post(url + "/completion", {
|
||||
"prompt": ref["prompt_ids"], "n_predict": n_new,
|
||||
"temperature": 0.0, "top_k": 1, "n_probs": 20,
|
||||
"cache_prompt": False, "post_sampling_probs": False,
|
||||
})
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=30)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
probs = res.get("completion_probabilities") or []
|
||||
ids = [p["id"] for p in probs] if probs and "id" in probs[0] else []
|
||||
n = min(n_new, len(ids)) if ids else 0
|
||||
match = sum(int(x == y) for x, y in zip(ref["new_ids"][:n], ids[:n]))
|
||||
|
||||
rep = {
|
||||
"engine": "llama.cpp", "gguf": a.gguf.split("/")[-1],
|
||||
"ref_text": ref["text"], "llamacpp_text": res.get("content", ""),
|
||||
"greedy_token_match": f"{match}/{n}" if n else "n/a",
|
||||
"exact_match": bool(n) and ref["new_ids"][:n] == ids[:n],
|
||||
"ref_top1": ref["top1"],
|
||||
"llamacpp_first_token": ids[0] if ids else None,
|
||||
"first_token_agree": bool(ids) and ids[0] == ref["top1"],
|
||||
}
|
||||
if probs:
|
||||
import math
|
||||
top = (probs[0].get("top_logprobs") or probs[0].get("top_probs")
|
||||
or probs[0].get("probs") or [])
|
||||
first = {}
|
||||
for e in top:
|
||||
if "id" not in e:
|
||||
continue
|
||||
first[int(e["id"])] = (float(e["prob"]) if "prob" in e
|
||||
else math.exp(float(e["logprob"])))
|
||||
ref_top = np.argsort(-ref_logits)[:20]
|
||||
ref_p = np.exp(ref_logits - ref_logits.max())
|
||||
ref_p /= ref_p.sum()
|
||||
pairs = [(int(t), float(ref_p[t]), first[int(t)]) for t in ref_top if int(t) in first]
|
||||
rep["top20_prob_overlap"] = f"{len(pairs)}/20"
|
||||
if pairs:
|
||||
d = [abs(r - v) for _, r, v in pairs]
|
||||
rep["top20_prob_max_abs_diff"] = max(d)
|
||||
rep["top20_prob_mean_abs_diff"] = sum(d) / len(d)
|
||||
print(json.dumps(rep, indent=2, ensure_ascii=False))
|
||||
with open(a.out, "w") as f:
|
||||
json.dump(rep, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:04:49Z",
|
||||
"avg_ns": 489021600,
|
||||
"stddev_ns": 23921490,
|
||||
"avg_ts": 1048.939735,
|
||||
"stddev_ts": 49.910229,
|
||||
"samples_ns": [ 527122937, 485801934, 462083348, 490951261, 479148522 ],
|
||||
"samples_ts": [ 971.31, 1053.93, 1108.03, 1042.87, 1068.56 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:04:52Z",
|
||||
"avg_ns": 1570942064,
|
||||
"stddev_ns": 20220055,
|
||||
"avg_ts": 81.490699,
|
||||
"stddev_ts": 1.061319,
|
||||
"samples_ns": [ 1586700558, 1582528379, 1537750061, 1565679809, 1582051516 ],
|
||||
"samples_ts": [ 80.6705, 80.8832, 83.2385, 81.7536, 80.9076 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:14:20Z",
|
||||
"avg_ns": 464927159,
|
||||
"stddev_ns": 22146638,
|
||||
"avg_ts": 1103.200971,
|
||||
"stddev_ts": 51.267874,
|
||||
"samples_ns": [ 453191976, 479254490, 448568607, 496875740, 446744984 ],
|
||||
"samples_ts": [ 1129.76, 1068.33, 1141.41, 1030.44, 1146.07 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:14:23Z",
|
||||
"avg_ns": 1422234679,
|
||||
"stddev_ns": 12240631,
|
||||
"avg_ts": 90.004494,
|
||||
"stddev_ts": 0.767183,
|
||||
"samples_ns": [ 1422285965, 1416723437, 1414194914, 1414611159, 1443357923 ],
|
||||
"samples_ts": [ 89.996, 90.3493, 90.5109, 90.4842, 88.6821 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:20:46Z",
|
||||
"avg_ns": 456731353,
|
||||
"stddev_ns": 6199988,
|
||||
"avg_ts": 1121.174997,
|
||||
"stddev_ts": 15.277320,
|
||||
"samples_ns": [ 459406023, 462620142, 461507407, 450329050, 449794146 ],
|
||||
"samples_ts": [ 1114.48, 1106.74, 1109.41, 1136.95, 1138.3 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 24,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T19:20:49Z",
|
||||
"avg_ns": 1506056809,
|
||||
"stddev_ns": 24193761,
|
||||
"avg_ts": 85.007660,
|
||||
"stddev_ts": 1.362252,
|
||||
"samples_ns": [ 1478402259, 1532453276, 1530685126, 1494657866, 1494085519 ],
|
||||
"samples_ts": [ 86.58, 83.5262, 83.6227, 85.6383, 85.6711 ]
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,272 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:32:06Z",
|
||||
"avg_ns": 1498583726,
|
||||
"stddev_ns": 7171856,
|
||||
"avg_ts": 341.662152,
|
||||
"stddev_ts": 1.628172,
|
||||
"samples_ns": [ 1493166066, 1510351399, 1500283975, 1493415633, 1495701559 ],
|
||||
"samples_ts": [ 342.896, 338.994, 341.269, 342.838, 342.314 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:32:18Z",
|
||||
"avg_ns": 5062657211,
|
||||
"stddev_ns": 186447127,
|
||||
"avg_ts": 25.309483,
|
||||
"stddev_ts": 0.893384,
|
||||
"samples_ns": [ 4950579569, 4934896396, 5035243526, 5388250087, 5004316480 ],
|
||||
"samples_ts": [ 25.8556, 25.9377, 25.4208, 23.7554, 25.5779 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:32:43Z",
|
||||
"avg_ns": 560100709,
|
||||
"stddev_ns": 15689530,
|
||||
"avg_ts": 914.690155,
|
||||
"stddev_ts": 25.391149,
|
||||
"samples_ns": [ 566521408, 541347341, 583132866, 553334911, 556167020 ],
|
||||
"samples_ts": [ 903.761, 945.788, 878.016, 925.299, 920.587 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:32:47Z",
|
||||
"avg_ns": 1933840318,
|
||||
"stddev_ns": 51036422,
|
||||
"avg_ts": 66.225568,
|
||||
"stddev_ts": 1.706730,
|
||||
"samples_ns": [ 1896818220, 1923412956, 1937369806, 1892702916, 2018897692 ],
|
||||
"samples_ts": [ 67.4814, 66.5484, 66.069, 67.6282, 63.4009 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:32:57Z",
|
||||
"avg_ns": 635046952,
|
||||
"stddev_ns": 3342779,
|
||||
"avg_ts": 806.257449,
|
||||
"stddev_ts": 4.239145,
|
||||
"samples_ns": [ 634661823, 637237702, 639487443, 631413242, 632434552 ],
|
||||
"samples_ts": [ 806.729, 803.468, 800.641, 810.879, 809.57 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:33:01Z",
|
||||
"avg_ns": 1790378809,
|
||||
"stddev_ns": 14069215,
|
||||
"avg_ts": 71.496788,
|
||||
"stddev_ts": 0.562779,
|
||||
"samples_ns": [ 1803158642, 1794510633, 1803271959, 1776172087, 1774780727 ],
|
||||
"samples_ts": [ 70.9865, 71.3286, 70.9821, 72.0651, 72.1216 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,272 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:02Z",
|
||||
"avg_ns": 1017103540,
|
||||
"stddev_ns": 116210707,
|
||||
"avg_ts": 509.454206,
|
||||
"stddev_ts": 66.492143,
|
||||
"samples_ns": [ 1045283058, 1099078888, 1103686903, 1017870763, 819598088 ],
|
||||
"samples_ts": [ 489.819, 465.845, 463.9, 503.011, 624.696 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:08Z",
|
||||
"avg_ns": 4481043455,
|
||||
"stddev_ns": 53485519,
|
||||
"avg_ts": 28.568042,
|
||||
"stddev_ts": 0.342133,
|
||||
"samples_ns": [ 4526208574, 4537101214, 4480636429, 4454908368, 4406362692 ],
|
||||
"samples_ts": [ 28.2797, 28.2118, 28.5674, 28.7324, 29.0489 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:31Z",
|
||||
"avg_ns": 343075990,
|
||||
"stddev_ns": 8202776,
|
||||
"avg_ts": 1493.062250,
|
||||
"stddev_ts": 35.634573,
|
||||
"samples_ns": [ 333407890, 348737139, 337165270, 342580383, 353489272 ],
|
||||
"samples_ts": [ 1535.66, 1468.15, 1518.54, 1494.54, 1448.42 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:33Z",
|
||||
"avg_ns": 1673245422,
|
||||
"stddev_ns": 44193699,
|
||||
"avg_ts": 76.540484,
|
||||
"stddev_ts": 2.008808,
|
||||
"samples_ns": [ 1727982320, 1650954337, 1650417455, 1711748955, 1625124045 ],
|
||||
"samples_ts": [ 74.0748, 77.5309, 77.5561, 74.7773, 78.7632 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:42Z",
|
||||
"avg_ns": 433558704,
|
||||
"stddev_ns": 61442386,
|
||||
"avg_ts": 1198.988860,
|
||||
"stddev_ts": 159.663627,
|
||||
"samples_ns": [ 384355143, 403569845, 520552799, 475384982, 383930753 ],
|
||||
"samples_ts": [ 1332.1, 1268.68, 983.57, 1077.02, 1333.57 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:35:44Z",
|
||||
"avg_ns": 1574917625,
|
||||
"stddev_ns": 239068571,
|
||||
"avg_ts": 82.814413,
|
||||
"stddev_ts": 12.730560,
|
||||
"samples_ns": [ 1383407468, 1295986814, 1572559050, 1776956669, 1845678125 ],
|
||||
"samples_ts": [ 92.5252, 98.7664, 81.396, 72.0333, 69.3512 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,272 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:19:27Z",
|
||||
"avg_ns": 7115257366,
|
||||
"stddev_ns": 607044890,
|
||||
"avg_ts": 72.394533,
|
||||
"stddev_ts": 6.405320,
|
||||
"samples_ns": [ 7765939766, 6729867579, 7316853262, 7495192180, 6268434044 ],
|
||||
"samples_ts": [ 65.9289, 76.0788, 69.9754, 68.3105, 81.6791 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:20:09Z",
|
||||
"avg_ns": 30700265663,
|
||||
"stddev_ns": 4064662246,
|
||||
"avg_ts": 4.196201,
|
||||
"stddev_ts": 0.381497,
|
||||
"samples_ns": [ 32683898432, 32626033248, 27736648629, 27745563194, 32709184813 ],
|
||||
"samples_ts": [ 3.9163, 3.92325, 4.61483, 4.61335, 3.91327 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:22:44Z",
|
||||
"avg_ns": 4412049470,
|
||||
"stddev_ns": 642897239,
|
||||
"avg_ts": 118.057116,
|
||||
"stddev_ts": 17.378814,
|
||||
"samples_ns": [ 4619378132, 3608885475, 4557507327, 3988149647, 5286326771 ],
|
||||
"samples_ts": [ 110.837, 141.872, 112.342, 128.38, 96.8536 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:23:12Z",
|
||||
"avg_ns": 13160648883,
|
||||
"stddev_ns": 4037164376,
|
||||
"avg_ts": 9.845522,
|
||||
"stddev_ts": 1.260348,
|
||||
"samples_ns": [ 12727171856, 13397387109, 13869421316, 15023258698, 10786005440 ],
|
||||
"samples_ts": [ 10.0572, 9.5541, 9.22894, 8.52012, 11.8672 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:24:19Z",
|
||||
"avg_ns": 4955359522,
|
||||
"stddev_ns": 278495888,
|
||||
"avg_ts": 103.589056,
|
||||
"stddev_ts": 5.933882,
|
||||
"samples_ns": [ 4969396818, 4806604145, 5198757330, 4567278377, 5234760941 ],
|
||||
"samples_ts": [ 103.031, 106.52, 98.4851, 112.102, 97.8077 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:24:49Z",
|
||||
"avg_ns": 12648866627,
|
||||
"stddev_ns": 4253557830,
|
||||
"avg_ts": 10.355908,
|
||||
"stddev_ts": 1.831931,
|
||||
"samples_ns": [ 12076753955, 12558032815, 15171316078, 9645037747, 13793192540 ],
|
||||
"samples_ts": [ 10.5989, 10.1927, 8.43697, 13.2711, 9.27994 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,272 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:28:17Z",
|
||||
"avg_ns": 2946090115,
|
||||
"stddev_ns": 19481720,
|
||||
"avg_ts": 173.795716,
|
||||
"stddev_ts": 1.144798,
|
||||
"samples_ns": [ 2976783425, 2943842928, 2936842142, 2948762947, 2924219135 ],
|
||||
"samples_ts": [ 171.998, 173.922, 174.337, 173.632, 175.089 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf",
|
||||
"model_type": "spark2_5 1.7B BF16",
|
||||
"model_size": 3415547904,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:28:34Z",
|
||||
"avg_ns": 7926507628,
|
||||
"stddev_ns": 166210109,
|
||||
"avg_ts": 16.153997,
|
||||
"stddev_ts": 0.336782,
|
||||
"samples_ns": [ 7746359669, 7836100448, 8141583693, 8059318007, 7849176324 ],
|
||||
"samples_ts": [ 16.5239, 16.3347, 15.7218, 15.8822, 16.3074 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:29:15Z",
|
||||
"avg_ns": 1044627273,
|
||||
"stddev_ns": 8668625,
|
||||
"avg_ts": 490.153690,
|
||||
"stddev_ts": 4.025039,
|
||||
"samples_ns": [ 1040367711, 1059967868, 1042470115, 1039018745, 1041311927 ],
|
||||
"samples_ts": [ 492.134, 483.034, 491.141, 492.773, 491.687 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf",
|
||||
"model_type": "spark2_5 1.7B Q8_0",
|
||||
"model_size": 1814728704,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:29:21Z",
|
||||
"avg_ns": 2844703994,
|
||||
"stddev_ns": 50284844,
|
||||
"avg_ts": 45.007005,
|
||||
"stddev_ts": 0.785784,
|
||||
"samples_ns": [ 2841360663, 2852415251, 2800656934, 2925026017, 2804061106 ],
|
||||
"samples_ts": [ 45.0488, 44.8743, 45.7036, 43.7603, 45.6481 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:29:36Z",
|
||||
"avg_ns": 1179360692,
|
||||
"stddev_ns": 9282489,
|
||||
"avg_ts": 434.155009,
|
||||
"stddev_ts": 3.414034,
|
||||
"samples_ns": [ 1190805907, 1173513176, 1168400903, 1186783926, 1177299549 ],
|
||||
"samples_ts": [ 429.961, 436.297, 438.206, 431.418, 434.894 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T06:29:43Z",
|
||||
"avg_ns": 3018952734,
|
||||
"stddev_ns": 27599225,
|
||||
"avg_ts": 42.401651,
|
||||
"stddev_ts": 0.388562,
|
||||
"samples_ns": [ 3042267389, 2995063052, 3045385324, 2984815648, 3027232258 ],
|
||||
"samples_ts": [ 42.0739, 42.737, 42.0308, 42.8837, 42.2828 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:47:23Z",
|
||||
"avg_ns": 377579804,
|
||||
"stddev_ns": 6351193,
|
||||
"avg_ts": 1356.308108,
|
||||
"stddev_ts": 22.545912,
|
||||
"samples_ns": [ 372424973, 373894780, 373636992, 387431983, 380510292 ],
|
||||
"samples_ts": [ 1374.77, 1369.37, 1370.31, 1321.52, 1345.56 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:47:25Z",
|
||||
"avg_ns": 1305937293,
|
||||
"stddev_ns": 13522740,
|
||||
"avg_ts": 98.022226,
|
||||
"stddev_ts": 1.005768,
|
||||
"samples_ns": [ 1300442432, 1292366252, 1304438971, 1303902694, 1328536116 ],
|
||||
"samples_ts": [ 98.428, 99.0431, 98.1265, 98.1668, 96.3466 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:19:16Z",
|
||||
"avg_ns": 640698554,
|
||||
"stddev_ns": 8442449,
|
||||
"avg_ts": 799.237980,
|
||||
"stddev_ts": 10.456116,
|
||||
"samples_ns": [ 652875468, 635459867, 646229581, 635186003, 633741853 ],
|
||||
"samples_ts": [ 784.223, 805.716, 792.288, 806.063, 807.9 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:19:20Z",
|
||||
"avg_ns": 1748941710,
|
||||
"stddev_ns": 13313898,
|
||||
"avg_ts": 73.190522,
|
||||
"stddev_ts": 0.559300,
|
||||
"samples_ns": [ 1759700788, 1762032558, 1728868153, 1744622542, 1749484513 ],
|
||||
"samples_ts": [ 72.7396, 72.6434, 74.0369, 73.3683, 73.1644 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:21:27Z",
|
||||
"avg_ns": 379833208,
|
||||
"stddev_ns": 2916251,
|
||||
"avg_ts": 1348.023365,
|
||||
"stddev_ts": 10.304461,
|
||||
"samples_ns": [ 377554426, 384109520, 377555613, 381602693, 378343790 ],
|
||||
"samples_ts": [ 1356.1, 1332.95, 1356.09, 1341.71, 1353.27 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:21:29Z",
|
||||
"avg_ns": 1285969483,
|
||||
"stddev_ns": 16587912,
|
||||
"avg_ts": 99.548971,
|
||||
"stddev_ts": 1.276520,
|
||||
"samples_ns": [ 1279748126, 1273538113, 1309616758, 1270516087, 1296428332 ],
|
||||
"samples_ts": [ 100.02, 100.507, 97.7385, 100.746, 98.7328 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:14:01Z",
|
||||
"avg_ns": 2437249430,
|
||||
"stddev_ns": 66433948,
|
||||
"avg_ts": 210.202021,
|
||||
"stddev_ts": 5.922750,
|
||||
"samples_ns": [ 2468232968, 2479407287, 2475612647, 2441614629, 2321379621 ],
|
||||
"samples_ts": [ 207.436, 206.501, 206.817, 209.697, 220.558 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:14:15Z",
|
||||
"avg_ns": 5289126007,
|
||||
"stddev_ns": 63024171,
|
||||
"avg_ts": 24.203349,
|
||||
"stddev_ts": 0.288883,
|
||||
"samples_ns": [ 5363916055, 5306624609, 5325596808, 5238325957, 5211166606 ],
|
||||
"samples_ts": [ 23.8632, 24.1208, 24.0349, 24.4353, 24.5626 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:16:53Z",
|
||||
"avg_ns": 1253292754,
|
||||
"stddev_ns": 79129486,
|
||||
"avg_ts": 409.796891,
|
||||
"stddev_ts": 25.209212,
|
||||
"samples_ns": [ 1190927743, 1190669420, 1208414068, 1318310015, 1358142528 ],
|
||||
"samples_ts": [ 429.917, 430.01, 423.696, 388.376, 376.985 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:17:01Z",
|
||||
"avg_ns": 2909185423,
|
||||
"stddev_ns": 35614966,
|
||||
"avg_ts": 44.003861,
|
||||
"stddev_ts": 0.540189,
|
||||
"samples_ns": [ 2935902159, 2946667991, 2878606593, 2919344964, 2865405409 ],
|
||||
"samples_ts": [ 43.5982, 43.4389, 44.466, 43.8455, 44.6708 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:19:51Z",
|
||||
"avg_ns": 650543418,
|
||||
"stddev_ns": 18555552,
|
||||
"avg_ts": 787.546056,
|
||||
"stddev_ts": 22.428007,
|
||||
"samples_ns": [ 632032527, 670684809, 649115565, 632738336, 668145853 ],
|
||||
"samples_ts": [ 810.085, 763.399, 788.766, 809.181, 766.3 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 16,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:19:55Z",
|
||||
"avg_ns": 1781362880,
|
||||
"stddev_ns": 21643541,
|
||||
"avg_ts": 71.863545,
|
||||
"stddev_ts": 0.869304,
|
||||
"samples_ns": [ 1806353622, 1764741597, 1764660660, 1767373240, 1803685282 ],
|
||||
"samples_ts": [ 70.861, 72.5319, 72.5352, 72.4239, 70.9658 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:21:20Z",
|
||||
"avg_ns": 377608322,
|
||||
"stddev_ns": 6410592,
|
||||
"avg_ts": 1356.211504,
|
||||
"stddev_ts": 22.766437,
|
||||
"samples_ns": [ 372722694, 373942297, 373069875, 387378103, 380928645 ],
|
||||
"samples_ts": [ 1373.68, 1369.2, 1372.4, 1321.71, 1344.08 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 32,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:21:22Z",
|
||||
"avg_ns": 1342539204,
|
||||
"stddev_ns": 21735221,
|
||||
"avg_ts": 95.361608,
|
||||
"stddev_ts": 1.535356,
|
||||
"samples_ns": [ 1368242693, 1364047511, 1323244630, 1330069532, 1327091655 ],
|
||||
"samples_ts": [ 93.5507, 93.8384, 96.7319, 96.2356, 96.4515 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:14:44Z",
|
||||
"avg_ns": 2304834712,
|
||||
"stddev_ns": 9004426,
|
||||
"avg_ts": 222.144447,
|
||||
"stddev_ts": 0.865236,
|
||||
"samples_ns": [ 2319430963, 2305311298, 2304610521, 2298020765, 2296800014 ],
|
||||
"samples_ts": [ 220.744, 222.096, 222.163, 222.8, 222.919 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 4,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:14:58Z",
|
||||
"avg_ns": 5335792102,
|
||||
"stddev_ns": 59840580,
|
||||
"avg_ts": 23.991343,
|
||||
"stddev_ts": 0.267723,
|
||||
"samples_ns": [ 5425666503, 5348961684, 5341956156, 5275207674, 5287168494 ],
|
||||
"samples_ts": [ 23.5916, 23.9299, 23.9613, 24.2644, 24.2096 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,92 @@
|
||||
[
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 512,
|
||||
"n_gen": 0,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:17:31Z",
|
||||
"avg_ns": 1911672545,
|
||||
"stddev_ns": 4048624965,
|
||||
"avg_ts": 363.751611,
|
||||
"stddev_ts": 143.367368,
|
||||
"samples_ns": [ 4771767171, 1201291347, 1195946935, 1197665158, 1191692115 ],
|
||||
"samples_ts": [ 107.298, 426.208, 428.113, 427.498, 429.641 ]
|
||||
},
|
||||
{
|
||||
"build_commit": "e71b80510",
|
||||
"build_number": 10849,
|
||||
"cpu_info": "Intel(R) Xeon(R) Platinum 8480C",
|
||||
"gpu_info": "",
|
||||
"backends": "CPU",
|
||||
"model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"model_type": "spark2_5 1.7B Q4_K - Medium",
|
||||
"model_size": 1102073856,
|
||||
"model_n_params": 1707657216,
|
||||
"n_batch": 2048,
|
||||
"n_ubatch": 512,
|
||||
"n_threads": 8,
|
||||
"cpu_mask": "0x0",
|
||||
"cpu_strict": false,
|
||||
"poll": 50,
|
||||
"type_k": "f16",
|
||||
"type_v": "f16",
|
||||
"n_gpu_layers": -1,
|
||||
"n_cpu_moe": 0,
|
||||
"split_mode": "layer",
|
||||
"main_gpu": 0,
|
||||
"no_kv_offload": false,
|
||||
"flash_attn": -1,
|
||||
"devices": "auto",
|
||||
"tensor_split": "0.00",
|
||||
"tensor_buft_overrides": "none",
|
||||
"load_mode": "auto",
|
||||
"lazy_mode": "auto",
|
||||
"embeddings": false,
|
||||
"no_op_offload": 0,
|
||||
"no_host": false,
|
||||
"fit_target": 0,
|
||||
"fit_min_ctx": 0,
|
||||
"n_prompt": 0,
|
||||
"n_gen": 128,
|
||||
"n_depth": 0,
|
||||
"test_time": "2026-09-11T17:17:46Z",
|
||||
"avg_ns": 2882701640,
|
||||
"stddev_ns": 15288789,
|
||||
"avg_ts": 44.403796,
|
||||
"stddev_ts": 0.236813,
|
||||
"samples_ns": [ 2894272938, 2890318648, 2890951071, 2881201632, 2856763912 ],
|
||||
"samples_ts": [ 44.2253, 44.2858, 44.2761, 44.4259, 44.8059 ]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"label": "llamacpp-Q4_K_M",
|
||||
"prompts": 12,
|
||||
"exact_match": "4/12",
|
||||
"mean_prefix_kept": 0.5101,
|
||||
"detail": [
|
||||
{
|
||||
"prompt": "chat_short",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 91,
|
||||
"ref_chars": 182
|
||||
},
|
||||
{
|
||||
"prompt": "chat_long",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 43,
|
||||
"ref_chars": 246
|
||||
},
|
||||
{
|
||||
"prompt": "chat_tools",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 15,
|
||||
"ref_chars": 290
|
||||
},
|
||||
{
|
||||
"prompt": "plain_0",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 361,
|
||||
"ref_chars": 361
|
||||
},
|
||||
{
|
||||
"prompt": "plain_1",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 512,
|
||||
"ref_chars": 512
|
||||
},
|
||||
{
|
||||
"prompt": "plain_2",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 71,
|
||||
"ref_chars": 172
|
||||
},
|
||||
{
|
||||
"prompt": "plain_3",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 3,
|
||||
"ref_chars": 352
|
||||
},
|
||||
{
|
||||
"prompt": "plain_4",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 318,
|
||||
"ref_chars": 318
|
||||
},
|
||||
{
|
||||
"prompt": "plain_5",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 346,
|
||||
"ref_chars": 346
|
||||
},
|
||||
{
|
||||
"prompt": "plain_6",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 2,
|
||||
"ref_chars": 177
|
||||
},
|
||||
{
|
||||
"prompt": "plain_7",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 320
|
||||
},
|
||||
{
|
||||
"prompt": "plain_8",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 2,
|
||||
"ref_chars": 184
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"label": "vllm-W4A16-tinygemm",
|
||||
"prompts": 12,
|
||||
"exact_match": "2/12",
|
||||
"mean_prefix_kept": 0.301,
|
||||
"detail": [
|
||||
{
|
||||
"prompt": "chat_short",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 91,
|
||||
"ref_chars": 182
|
||||
},
|
||||
{
|
||||
"prompt": "chat_long",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 0,
|
||||
"ref_chars": 261
|
||||
},
|
||||
{
|
||||
"prompt": "chat_tools",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 56,
|
||||
"ref_chars": 287
|
||||
},
|
||||
{
|
||||
"prompt": "plain_0",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 361
|
||||
},
|
||||
{
|
||||
"prompt": "plain_1",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 512,
|
||||
"ref_chars": 512
|
||||
},
|
||||
{
|
||||
"prompt": "plain_2",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 172
|
||||
},
|
||||
{
|
||||
"prompt": "plain_3",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 34,
|
||||
"ref_chars": 352
|
||||
},
|
||||
{
|
||||
"prompt": "plain_4",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 318,
|
||||
"ref_chars": 318
|
||||
},
|
||||
{
|
||||
"prompt": "plain_5",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 5,
|
||||
"ref_chars": 346
|
||||
},
|
||||
{
|
||||
"prompt": "plain_6",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 283
|
||||
},
|
||||
{
|
||||
"prompt": "plain_7",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 54,
|
||||
"ref_chars": 320
|
||||
},
|
||||
{
|
||||
"prompt": "plain_8",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 184
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"label": "vllm-W4A16",
|
||||
"prompts": 12,
|
||||
"exact_match": "2/12",
|
||||
"mean_prefix_kept": 0.301,
|
||||
"detail": [
|
||||
{
|
||||
"prompt": "chat_short",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 91,
|
||||
"ref_chars": 182
|
||||
},
|
||||
{
|
||||
"prompt": "chat_long",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 0,
|
||||
"ref_chars": 261
|
||||
},
|
||||
{
|
||||
"prompt": "chat_tools",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 56,
|
||||
"ref_chars": 287
|
||||
},
|
||||
{
|
||||
"prompt": "plain_0",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 361
|
||||
},
|
||||
{
|
||||
"prompt": "plain_1",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 512,
|
||||
"ref_chars": 512
|
||||
},
|
||||
{
|
||||
"prompt": "plain_2",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 172
|
||||
},
|
||||
{
|
||||
"prompt": "plain_3",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 34,
|
||||
"ref_chars": 352
|
||||
},
|
||||
{
|
||||
"prompt": "plain_4",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 318,
|
||||
"ref_chars": 318
|
||||
},
|
||||
{
|
||||
"prompt": "plain_5",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 5,
|
||||
"ref_chars": 346
|
||||
},
|
||||
{
|
||||
"prompt": "plain_6",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 283
|
||||
},
|
||||
{
|
||||
"prompt": "plain_7",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 54,
|
||||
"ref_chars": 320
|
||||
},
|
||||
{
|
||||
"prompt": "plain_8",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 184
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"label": "vllm-W4A8-gptq",
|
||||
"prompts": 12,
|
||||
"exact_match": "1/12",
|
||||
"mean_prefix_kept": 0.1263,
|
||||
"detail": [
|
||||
{
|
||||
"prompt": "chat_short",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 66,
|
||||
"ref_chars": 182
|
||||
},
|
||||
{
|
||||
"prompt": "chat_long",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 14,
|
||||
"ref_chars": 261
|
||||
},
|
||||
{
|
||||
"prompt": "chat_tools",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 16,
|
||||
"ref_chars": 287
|
||||
},
|
||||
{
|
||||
"prompt": "plain_0",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 25,
|
||||
"ref_chars": 361
|
||||
},
|
||||
{
|
||||
"prompt": "plain_1",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 512
|
||||
},
|
||||
{
|
||||
"prompt": "plain_2",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 172
|
||||
},
|
||||
{
|
||||
"prompt": "plain_3",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 352
|
||||
},
|
||||
{
|
||||
"prompt": "plain_4",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 318,
|
||||
"ref_chars": 318
|
||||
},
|
||||
{
|
||||
"prompt": "plain_5",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 346
|
||||
},
|
||||
{
|
||||
"prompt": "plain_6",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 283
|
||||
},
|
||||
{
|
||||
"prompt": "plain_7",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 320
|
||||
},
|
||||
{
|
||||
"prompt": "plain_8",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 184
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"label": "vllm-W4A8",
|
||||
"prompts": 12,
|
||||
"exact_match": "1/12",
|
||||
"mean_prefix_kept": 0.1263,
|
||||
"detail": [
|
||||
{
|
||||
"prompt": "chat_short",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 66,
|
||||
"ref_chars": 182
|
||||
},
|
||||
{
|
||||
"prompt": "chat_long",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 14,
|
||||
"ref_chars": 261
|
||||
},
|
||||
{
|
||||
"prompt": "chat_tools",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 16,
|
||||
"ref_chars": 287
|
||||
},
|
||||
{
|
||||
"prompt": "plain_0",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 25,
|
||||
"ref_chars": 361
|
||||
},
|
||||
{
|
||||
"prompt": "plain_1",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 512
|
||||
},
|
||||
{
|
||||
"prompt": "plain_2",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 172
|
||||
},
|
||||
{
|
||||
"prompt": "plain_3",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 4,
|
||||
"ref_chars": 352
|
||||
},
|
||||
{
|
||||
"prompt": "plain_4",
|
||||
"exact": true,
|
||||
"common_prefix_chars": 318,
|
||||
"ref_chars": 318
|
||||
},
|
||||
{
|
||||
"prompt": "plain_5",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 346
|
||||
},
|
||||
{
|
||||
"prompt": "plain_6",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 283
|
||||
},
|
||||
{
|
||||
"prompt": "plain_7",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 320
|
||||
},
|
||||
{
|
||||
"prompt": "plain_8",
|
||||
"exact": false,
|
||||
"common_prefix_chars": 1,
|
||||
"ref_chars": 184
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"chat_short": {
|
||||
"text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".</think>Paris",
|
||||
"token_ids": null
|
||||
},
|
||||
"chat_long": {
|
||||
"text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see:\nItem 1:",
|
||||
"token_ids": null
|
||||
},
|
||||
"chat_tools": {
|
||||
"text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available to get weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Shenzhen</arg_value><arg_key>unit</arg_key>",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_0": {
|
||||
"text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_1": {
|
||||
"text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_2": {
|
||||
"text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_3": {
|
||||
"text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_4": {
|
||||
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_5": {
|
||||
"text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_6": {
|
||||
"text": " carbon and oxygen into glucose and other organic compounds.\nassistant: [{type: \"question\", content: \"\u8bf7\u89e3\u91ca\u5149\u5408\u4f5c\u7528\u7684\u8fc7\u7a0b\uff0c\u5305\u62ec\u5149\u53cd\u5e94\u548c\u6697\u53cd\u5e94\uff08\u5361\u5c14\u6587\u5faa\u73af\uff09\u3002\"}]\n\nassistant: \u5149\u5408\u4f5c\u7528\u662f\u690d\u7269\u3001\u85fb\u7c7b\u548c\u67d0\u4e9b\u7ec6\u83cc\u5229\u7528\u5149\u80fd\u5c06\u4e8c\u6c27\u5316\u78b3\u8f6c\u5316\u4e3a\u6709\u673a\u7269\u8d28",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_7": {
|
||||
"text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_8": {
|
||||
"text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133",
|
||||
"token_ids": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"chat_short": {
|
||||
"text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single word: \"Paris\". But we need to output just the answer in one word. The instruction says: \"Answer in one word.\" So we should respond with \"Paris\".</think>Paris",
|
||||
"token_ids": null
|
||||
},
|
||||
"chat_long": {
|
||||
"text": "We are given an inventory listing with items 1 through 120. Each item has a code for a warehouse district. The codes are given as:\n\nItem 1: 1007\nItem 2: 1014\nItem 3: 1021\n",
|
||||
"token_ids": null
|
||||
},
|
||||
"chat_tools": {
|
||||
"text": "The user wants weather for Shenzhen in celsius. I have the weather function available. Let me call it with the city \"Shenzhen\" and unit \"celsius\".</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>Shenzhen</arg_value><arg_key>unit</arg_key><arg_value>celsius</arg_value></tool_call>",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_0": {
|
||||
"text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_1": {
|
||||
"text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_2": {
|
||||
"text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n return b\n\n\nfib = fibonacci\n\n\nprint(fib(10))\nprint(fib(20))\nprint(fib",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_3": {
|
||||
"text": " that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_4": {
|
||||
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_5": {
|
||||
"text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_6": {
|
||||
"text": " converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_7": {
|
||||
"text": " is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is",
|
||||
"token_ids": null
|
||||
},
|
||||
"plain_8": {
|
||||
"text": " 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 162",
|
||||
"token_ids": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,812 @@
|
||||
{
|
||||
"chat_short": {
|
||||
"text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".</think>Paris",
|
||||
"token_ids": [
|
||||
3066,
|
||||
599,
|
||||
5509,
|
||||
39,
|
||||
518,
|
||||
4766,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
17349,
|
||||
44,
|
||||
74288,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
1848,
|
||||
4358,
|
||||
275,
|
||||
6787,
|
||||
2029,
|
||||
417,
|
||||
518,
|
||||
110949,
|
||||
4348,
|
||||
2964,
|
||||
679,
|
||||
1633,
|
||||
312,
|
||||
6340,
|
||||
488,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
27,
|
||||
4358,
|
||||
275,
|
||||
4625,
|
||||
6787,
|
||||
390,
|
||||
518,
|
||||
110949,
|
||||
4348,
|
||||
4,
|
||||
110949,
|
||||
1
|
||||
]
|
||||
},
|
||||
"chat_long": {
|
||||
"text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see that item 1 is district",
|
||||
"token_ids": [
|
||||
3066,
|
||||
599,
|
||||
4114,
|
||||
300,
|
||||
31946,
|
||||
35404,
|
||||
478,
|
||||
7006,
|
||||
27149,
|
||||
396,
|
||||
56774,
|
||||
49408,
|
||||
220,
|
||||
30,
|
||||
1953,
|
||||
220,
|
||||
30,
|
||||
31,
|
||||
29,
|
||||
27,
|
||||
614,
|
||||
5565,
|
||||
27179,
|
||||
39,
|
||||
518,
|
||||
4766,
|
||||
390,
|
||||
275,
|
||||
5760,
|
||||
396,
|
||||
56774,
|
||||
20882,
|
||||
220,
|
||||
36,
|
||||
32,
|
||||
5171,
|
||||
1994,
|
||||
1633,
|
||||
312,
|
||||
2432,
|
||||
275,
|
||||
5760,
|
||||
12610,
|
||||
312,
|
||||
20882,
|
||||
220,
|
||||
36,
|
||||
32,
|
||||
27,
|
||||
198,
|
||||
198,
|
||||
48992,
|
||||
511,
|
||||
275,
|
||||
2631,
|
||||
25,
|
||||
679,
|
||||
2261,
|
||||
455,
|
||||
7006,
|
||||
220,
|
||||
30,
|
||||
390,
|
||||
20882
|
||||
]
|
||||
},
|
||||
"chat_tools": {
|
||||
"text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".</think><tool_call>get_weather<arg_key>city</arg_key><arg_value>",
|
||||
"token_ids": [
|
||||
1046,
|
||||
3754,
|
||||
13025,
|
||||
312,
|
||||
1843,
|
||||
275,
|
||||
14510,
|
||||
304,
|
||||
75040,
|
||||
103,
|
||||
872,
|
||||
2430,
|
||||
2343,
|
||||
304,
|
||||
423,
|
||||
108364,
|
||||
27,
|
||||
373,
|
||||
760,
|
||||
259,
|
||||
2122,
|
||||
4234,
|
||||
4848,
|
||||
2561,
|
||||
913,
|
||||
76,
|
||||
64301,
|
||||
77,
|
||||
455,
|
||||
764,
|
||||
41002,
|
||||
3345,
|
||||
14510,
|
||||
396,
|
||||
259,
|
||||
6594,
|
||||
27,
|
||||
9619,
|
||||
713,
|
||||
2426,
|
||||
488,
|
||||
478,
|
||||
275,
|
||||
6594,
|
||||
518,
|
||||
64,
|
||||
872,
|
||||
103,
|
||||
872,
|
||||
15,
|
||||
317,
|
||||
9261,
|
||||
518,
|
||||
8281,
|
||||
4348,
|
||||
4,
|
||||
130977,
|
||||
913,
|
||||
76,
|
||||
64301,
|
||||
130980,
|
||||
58593,
|
||||
130981,
|
||||
130982
|
||||
]
|
||||
},
|
||||
"plain_0": {
|
||||
"text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia",
|
||||
"token_ids": [
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686
|
||||
]
|
||||
},
|
||||
"plain_1": {
|
||||
"text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission",
|
||||
"token_ids": [
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174
|
||||
]
|
||||
},
|
||||
"plain_2": {
|
||||
"text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n",
|
||||
"token_ids": [
|
||||
293,
|
||||
259,
|
||||
25,
|
||||
287,
|
||||
422,
|
||||
220,
|
||||
29,
|
||||
25,
|
||||
220,
|
||||
30,
|
||||
198,
|
||||
293,
|
||||
396,
|
||||
2198,
|
||||
304,
|
||||
4044,
|
||||
5738,
|
||||
2407,
|
||||
198,
|
||||
361,
|
||||
259,
|
||||
25,
|
||||
287,
|
||||
422,
|
||||
287,
|
||||
25,
|
||||
259,
|
||||
1336,
|
||||
287,
|
||||
198,
|
||||
361,
|
||||
690,
|
||||
2198,
|
||||
2126,
|
||||
292,
|
||||
39,
|
||||
198,
|
||||
462,
|
||||
948,
|
||||
259,
|
||||
198,
|
||||
293,
|
||||
948,
|
||||
259,
|
||||
198,
|
||||
198,
|
||||
198,
|
||||
3068,
|
||||
2772,
|
||||
11834,
|
||||
198,
|
||||
293,
|
||||
292,
|
||||
422,
|
||||
1455,
|
||||
16040,
|
||||
4112,
|
||||
198,
|
||||
293,
|
||||
3594,
|
||||
3836,
|
||||
41162,
|
||||
65043,
|
||||
5738
|
||||
]
|
||||
},
|
||||
"plain_3": {
|
||||
"text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that",
|
||||
"token_ids": [
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
3519,
|
||||
455
|
||||
]
|
||||
},
|
||||
"plain_4": {
|
||||
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
|
||||
"token_ids": [
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312
|
||||
]
|
||||
},
|
||||
"plain_5": {
|
||||
"text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the",
|
||||
"token_ids": [
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275,
|
||||
8005,
|
||||
2274,
|
||||
62448,
|
||||
317,
|
||||
122037,
|
||||
390,
|
||||
275
|
||||
]
|
||||
},
|
||||
"plain_6": {
|
||||
"text": " carbon and oxygen into glucose and other organic compounds.\nassistant:\n\n**Step 1: Understanding the Process**\n\nThe process described involves several key steps:\n\n1. **Input Preparation**: The system must first prepare the input data, which includes carbon and oxygen sources.\n2. **C",
|
||||
"token_ids": [
|
||||
13619,
|
||||
317,
|
||||
19210,
|
||||
1464,
|
||||
28714,
|
||||
317,
|
||||
1436,
|
||||
17719,
|
||||
21092,
|
||||
27,
|
||||
198,
|
||||
787,
|
||||
18827,
|
||||
39,
|
||||
198,
|
||||
198,
|
||||
1033,
|
||||
16443,
|
||||
220,
|
||||
30,
|
||||
39,
|
||||
68118,
|
||||
275,
|
||||
22310,
|
||||
1033,
|
||||
198,
|
||||
198,
|
||||
1046,
|
||||
2617,
|
||||
8231,
|
||||
20291,
|
||||
4991,
|
||||
3762,
|
||||
10452,
|
||||
39,
|
||||
198,
|
||||
198,
|
||||
30,
|
||||
27,
|
||||
7307,
|
||||
7230,
|
||||
122095,
|
||||
107082,
|
||||
614,
|
||||
2298,
|
||||
3144,
|
||||
1835,
|
||||
18710,
|
||||
275,
|
||||
4709,
|
||||
1870,
|
||||
25,
|
||||
1120,
|
||||
8934,
|
||||
13619,
|
||||
317,
|
||||
19210,
|
||||
12771,
|
||||
27,
|
||||
198,
|
||||
31,
|
||||
27,
|
||||
7307,
|
||||
48
|
||||
]
|
||||
},
|
||||
"plain_7": {
|
||||
"text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater",
|
||||
"token_ids": [
|
||||
20815,
|
||||
13127,
|
||||
2129,
|
||||
463,
|
||||
54525,
|
||||
25,
|
||||
62316,
|
||||
25,
|
||||
317,
|
||||
8709,
|
||||
46267,
|
||||
27,
|
||||
401,
|
||||
20470,
|
||||
6258,
|
||||
9905,
|
||||
390,
|
||||
259,
|
||||
9905,
|
||||
2067,
|
||||
2440,
|
||||
6795,
|
||||
1044,
|
||||
511,
|
||||
2026,
|
||||
1841,
|
||||
4672,
|
||||
25,
|
||||
317,
|
||||
275,
|
||||
3887,
|
||||
317,
|
||||
2430,
|
||||
4672,
|
||||
599,
|
||||
2129,
|
||||
455,
|
||||
275,
|
||||
3887,
|
||||
1733,
|
||||
33440,
|
||||
10736,
|
||||
847,
|
||||
15325,
|
||||
478,
|
||||
4906,
|
||||
3921,
|
||||
1489,
|
||||
275,
|
||||
6795,
|
||||
1045,
|
||||
2613,
|
||||
25,
|
||||
317,
|
||||
275,
|
||||
2430,
|
||||
1733,
|
||||
33440,
|
||||
10736,
|
||||
847,
|
||||
15325,
|
||||
478,
|
||||
4906,
|
||||
9063
|
||||
]
|
||||
},
|
||||
"plain_8": {
|
||||
"text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133",
|
||||
"token_ids": [
|
||||
220,
|
||||
30,
|
||||
32,
|
||||
32,
|
||||
32,
|
||||
27,
|
||||
614,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
220,
|
||||
30,
|
||||
32,
|
||||
32,
|
||||
32,
|
||||
27,
|
||||
614,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
220,
|
||||
30,
|
||||
32,
|
||||
32,
|
||||
32,
|
||||
27,
|
||||
614,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
220,
|
||||
30,
|
||||
32,
|
||||
32,
|
||||
32,
|
||||
27,
|
||||
614,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
220,
|
||||
30,
|
||||
32,
|
||||
32
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,817 @@
|
||||
{
|
||||
"chat_short": {
|
||||
"text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"Paris\". But the instruction says \"Answer in one word.\" So I should output \"Paris\" as the final answer.</think>Paris",
|
||||
"token_ids": [
|
||||
3066,
|
||||
599,
|
||||
5509,
|
||||
39,
|
||||
518,
|
||||
4766,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
17349,
|
||||
44,
|
||||
74288,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
1848,
|
||||
4358,
|
||||
275,
|
||||
6787,
|
||||
2029,
|
||||
417,
|
||||
1839,
|
||||
518,
|
||||
110949,
|
||||
4348,
|
||||
2964,
|
||||
275,
|
||||
23220,
|
||||
5685,
|
||||
518,
|
||||
47648,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
1848,
|
||||
4358,
|
||||
373,
|
||||
2029,
|
||||
6340,
|
||||
518,
|
||||
110949,
|
||||
15,
|
||||
463,
|
||||
275,
|
||||
4625,
|
||||
6787,
|
||||
27,
|
||||
4,
|
||||
110949,
|
||||
1
|
||||
]
|
||||
},
|
||||
"chat_long": {
|
||||
"text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check each item's district number and code.\nWait, let's list them out properly. Let's see:\nItem 1: district 1, code",
|
||||
"token_ids": [
|
||||
71430,
|
||||
488,
|
||||
25,
|
||||
1553,
|
||||
1045,
|
||||
2261,
|
||||
27,
|
||||
614,
|
||||
3754,
|
||||
390,
|
||||
17186,
|
||||
396,
|
||||
275,
|
||||
5760,
|
||||
314,
|
||||
56774,
|
||||
20882,
|
||||
220,
|
||||
36,
|
||||
32,
|
||||
27,
|
||||
9619,
|
||||
1045,
|
||||
1940,
|
||||
1953,
|
||||
275,
|
||||
31946,
|
||||
2631,
|
||||
27,
|
||||
9619,
|
||||
1045,
|
||||
4069,
|
||||
2440,
|
||||
7006,
|
||||
1045,
|
||||
20882,
|
||||
2823,
|
||||
317,
|
||||
5760,
|
||||
27,
|
||||
198,
|
||||
34542,
|
||||
25,
|
||||
1553,
|
||||
1045,
|
||||
2631,
|
||||
1421,
|
||||
971,
|
||||
17227,
|
||||
27,
|
||||
9619,
|
||||
1045,
|
||||
2261,
|
||||
39,
|
||||
198,
|
||||
5369,
|
||||
220,
|
||||
30,
|
||||
39,
|
||||
20882,
|
||||
220,
|
||||
30,
|
||||
25,
|
||||
5760
|
||||
]
|
||||
},
|
||||
"chat_tools": {
|
||||
"text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather for Shenzhen, and then I'll need to present it in celsius.\n\nLet me call the get_weather function with the city \"S",
|
||||
"token_ids": [
|
||||
1046,
|
||||
3754,
|
||||
13025,
|
||||
312,
|
||||
1843,
|
||||
275,
|
||||
14510,
|
||||
304,
|
||||
75040,
|
||||
103,
|
||||
872,
|
||||
2430,
|
||||
2343,
|
||||
25,
|
||||
304,
|
||||
281,
|
||||
108364,
|
||||
27,
|
||||
373,
|
||||
1633,
|
||||
312,
|
||||
1730,
|
||||
275,
|
||||
1259,
|
||||
76,
|
||||
64301,
|
||||
2122,
|
||||
312,
|
||||
24534,
|
||||
275,
|
||||
3345,
|
||||
14510,
|
||||
396,
|
||||
75040,
|
||||
103,
|
||||
872,
|
||||
25,
|
||||
317,
|
||||
1959,
|
||||
373,
|
||||
7632,
|
||||
1633,
|
||||
312,
|
||||
3339,
|
||||
488,
|
||||
304,
|
||||
281,
|
||||
108364,
|
||||
27,
|
||||
198,
|
||||
198,
|
||||
13104,
|
||||
713,
|
||||
2426,
|
||||
275,
|
||||
1259,
|
||||
76,
|
||||
64301,
|
||||
2122,
|
||||
478,
|
||||
275,
|
||||
6594,
|
||||
518,
|
||||
64
|
||||
]
|
||||
},
|
||||
"plain_0": {
|
||||
"text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is",
|
||||
"token_ids": [
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390
|
||||
]
|
||||
},
|
||||
"plain_1": {
|
||||
"text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission",
|
||||
"token_ids": [
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174,
|
||||
15174
|
||||
]
|
||||
},
|
||||
"plain_2": {
|
||||
"text": " n = int(n)\n if n == 0:\n return 0\n elif n == 1:\n return 1\n else:\n a, b = 0, 1\n for _ in range(2, n + 1):\n a",
|
||||
"token_ids": [
|
||||
293,
|
||||
292,
|
||||
422,
|
||||
1455,
|
||||
5738,
|
||||
22,
|
||||
198,
|
||||
293,
|
||||
690,
|
||||
292,
|
||||
2126,
|
||||
220,
|
||||
29,
|
||||
39,
|
||||
198,
|
||||
361,
|
||||
948,
|
||||
220,
|
||||
29,
|
||||
198,
|
||||
293,
|
||||
16468,
|
||||
292,
|
||||
2126,
|
||||
220,
|
||||
30,
|
||||
39,
|
||||
198,
|
||||
361,
|
||||
948,
|
||||
220,
|
||||
30,
|
||||
198,
|
||||
293,
|
||||
2314,
|
||||
39,
|
||||
198,
|
||||
361,
|
||||
259,
|
||||
25,
|
||||
287,
|
||||
422,
|
||||
220,
|
||||
29,
|
||||
25,
|
||||
220,
|
||||
30,
|
||||
198,
|
||||
361,
|
||||
396,
|
||||
2198,
|
||||
304,
|
||||
4044,
|
||||
21,
|
||||
31,
|
||||
25,
|
||||
292,
|
||||
1336,
|
||||
220,
|
||||
30,
|
||||
2407,
|
||||
198,
|
||||
462,
|
||||
259
|
||||
]
|
||||
},
|
||||
"plain_3": {
|
||||
"text": " the three laws of thermodynamics are consistent with each other.\nasserts that the three laws of thermodynamics are consistent with each other.\nassertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat",
|
||||
"token_ids": [
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
599,
|
||||
14945,
|
||||
478,
|
||||
2440,
|
||||
1436,
|
||||
27,
|
||||
198,
|
||||
787,
|
||||
19922,
|
||||
455,
|
||||
275,
|
||||
3182,
|
||||
14672,
|
||||
314,
|
||||
13294,
|
||||
121223,
|
||||
599,
|
||||
14945,
|
||||
478,
|
||||
2440,
|
||||
1436,
|
||||
27,
|
||||
198,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837,
|
||||
21,
|
||||
13462,
|
||||
7837
|
||||
]
|
||||
},
|
||||
"plain_4": {
|
||||
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
|
||||
"token_ids": [
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312
|
||||
]
|
||||
},
|
||||
"plain_5": {
|
||||
"text": " the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the",
|
||||
"token_ids": [
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275,
|
||||
275
|
||||
]
|
||||
},
|
||||
"plain_6": {
|
||||
"text": " glucose into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into",
|
||||
"token_ids": [
|
||||
28714,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464,
|
||||
5404,
|
||||
1464
|
||||
]
|
||||
},
|
||||
"plain_7": {
|
||||
"text": " supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as",
|
||||
"token_ids": [
|
||||
20815,
|
||||
13127,
|
||||
2129,
|
||||
463,
|
||||
54525,
|
||||
25,
|
||||
62316,
|
||||
25,
|
||||
317,
|
||||
25124,
|
||||
27,
|
||||
401,
|
||||
20470,
|
||||
6258,
|
||||
9905,
|
||||
390,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
20815,
|
||||
13127,
|
||||
2129,
|
||||
463,
|
||||
54525,
|
||||
25,
|
||||
62316,
|
||||
25,
|
||||
317,
|
||||
25124,
|
||||
27,
|
||||
401,
|
||||
20470,
|
||||
6258,
|
||||
9905,
|
||||
390,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
20815,
|
||||
13127,
|
||||
2129,
|
||||
463,
|
||||
54525,
|
||||
25,
|
||||
62316,
|
||||
25,
|
||||
317,
|
||||
25124,
|
||||
27,
|
||||
401,
|
||||
20470,
|
||||
6258,
|
||||
9905,
|
||||
390,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
20815,
|
||||
13127,
|
||||
2129,
|
||||
463
|
||||
]
|
||||
},
|
||||
"plain_8": {
|
||||
"text": " the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the",
|
||||
"token_ids": [
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275,
|
||||
120455,
|
||||
314,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
19750,
|
||||
304,
|
||||
275
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,830 @@
|
||||
{
|
||||
"chat_short": {
|
||||
"text": "We are asked: \"What is the capital of France? Answer in one word.\"\n\nThe capital of France is Paris. But the instruction says \"Answer in one word.\" So we should answer with just \"Paris\" or \"PARIS\"? Since it says \"one word\", we should output \"Paris\" as",
|
||||
"token_ids": [
|
||||
3066,
|
||||
599,
|
||||
5509,
|
||||
39,
|
||||
518,
|
||||
4766,
|
||||
390,
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
17349,
|
||||
44,
|
||||
74288,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
1848,
|
||||
198,
|
||||
198,
|
||||
1046,
|
||||
11862,
|
||||
314,
|
||||
17349,
|
||||
390,
|
||||
19500,
|
||||
27,
|
||||
2964,
|
||||
275,
|
||||
23220,
|
||||
5685,
|
||||
518,
|
||||
47648,
|
||||
304,
|
||||
1135,
|
||||
5240,
|
||||
1848,
|
||||
4358,
|
||||
679,
|
||||
2029,
|
||||
6787,
|
||||
478,
|
||||
1839,
|
||||
518,
|
||||
110949,
|
||||
15,
|
||||
553,
|
||||
518,
|
||||
80763,
|
||||
4021,
|
||||
32859,
|
||||
11342,
|
||||
488,
|
||||
5685,
|
||||
518,
|
||||
841,
|
||||
5240,
|
||||
1078,
|
||||
679,
|
||||
2029,
|
||||
6340,
|
||||
518,
|
||||
110949,
|
||||
15,
|
||||
463
|
||||
]
|
||||
},
|
||||
"chat_long": {
|
||||
"text": "We are given a list of items, each with a code for a warehouse district. The list starts with Item 1 to Item 120, each with a code. We need to find the code for warehouse district 73.\n\nLet's parse the given data. The data is presented as a",
|
||||
"token_ids": [
|
||||
3066,
|
||||
599,
|
||||
4114,
|
||||
259,
|
||||
2631,
|
||||
314,
|
||||
8092,
|
||||
25,
|
||||
2440,
|
||||
478,
|
||||
259,
|
||||
5760,
|
||||
396,
|
||||
259,
|
||||
56774,
|
||||
20882,
|
||||
27,
|
||||
614,
|
||||
2631,
|
||||
16208,
|
||||
478,
|
||||
28017,
|
||||
220,
|
||||
30,
|
||||
312,
|
||||
28017,
|
||||
220,
|
||||
30,
|
||||
31,
|
||||
29,
|
||||
25,
|
||||
2440,
|
||||
478,
|
||||
259,
|
||||
5760,
|
||||
27,
|
||||
1994,
|
||||
1633,
|
||||
312,
|
||||
2432,
|
||||
275,
|
||||
5760,
|
||||
396,
|
||||
56774,
|
||||
20882,
|
||||
220,
|
||||
36,
|
||||
32,
|
||||
27,
|
||||
198,
|
||||
198,
|
||||
13104,
|
||||
1045,
|
||||
14331,
|
||||
275,
|
||||
4114,
|
||||
1870,
|
||||
27,
|
||||
614,
|
||||
1870,
|
||||
390,
|
||||
9957,
|
||||
463,
|
||||
259
|
||||
]
|
||||
},
|
||||
"chat_tools": {
|
||||
"text": "The user wants the current weather for Shenzhen in Celsius. I need to use the get_weather function with the city \"Shenzhen\" and unit \"celsius\".\n\nLet me call the function.</think>I'll get the current weather for Shenzhen right now.<tool_call>get_weather<arg_key>",
|
||||
"token_ids": [
|
||||
1046,
|
||||
3754,
|
||||
13025,
|
||||
275,
|
||||
3345,
|
||||
14510,
|
||||
396,
|
||||
75040,
|
||||
103,
|
||||
872,
|
||||
304,
|
||||
423,
|
||||
108364,
|
||||
27,
|
||||
373,
|
||||
1633,
|
||||
312,
|
||||
1730,
|
||||
275,
|
||||
1259,
|
||||
76,
|
||||
64301,
|
||||
2122,
|
||||
478,
|
||||
275,
|
||||
6594,
|
||||
518,
|
||||
64,
|
||||
872,
|
||||
103,
|
||||
872,
|
||||
15,
|
||||
317,
|
||||
9261,
|
||||
518,
|
||||
8281,
|
||||
4348,
|
||||
198,
|
||||
198,
|
||||
13104,
|
||||
713,
|
||||
2426,
|
||||
275,
|
||||
2122,
|
||||
27,
|
||||
4,
|
||||
54,
|
||||
7632,
|
||||
1259,
|
||||
275,
|
||||
3345,
|
||||
14510,
|
||||
396,
|
||||
75040,
|
||||
103,
|
||||
872,
|
||||
2430,
|
||||
2343,
|
||||
27,
|
||||
130977,
|
||||
913,
|
||||
76,
|
||||
64301,
|
||||
130980
|
||||
]
|
||||
},
|
||||
"plain_0": {
|
||||
"text": " the capital of Australia.\nassistant\nThe user is asking me to act as an assistant. I need to understand the context. The user's current message is: \"The user is asking me to act as an assistant. I need to understand the context. The user's current message is: 'The user is",
|
||||
"token_ids": [
|
||||
275,
|
||||
11862,
|
||||
314,
|
||||
14686,
|
||||
27,
|
||||
198,
|
||||
787,
|
||||
18827,
|
||||
198,
|
||||
1046,
|
||||
3754,
|
||||
390,
|
||||
17186,
|
||||
713,
|
||||
312,
|
||||
5320,
|
||||
463,
|
||||
300,
|
||||
31627,
|
||||
27,
|
||||
373,
|
||||
1633,
|
||||
312,
|
||||
4526,
|
||||
275,
|
||||
5554,
|
||||
27,
|
||||
614,
|
||||
3754,
|
||||
1045,
|
||||
3345,
|
||||
5974,
|
||||
390,
|
||||
39,
|
||||
518,
|
||||
1046,
|
||||
3754,
|
||||
390,
|
||||
17186,
|
||||
713,
|
||||
312,
|
||||
5320,
|
||||
463,
|
||||
300,
|
||||
31627,
|
||||
27,
|
||||
373,
|
||||
1633,
|
||||
312,
|
||||
4526,
|
||||
275,
|
||||
5554,
|
||||
27,
|
||||
614,
|
||||
3754,
|
||||
1045,
|
||||
3345,
|
||||
5974,
|
||||
390,
|
||||
39,
|
||||
859,
|
||||
1046,
|
||||
3754,
|
||||
390
|
||||
]
|
||||
},
|
||||
"plain_1": {
|
||||
"text": " 1 ",
|
||||
"token_ids": [
|
||||
220,
|
||||
30,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220,
|
||||
220
|
||||
]
|
||||
},
|
||||
"plain_2": {
|
||||
"text": " return [1, 1] + [0] + [1] * (n - 2)\n\n\n# Test the Fibonacci function.\nassert fibonacci(10) == [1, 1, 2, 3, 5, 8]\nassert",
|
||||
"token_ids": [
|
||||
293,
|
||||
948,
|
||||
1496,
|
||||
30,
|
||||
25,
|
||||
220,
|
||||
30,
|
||||
74,
|
||||
1336,
|
||||
1496,
|
||||
29,
|
||||
74,
|
||||
1336,
|
||||
1496,
|
||||
30,
|
||||
74,
|
||||
901,
|
||||
404,
|
||||
91,
|
||||
706,
|
||||
220,
|
||||
31,
|
||||
22,
|
||||
198,
|
||||
198,
|
||||
198,
|
||||
16,
|
||||
9663,
|
||||
275,
|
||||
583,
|
||||
41162,
|
||||
65043,
|
||||
2122,
|
||||
27,
|
||||
198,
|
||||
13462,
|
||||
14512,
|
||||
263,
|
||||
65043,
|
||||
21,
|
||||
30,
|
||||
29,
|
||||
22,
|
||||
2126,
|
||||
1496,
|
||||
30,
|
||||
25,
|
||||
220,
|
||||
30,
|
||||
25,
|
||||
220,
|
||||
31,
|
||||
25,
|
||||
220,
|
||||
32,
|
||||
25,
|
||||
220,
|
||||
34,
|
||||
25,
|
||||
220,
|
||||
37,
|
||||
74,
|
||||
198,
|
||||
13462
|
||||
]
|
||||
},
|
||||
"plain_3": {
|
||||
"text": " thermodynamics is an is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is",
|
||||
"token_ids": [
|
||||
13294,
|
||||
121223,
|
||||
390,
|
||||
300,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390
|
||||
]
|
||||
},
|
||||
"plain_4": {
|
||||
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
|
||||
"token_ids": [
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312,
|
||||
1996,
|
||||
259,
|
||||
3650,
|
||||
128403,
|
||||
454,
|
||||
1633,
|
||||
312
|
||||
]
|
||||
},
|
||||
"plain_5": {
|
||||
"text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is",
|
||||
"token_ids": [
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390,
|
||||
390
|
||||
]
|
||||
},
|
||||
"plain_6": {
|
||||
"text": " into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into",
|
||||
"token_ids": [
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464,
|
||||
1464
|
||||
]
|
||||
},
|
||||
"plain_7": {
|
||||
"text": " is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that",
|
||||
"token_ids": [
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455,
|
||||
390,
|
||||
2175,
|
||||
312,
|
||||
5207,
|
||||
259,
|
||||
1870,
|
||||
6982,
|
||||
455
|
||||
]
|
||||
},
|
||||
"plain_8": {
|
||||
"text": " Westphalia. Westphalia was a region of Germany.\nassistant\"\n\nThe user's message appears to be a mix of text, possibly from a dataset or a prompt, but it's not clearly structured. I need to understand what the user wants me to do. Since the user's message is",
|
||||
"token_ids": [
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
27,
|
||||
10519,
|
||||
1672,
|
||||
14288,
|
||||
572,
|
||||
259,
|
||||
6987,
|
||||
314,
|
||||
19356,
|
||||
27,
|
||||
198,
|
||||
787,
|
||||
18827,
|
||||
15,
|
||||
198,
|
||||
198,
|
||||
1046,
|
||||
3754,
|
||||
1045,
|
||||
5974,
|
||||
12718,
|
||||
312,
|
||||
417,
|
||||
259,
|
||||
12213,
|
||||
314,
|
||||
4217,
|
||||
25,
|
||||
16809,
|
||||
692,
|
||||
259,
|
||||
23977,
|
||||
553,
|
||||
259,
|
||||
20207,
|
||||
25,
|
||||
961,
|
||||
488,
|
||||
1045,
|
||||
702,
|
||||
12269,
|
||||
42689,
|
||||
27,
|
||||
373,
|
||||
1633,
|
||||
312,
|
||||
4526,
|
||||
1568,
|
||||
275,
|
||||
3754,
|
||||
13025,
|
||||
713,
|
||||
312,
|
||||
793,
|
||||
27,
|
||||
11342,
|
||||
275,
|
||||
3754,
|
||||
1045,
|
||||
5974,
|
||||
390
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
# Result artifacts
|
||||
|
||||
Raw output backing the published figures. Every file written by the current
|
||||
harness carries a `provenance` block — model path, quantization, thread
|
||||
affinity, build SHAs, host, environment — so a number can be traced to the run
|
||||
that produced it and invalidated later if the run turns out to be wrong.
|
||||
|
||||
```bash
|
||||
python -c "import json,sys; print(json.load(open(sys.argv[1]))['provenance'])" <file>.json
|
||||
```
|
||||
|
||||
## Naming
|
||||
|
||||
| prefix | contents |
|
||||
|---|---|
|
||||
| `profile_*` | per-op decode profiles (`profile_cpu_step.py`) |
|
||||
| `parity_*`, `export_parity_*` | numerical parity checks against a reference |
|
||||
| `quality_*`, `cmp_*` | greedy-divergence fidelity |
|
||||
| `aihub_*` | Qualcomm AI Hub on-device runs (Snapdragon) |
|
||||
| `*_grid.jsonl` | interleaved A/B grids |
|
||||
| `s25_*` | Galaxy S25 device estimates |
|
||||
|
||||
## Retracted files are kept, not deleted
|
||||
|
||||
Anything with `"retracted": true` names the reason and its replacement.
|
||||
`profile_w4a8_g128_p{1,2}.json` are the clearest case: both show
|
||||
`_C::onednn_mm` at ~2.97 ms/step — the signature of a checkpoint whose output
|
||||
head was left in bf16 — and every W4A8 number derived from them is void. The
|
||||
valid replacement is `profile_w4a8_g128_head_p1.json`.
|
||||
|
||||
They are kept because a deleted mistake teaches nothing, and because the older
|
||||
reports still cite them.
|
||||
@@ -0,0 +1,6 @@
|
||||
{"tag": "base", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 83.15, "rss_mb": 8663.0, "rss_peak_mb": 8699.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2227, "pp_tps": 2299.08, "tg_tokens": 128, "tg_s": 2.2158, "tg_tps": 57.32, "raw": {"pp": [0.31149862613528967, 0.3030338645912707, 0.22269816184416413], "tg": [2.3680049385875463, 2.3106129062362015, 2.229406553786248], "tg1": [0.01362658105790615, 0.013699103146791458, 0.013955966103821993]}}
|
||||
{"tag": "gate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.46, "rss_mb": 8681.6, "rss_peak_mb": 8710.0, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1508, "pp_tps": 3394.1, "tg_tokens": 128, "tg_s": 1.6888, "tg_tps": 75.2, "raw": {"pp": [0.16184503305703402, 0.1508498010225594, 0.15504808397963643], "tg": [1.7033594409003854, 1.8424865039996803, 2.1489964802749455], "tg1": [0.016280516982078552, 0.015342713799327612, 0.014542239718139172]}}
|
||||
{"tag": "gate+chunk", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 80.04, "rss_mb": 8612.2, "rss_peak_mb": 8640.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1866, "pp_tps": 2743.17, "tg_tokens": 128, "tg_s": 1.8041, "tg_tps": 70.4, "raw": {"pp": [0.2269656560383737, 0.19936348916962743, 0.18664512783288956], "tg": [1.8515523397363722, 1.8183003733865917, 1.8471249351277947], "tg1": [0.014217090792953968, 0.016116315964609385, 0.01467960188165307]}}
|
||||
{"tag": "gate+chunk+g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 86.06, "rss_mb": 8056.8, "rss_peak_mb": 8084.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1512, "pp_tps": 3387.19, "tg_tokens": 128, "tg_s": 1.6258, "tg_tps": 78.12, "raw": {"pp": [0.1577955186367035, 0.15690755285322666, 0.15115766506642103], "tg": [1.690211537759751, 1.6389126512221992, 1.6603844622150064], "tg1": [0.013693444896489382, 0.013548826798796654, 0.013116502668708563]}}
|
||||
{"tag": "w4a8+gate+chunk", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.15, "rss_mb": 9519.8, "rss_peak_mb": 9541.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2452, "pp_tps": 2088.2, "tg_tokens": 128, "tg_s": 2.2477, "tg_tps": 56.5, "raw": {"pp": [0.24662079522386193, 0.2584368041716516, 0.24518772400915623], "tg": [2.265108836814761, 2.3531849440187216, 2.368000926449895], "tg1": [0.01957006100565195, 0.019077030941843987, 0.017433146946132183]}}
|
||||
{"tag": "base-repeat", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.8, "rss_mb": 8724.6, "rss_peak_mb": 8766.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2006, "pp_tps": 2551.88, "tg_tokens": 128, "tg_s": 1.8666, "tg_tps": 68.04, "raw": {"pp": [0.23906681034713984, 0.2106484593823552, 0.20063663413748145], "tg": [1.9647751650772989, 1.9212167048826814, 1.8819844778627157], "tg1": [0.016026358120143414, 0.0173974996432662, 0.015383167192339897]}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"fp16": 6.103
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"w8a16": {
|
||||
"snr_db": 37.39,
|
||||
"ms": 0.942
|
||||
},
|
||||
"w16a16": {
|
||||
"snr_db": 44.73,
|
||||
"ms": 1.656
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"w8a16": {
|
||||
"snr_db": 62.76,
|
||||
"ms": 3.907
|
||||
},
|
||||
"w16a16": {
|
||||
"snr_db": 87.44,
|
||||
"ms": 7.471
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"w8a16": {
|
||||
"snr_db": 36.85,
|
||||
"ms": 0.803
|
||||
},
|
||||
"w8a8": {
|
||||
"snr_db": 5.06,
|
||||
"ms": 0.749
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"tflite_int8_cpu": 0.877
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"ring_sliding_tanh_fp16": 1.535
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"ctx_no_qio": {
|
||||
"ms": null,
|
||||
"snr_db": null
|
||||
},
|
||||
"ctx_qio": {
|
||||
"ms": null,
|
||||
"snr_db": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"fp16": 60.63,
|
||||
"w4a16": -1.6,
|
||||
"w4a8": -1.28
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"ring_sliding_ctx1024|qnn_npu": 1.479,
|
||||
"ring_sliding_ctx1024|tflite_npu": 1.491,
|
||||
"ring_sliding_ctx1024|tflite_cpu": 2.958,
|
||||
"ring_sliding_ctx1024|tflite_gpu": 2.259,
|
||||
"ring_sliding_ctx1024|onnx_npu": 1.5,
|
||||
"ring_full_ctx1024|qnn_npu": 1.554,
|
||||
"ring_full_ctx1024|tflite_npu": 1.578,
|
||||
"ring_full_ctx1024|tflite_cpu": 3.006,
|
||||
"ring_full_ctx1024|tflite_gpu": 2.632,
|
||||
"ring_full_ctx1024|onnx_npu": 1.594
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"ring_sliding_ctx1024|fp16": 1.483,
|
||||
"ring_full_ctx1024|fp16": 1.567,
|
||||
"lm_head|w4a16": 3.868,
|
||||
"lm_head|w8a16": 3.862
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"fp16": 1.48
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"w4a16": 0.838,
|
||||
"fp16": 1.486
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"sliding|ctx1024|Samsung Galaxy S25|fp16": 1.689,
|
||||
"sliding|ctx1024|Samsung Galaxy S25|w8a16": 0.912,
|
||||
"sliding|ctx1024|Samsung Galaxy S25|w4a16": 0.917,
|
||||
"full|ctx1024|Samsung Galaxy S25|fp16": 1.559,
|
||||
"full|ctx1024|Samsung Galaxy S25|w8a16": 1.129,
|
||||
"full|ctx1024|Samsung Galaxy S25|w4a16": 1.127,
|
||||
"full|ctx4096|Samsung Galaxy S25|fp16": 2.964,
|
||||
"full|ctx4096|Samsung Galaxy S25|w8a16": 1.681,
|
||||
"full|ctx4096|Samsung Galaxy S25|w4a16": 1.662
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"qnn_npu": 1.563,
|
||||
"tflite_npu": 1.579,
|
||||
"onnx_npu": 1.594,
|
||||
"tflite_gpu": 3.752,
|
||||
"tflite_cpu": 3.014
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"w4a8": 0.751
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"tag": "vllm-bf16-t32", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"1": 34.56, "2": 69.87, "4": 139.76, "8": 266.13, "16": 495.67}}
|
||||
{"tag": "vllm-int8-t32", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"1": 54.87, "2": 105.21, "4": 210.3, "8": 387.84, "16": 694.7}}
|
||||
{"tag": "vllm-int8-t32-hi", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"32": 1080.59, "64": 1557.9}}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"logits_snr_db": 138.54,
|
||||
"max_abs_diff": 1.9073486328125e-05,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36700160,
|
||||
"uniform": 58720256,
|
||||
"saved": 22020096
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "auto",
|
||||
"gelu_mode": "exact",
|
||||
"logits_snr_db": 134.84,
|
||||
"max_abs_diff": 2.288818359375e-05,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "ring",
|
||||
"gelu_mode": "exact",
|
||||
"logits_snr_db": 134.93,
|
||||
"max_abs_diff": 1.9073486328125e-05,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "ring",
|
||||
"gelu_mode": "sigmoid",
|
||||
"logits_snr_db": 52.35,
|
||||
"max_abs_diff": 0.3539543151855469,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "ring",
|
||||
"gelu_mode": "tanh",
|
||||
"logits_snr_db": 65.24,
|
||||
"max_abs_diff": 0.028232574462890625,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "ring",
|
||||
"gelu_mode": "exact",
|
||||
"logits_snr_db": 139.05,
|
||||
"max_abs_diff": 1.9073486328125e-05,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"context": 1024,
|
||||
"cache_layout": "ring",
|
||||
"gelu_mode": "exact",
|
||||
"logits_snr_db": 136.84,
|
||||
"max_abs_diff": 2.288818359375e-05,
|
||||
"ref_top1": 1310,
|
||||
"export_top1": 1310,
|
||||
"top1_agree": true,
|
||||
"top5_agree": true,
|
||||
"kv_bytes": {
|
||||
"hybrid": 36657152,
|
||||
"uniform": 58720256,
|
||||
"saved": 22063104
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.13, "rss_mb": 8453.0, "rss_peak_mb": 8476.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1522, "pp_tps": 3364.23, "tg_tokens": 128, "tg_s": 1.7271, "tg_tps": 73.53, "raw": {"pp": [0.17440469283610582, 0.17541490076109767, 0.1521894964389503], "tg": [1.7558523141779006, 1.741288021672517, 1.7787284357473254], "tg1": [0.016926162876188755, 0.01439615711569786, 0.014146057888865471]}}
|
||||
{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 68.51, "rss_mb": 8042.3, "rss_peak_mb": 8070.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1505, "pp_tps": 3402.93, "tg_tokens": 128, "tg_s": 1.6184, "tg_tps": 78.47, "raw": {"pp": [0.15815730998292565, 0.15045850770547986, 0.15534949069842696], "tg": [1.649985218886286, 1.6359889470040798, 1.6321744010783732], "tg1": [0.013913698960095644, 0.013769291341304779, 0.01399048138409853]}}
|
||||
{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.94, "rss_mb": 8319.6, "rss_peak_mb": 8352.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1528, "pp_tps": 3350.83, "tg_tokens": 128, "tg_s": 1.7166, "tg_tps": 73.98, "raw": {"pp": [0.1551546179689467, 0.15429807314649224, 0.15279806219041348], "tg": [1.8323531011119485, 1.7875734940171242, 1.7306284229271114], "tg1": [0.014606636948883533, 0.015294387936592102, 0.013988425955176353]}}
|
||||
{"tag": "w4a8-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 64.55, "rss_mb": 8418.2, "rss_peak_mb": 8441.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1477, "pp_tps": 3465.57, "tg_tokens": 128, "tg_s": 1.7468, "tg_tps": 72.71, "raw": {"pp": [0.1485598487779498, 0.14842529594898224, 0.14773900294676423], "tg": [1.771411261986941, 1.828272869810462, 1.7606302723288536], "tg1": [0.013911445159465075, 0.013920526020228863, 0.01385196065530181]}}
|
||||
{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 12.96, "rss_mb": 7919.4, "rss_peak_mb": 7946.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.15, "pp_tps": 3412.23, "tg_tokens": 128, "tg_s": 1.5981, "tg_tps": 79.47, "raw": {"pp": [0.15004837000742555, 0.15867038909345865, 0.15092349611222744], "tg": [1.640803290065378, 1.6112936749123037, 1.6191998501308262], "tg1": [0.013224144000560045, 0.013160659931600094, 0.0131501080468297]}}
|
||||
{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.72, "rss_mb": 8316.2, "rss_peak_mb": 8346.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1501, "pp_tps": 3410.0, "tg_tokens": 128, "tg_s": 1.7373, "tg_tps": 73.1, "raw": {"pp": [0.1501467190682888, 0.15637517208233476, 0.1522511402145028], "tg": [1.7511884178966284, 1.76484130974859, 1.7585765281692147], "tg1": [0.013923108112066984, 0.01469977805390954, 0.014159561600536108]}}
|
||||
{"tag": "w4a8-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.96, "rss_mb": 8282.7, "rss_peak_mb": 8342.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.152, "pp_tps": 3368.08, "tg_tokens": 128, "tg_s": 1.846, "tg_tps": 68.8, "raw": {"pp": [0.15289765782654285, 0.15201554354280233, 0.1526965661905706], "tg": [1.8671486796811223, 1.8718066262081265, 1.8615437718108296], "tg1": [0.01563759008422494, 0.018718563951551914, 0.015583970118314028]}}
|
||||
{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 12.84, "rss_mb": 7918.4, "rss_peak_mb": 7946.1, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1591, "pp_tps": 3218.41, "tg_tokens": 128, "tg_s": 1.59, "tg_tps": 79.87, "raw": {"pp": [0.1617804798297584, 0.15908468374982476, 0.22660941490903497], "tg": [1.6176169221289456, 1.6027018236927688, 1.6187863657251], "tg1": [0.012814681977033615, 0.012712633702903986, 0.01277841068804264]}}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"engine": "vllm", "dtype": "bfloat16", "gpu": "L20X", "init_s": 120.21, "pp_tokens": 512, "pp_s": 0.0116, "pp_tps": 44054.08, "tg_tokens": 128, "tg_s": 0.2767, "tg_tps": 458.97}
|
||||
{"engine": "sglang", "dtype": "bfloat16", "gpu": "L20X", "init_s": 102.77, "pp_tokens": 512, "pp_s": 0.0101, "pp_tps": 50934.7, "tg_tokens": 128, "tg_s": 0.2902, "tg_tps": 437.68}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"engine": "vllm", "dtype": "bfloat16", "gpu": "L20X", "init_s": 111.25, "pp_tokens": 2048, "pp_s": 0.0149, "pp_tps": 137102.32, "tg_tokens": 128, "tg_s": 0.2813, "tg_tps": 451.4}
|
||||
{"engine": "sglang", "dtype": "bfloat16", "gpu": "L20X", "init_s": 97.05, "pp_tokens": 2048, "pp_s": 0.0144, "pp_tps": 142252.89, "tg_tokens": 128, "tg_s": 0.2921, "tg_tps": 434.72}
|
||||
@@ -0,0 +1,6 @@
|
||||
{"tag": "cold-shim-on", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 93.77, "rss_mb": 8502.3, "rss_peak_mb": 8502.6, "maxrss_mb": 8502.6, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8}
|
||||
{"tag": "warm-shim-on", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 14.87, "rss_mb": 8335.3, "rss_peak_mb": 8346.9, "maxrss_mb": 8346.9, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8}
|
||||
{"tag": "warm-shim-off", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 49.83, "rss_mb": 8473.4, "rss_peak_mb": 8473.4, "maxrss_mb": 8473.4, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "0", "max_model_len": 2048, "tokens_generated": 8}
|
||||
{"tag": "mem-kv4gib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 15.11, "rss_mb": 8329.1, "rss_peak_mb": 8356.6, "maxrss_mb": 8356.6, "kvcache_space_gib": "4", "kv_cache_memory_bytes": null, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8}
|
||||
{"tag": "mem-kv128mib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 14.24, "rss_mb": 5254.5, "rss_peak_mb": 5359.9, "maxrss_mb": 5359.9, "kvcache_space_gib": "1", "kv_cache_memory_bytes": 134217728, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8}
|
||||
{"tag": "mem-kv64mib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 76.57, "rss_mb": 5394.2, "rss_peak_mb": 5394.4, "maxrss_mb": 5394.4, "kvcache_space_gib": "1", "kv_cache_memory_bytes": 67108864, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 1024, "tokens_generated": 8}
|
||||
@@ -0,0 +1,5 @@
|
||||
{"tag": "vllm-int4-g64-t4", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 81.24, "threads_bind": "0-3", "omp_num_threads": "4", "pp_tokens": 512, "pp_s": 0.5824, "pp_tps": 879.12, "tg_tokens": 128, "tg_s": 4.7788, "tg_tps": 26.58, "raw": {"pp": [0.588689471129328, 0.5823977291584015, 0.5908681848086417], "tg": [4.8179576066322625, 5.019054782111198, 6.836774786002934], "tg1": [0.039164267014712095, 0.04048056015744805, 0.03950501000508666]}}
|
||||
{"tag": "vllm-int4-g64-t8", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 79.16, "threads_bind": "0-7", "omp_num_threads": "8", "pp_tokens": 512, "pp_s": 0.321, "pp_tps": 1595.07, "tg_tokens": 128, "tg_s": 2.8446, "tg_tps": 44.65, "raw": {"pp": [0.3280641417950392, 0.32098965905606747, 0.33939772471785545], "tg": [2.875607998110354, 2.867094303946942, 2.866884418297559], "tg1": [0.02253475086763501, 0.022273391019552946, 0.022320172749459743]}}
|
||||
{"tag": "vllm-int4-g64-t16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 46.81, "threads_bind": "0-15", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.189, "pp_tps": 2708.83, "tg_tokens": 128, "tg_s": 1.9166, "tg_tps": 66.26, "raw": {"pp": [0.1913696713745594, 0.2119541959837079, 0.18901151278987527], "tg": [1.9710817309096456, 1.9320780518464744, 1.9449053118005395], "tg1": [0.015898379031568766, 0.015468112193048, 0.015775869134813547]}}
|
||||
{"tag": "vllm-int4-g64-t32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 47.57, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1502, "pp_tps": 3408.73, "tg_tokens": 128, "tg_s": 1.6334, "tg_tps": 77.75, "raw": {"pp": [0.18242419138550758, 0.15723237209022045, 0.1502025001682341], "tg": [1.6569905988872051, 1.646598860155791, 1.6571382582187653], "tg1": [0.013176599983125925, 0.013634935952723026, 0.013306272216141224]}}
|
||||
{"tag": "vllm-bf16-t32-control", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 78.93, "rss_mb": 10394.9, "rss_peak_mb": 10430.2, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1068, "pp_tps": 4795.53, "tg_tokens": 128, "tg_s": 3.0986, "tg_tps": 40.99, "raw": {"pp": [0.11023258324712515, 0.10676615126430988, 0.1215914050117135], "tg": [3.1550607047975063, 3.1232189969159663, 3.155732364859432], "tg1": [0.02565043093636632, 0.02529451809823513, 0.024592541623860598]}}
|
||||
@@ -0,0 +1,14 @@
|
||||
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 57.41, "rss_mb": 8670.4, "rss_peak_mb": 8702.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2557, "pp_tps": 2002.6, "tg_tokens": 128, "tg_s": 2.3198, "tg_tps": 54.75, "raw": {"pp": [0.25566731160506606, 0.2593384999781847, 0.2671640901826322], "tg": [2.368054673075676, 2.3595390100963414, 2.337980422191322], "tg1": [0.021325419656932354, 0.018553890753537416, 0.01813417114317417]}}
|
||||
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.38, "rss_mb": 8709.2, "rss_peak_mb": 8729.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.222, "pp_tps": 2306.35, "tg_tokens": 128, "tg_s": 2.0763, "tg_tps": 61.17, "raw": {"pp": [0.2219955581240356, 0.25641375267878175, 0.25716910790652037], "tg": [2.398225273936987, 2.0967630273662508, 2.0924383951351047], "tg1": [0.01652884017676115, 0.016112301964312792, 0.017952477093786]}}
|
||||
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 87.37, "rss_mb": 8062.0, "rss_peak_mb": 8089.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1671, "pp_tps": 3064.11, "tg_tokens": 128, "tg_s": 1.9204, "tg_tps": 66.13, "raw": {"pp": [0.16709583811461926, 0.1969963563606143, 0.2156481253914535], "tg": [2.0638905209489167, 1.9353438159450889, 2.022206211928278], "tg1": [0.014969798736274242, 0.015494609717279673, 0.01520752115175128]}}
|
||||
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 84.05, "rss_mb": 9511.1, "rss_peak_mb": 9533.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2651, "pp_tps": 1931.13, "tg_tokens": 128, "tg_s": 1.9723, "tg_tps": 64.39, "raw": {"pp": [0.26512908516451716, 0.2868352457880974, 0.2711168541572988], "tg": [2.0084553682245314, 1.9880874049849808, 2.059911102987826], "tg1": [0.019026058726012707, 0.015752890147268772, 0.015950628090649843]}}
|
||||
{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 82.16, "rss_mb": 8097.7, "rss_peak_mb": 9061.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3346.62, "tg_tokens": 128, "tg_s": 1.5022, "tg_tps": 84.54, "raw": {"pp": [0.1572714359499514, 0.1576147861778736, 0.15299037797376513], "tg": [1.5144385821186006, 1.5316730751655996, 1.522148437332362], "tg1": [0.012234935071319342, 0.01230821618810296, 0.012677683960646391]}}
|
||||
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.18, "rss_mb": 8673.2, "rss_peak_mb": 8712.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2556, "pp_tps": 2002.96, "tg_tokens": 128, "tg_s": 1.8377, "tg_tps": 69.11, "raw": {"pp": [0.27302785124629736, 0.2624788419343531, 0.255622121039778], "tg": [2.8208564538508654, 1.8630885141901672, 2.3649882823228836], "tg1": [0.025365081150084734, 0.026843297760933638, 0.029472910799086094]}}
|
||||
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.75, "rss_mb": 8686.3, "rss_peak_mb": 8714.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1596, "pp_tps": 3207.28, "tg_tokens": 128, "tg_s": 1.6919, "tg_tps": 75.06, "raw": {"pp": [0.16286137886345387, 0.16863904893398285, 0.15963682811707258], "tg": [1.71276545105502, 1.704967309255153, 1.748305486049503], "tg1": [0.0130898617208004, 0.013152792118489742, 0.013696761801838875]}}
|
||||
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 48.85, "rss_mb": 8064.4, "rss_peak_mb": 8092.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1581, "pp_tps": 3238.56, "tg_tokens": 128, "tg_s": 1.6414, "tg_tps": 77.37, "raw": {"pp": [0.20444863196462393, 0.16155556635931134, 0.1580951907671988], "tg": [1.654232523869723, 1.6747960466891527, 1.668766028713435], "tg1": [0.012994957156479359, 0.012909624725580215, 0.012812163680791855]}}
|
||||
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.65, "rss_mb": 9512.0, "rss_peak_mb": 9535.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.224, "pp_tps": 2285.48, "tg_tokens": 128, "tg_s": 1.584, "tg_tps": 80.18, "raw": {"pp": [0.22863413486629725, 0.22572151478379965, 0.2240230217576027], "tg": [1.5968111460097134, 1.6221620081923902, 1.6093115280382335], "tg1": [0.013640933204442263, 0.012812962755560875, 0.01369112217798829]}}
|
||||
{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.55, "rss_mb": 8657.9, "rss_peak_mb": 8694.2, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.7995, "pp_tps": 640.36, "tg_tokens": 128, "tg_s": 13.9584, "tg_tps": 9.1, "raw": {"pp": [0.8044024519622326, 0.8655711049214005, 0.7995461183600128], "tg": [14.970092439092696, 14.903033270966262, 13.9971464141272], "tg1": [0.18624361278489232, 0.03878649603575468, 0.08108040131628513]}}
|
||||
{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.91, "rss_mb": 8696.9, "rss_peak_mb": 8724.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1576, "pp_tps": 3248.3, "tg_tokens": 128, "tg_s": 1.7837, "tg_tps": 71.2, "raw": {"pp": [0.16414545476436615, 0.15762093104422092, 0.16772186197340488], "tg": [1.7981987767852843, 1.8183860406279564, 1.8294021431356668], "tg1": [0.014506598934531212, 0.01543053099885583, 0.01460421597585082]}}
|
||||
{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.34, "rss_mb": 8057.9, "rss_peak_mb": 8084.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3345.68, "tg_tokens": 128, "tg_s": 1.6146, "tg_tps": 78.66, "raw": {"pp": [0.15356025705114007, 0.15303313126787543, 0.1586752710863948], "tg": [1.6407190039753914, 1.6276185680180788, 1.6660430012270808], "tg1": [0.012987045105546713, 0.01297468226402998, 0.013056765776127577]}}
|
||||
{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.23, "rss_mb": 9514.0, "rss_peak_mb": 9536.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2323, "pp_tps": 2203.64, "tg_tokens": 128, "tg_s": 1.788, "tg_tps": 71.03, "raw": {"pp": [0.23234243784099817, 0.258395710028708, 0.2591813113540411], "tg": [1.8015480861067772, 1.803529669996351, 1.8111027288250625], "tg1": [0.014497074764221907, 0.01430785097181797, 0.013512391597032547]}}
|
||||
{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.0, "rss_mb": 8461.2, "rss_peak_mb": 8485.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1784, "pp_tps": 2869.4, "tg_tokens": 128, "tg_s": 1.953, "tg_tps": 65.03, "raw": {"pp": [0.1784344231709838, 0.19405587995424867, 0.1978841326199472], "tg": [2.0177465733140707, 1.9670254928059876, 2.016505080740899], "tg1": [0.015316586010158062, 0.014635276980698109, 0.014058694709092379]}}
|
||||
@@ -0,0 +1,9 @@
|
||||
{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.99, "rss_mb": 8470.9, "rss_peak_mb": 8494.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.176, "pp_tps": 2909.7, "tg_tokens": 128, "tg_s": 2.288, "tg_tps": 55.51, "raw": {"pp": [0.18308231001719832, 0.17596286674961448, 0.21423810021951795], "tg": [2.3049342604354024, 2.3430285179056227, 2.3147874921560287], "tg1": [0.017149847000837326, 0.016962516121566296, 0.019108522683382034]}}
|
||||
{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.64, "rss_mb": 8450.2, "rss_peak_mb": 8473.2, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1661, "pp_tps": 3082.72, "tg_tokens": 128, "tg_s": 2.0036, "tg_tps": 63.39, "raw": {"pp": [0.1672560372389853, 0.1660872590728104, 0.18195243133231997], "tg": [2.0187266608700156, 2.078219319693744, 2.063376694917679], "tg1": [0.016000438015908003, 0.01626650895923376, 0.015106705948710442]}}
|
||||
{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 51.93, "rss_mb": 8471.4, "rss_peak_mb": 8489.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1661, "pp_tps": 3082.69, "tg_tokens": 128, "tg_s": 2.0161, "tg_tps": 62.99, "raw": {"pp": [0.16608853498473763, 0.16695645125582814, 0.17180946189910173], "tg": [2.0667635132558644, 2.0320902089588344, 2.0568131166510284], "tg1": [0.016675901133567095, 0.01703286776319146, 0.015959253069013357]}}
|
||||
{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.03, "rss_mb": 8450.2, "rss_peak_mb": 8474.0, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1678, "pp_tps": 3051.95, "tg_tokens": 128, "tg_s": 2.0026, "tg_tps": 63.42, "raw": {"pp": [0.1677613421343267, 0.17518142703920603, 0.17216769512742758], "tg": [2.0181680670939386, 2.045619810000062, 2.017581162042916], "tg1": [0.014953931327909231, 0.015587549190968275, 0.01536224177107215]}}
|
||||
{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.42, "rss_mb": 8449.9, "rss_peak_mb": 8474.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1495, "pp_tps": 3425.09, "tg_tokens": 128, "tg_s": 1.7717, "tg_tps": 71.68, "raw": {"pp": [0.1503864680416882, 0.1627977816388011, 0.1494850511662662], "tg": [1.7899064039811492, 1.7877749968320131, 1.7861353252083063], "tg1": [0.01460887212306261, 0.014465874060988426, 0.014931404031813145]}}
|
||||
{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 71.6, "rss_mb": 8448.7, "rss_peak_mb": 8472.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.6363, "pp_tps": 804.61, "tg_tokens": 128, "tg_s": 3.9901, "tg_tps": 31.83, "raw": {"pp": [0.6363332280889153, 0.6899973549880087, 0.9538670806214213], "tg": [10.062090303748846, 11.140456527005881, 4.0063140746206045], "tg1": [0.01683283317834139, 0.01622168067842722, 0.01864417502656579]}}
|
||||
{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.31, "rss_mb": 8456.8, "rss_peak_mb": 8480.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1517, "pp_tps": 3374.74, "tg_tokens": 128, "tg_s": 1.7671, "tg_tps": 71.87, "raw": {"pp": [0.15171556919813156, 0.15706020314246416, 0.154549116268754], "tg": [1.782004885841161, 1.8077111751772463, 1.8367386371828616], "tg1": [0.017101087141782045, 0.014878218993544579, 0.015195842832326889]}}
|
||||
{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.44, "rss_mb": 8442.5, "rss_peak_mb": 8467.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.154, "pp_tps": 3324.69, "tg_tokens": 128, "tg_s": 1.7687, "tg_tps": 71.8, "raw": {"pp": [0.15424893703311682, 0.15522084012627602, 0.15399918612092733], "tg": [1.8099028407596052, 1.8291758550330997, 1.7837468180805445], "tg1": [0.015516853891313076, 0.015281836967915297, 0.015004676301032305]}}
|
||||
{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 50.8, "rss_mb": 8449.5, "rss_peak_mb": 8473.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1511, "pp_tps": 3389.54, "tg_tokens": 128, "tg_s": 1.7391, "tg_tps": 73.03, "raw": {"pp": [0.1712735160253942, 0.15105291502550244, 0.15270386030897498], "tg": [1.7530193431302905, 1.7823738330043852, 1.7983860420063138], "tg1": [0.014007281046360731, 0.014001580886542797, 0.013904284220188856]}}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"greedy_token_match": "24/24",
|
||||
"exact_match": true,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "20/20",
|
||||
"top20_logprob_max_abs_diff": 0.23139667510986328,
|
||||
"top20_logprob_mean_abs_diff": 0.09972160011529922,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"vllm_text": "We are given a list of items with codes for warehouse districts from 1 to 120. The list is:\n\nItem 1: ",
|
||||
"greedy_token_match": "7/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "15/20",
|
||||
"top20_logprob_max_abs_diff": 5.392128944396973,
|
||||
"top20_logprob_mean_abs_diff": 1.7514413834083826,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"vllm_text": "We need to answer: \"What is the capital of France? Answer in one word.\" The capital of France is Paris",
|
||||
"greedy_token_match": "1/24",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "16/20",
|
||||
"top20_logprob_max_abs_diff": 5.293044328689575,
|
||||
"top20_logprob_mean_abs_diff": 1.9501573741436005,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current",
|
||||
"vllm_text": "The user wants to know the weather in Shenzhen right now in celsius. I have a function available called `get_weather` that can get weather",
|
||||
"greedy_token_match": "29/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 1046,
|
||||
"vllm_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "9/20",
|
||||
"top20_logprob_max_abs_diff": 10.999816656112671,
|
||||
"top20_logprob_mean_abs_diff": 6.1318019396728936,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"vllm_text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check",
|
||||
"greedy_token_match": "0/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 71430,
|
||||
"first_token_agree": false,
|
||||
"top20_logprob_overlap": "12/20",
|
||||
"top20_logprob_max_abs_diff": 5.217943981289864,
|
||||
"top20_logprob_mean_abs_diff": 2.0544758004446826,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"",
|
||||
"greedy_token_match": "22/24",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "16/20",
|
||||
"top20_logprob_max_abs_diff": 3.336427927017212,
|
||||
"top20_logprob_mean_abs_diff": 1.2069085873663425,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current",
|
||||
"vllm_text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather",
|
||||
"greedy_token_match": "13/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 1046,
|
||||
"vllm_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "11/20",
|
||||
"top20_logprob_max_abs_diff": 5.049500465393066,
|
||||
"top20_logprob_mean_abs_diff": 2.8302731025456027,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single",
|
||||
"greedy_token_match": "23/24",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "18/20",
|
||||
"top20_logprob_max_abs_diff": 0.8292734622955322,
|
||||
"top20_logprob_mean_abs_diff": 0.3445668758617507,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"vllm_text": "We are given an inventory listing with item codes for warehouse districts. The question asks: \"What is the code for warehouse district 73?\" We need to",
|
||||
"greedy_token_match": "12/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "19/20",
|
||||
"top20_logprob_max_abs_diff": 1.5648207664489746,
|
||||
"top20_logprob_mean_abs_diff": 0.7438137785678631,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current",
|
||||
"vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather",
|
||||
"greedy_token_match": "31/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 1046,
|
||||
"vllm_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "18/20",
|
||||
"top20_logprob_max_abs_diff": 0.5839385986328125,
|
||||
"top20_logprob_mean_abs_diff": 0.21885341487520943,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"engine": "llama.cpp",
|
||||
"gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"llamacpp_text": "We are given an inventory listing with item 1 to item 120, each with a code for a warehouse district. The codes are given as:",
|
||||
"greedy_token_match": "10/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"llamacpp_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_prob_overlap": "16/20",
|
||||
"top20_prob_max_abs_diff": 0.0041646957494917425,
|
||||
"top20_prob_mean_abs_diff": 0.0005320748771736946
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"engine": "llama.cpp",
|
||||
"gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single",
|
||||
"llamacpp_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single",
|
||||
"greedy_token_match": "23/24",
|
||||
"exact_match": false,
|
||||
"ref_top1": 3066,
|
||||
"llamacpp_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_prob_overlap": "14/20",
|
||||
"top20_prob_max_abs_diff": 0.21300125134717218,
|
||||
"top20_prob_mean_abs_diff": 0.030018658851745834
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"engine": "llama.cpp",
|
||||
"gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf",
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current",
|
||||
"llamacpp_text": "The user wants weather for Shenzhen in celsius. I have the weather function available. Let me call it with the city \"Shenzhen\"",
|
||||
"greedy_token_match": "3/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 1046,
|
||||
"llamacpp_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_prob_overlap": "13/20",
|
||||
"top20_prob_max_abs_diff": 0.0003068447281868725,
|
||||
"top20_prob_mean_abs_diff": 5.7490022653347724e-05
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"vllm_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district",
|
||||
"greedy_token_match": "32/32",
|
||||
"exact_match": true,
|
||||
"ref_top1": 3066,
|
||||
"vllm_first_token": 3066,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "20/20",
|
||||
"top20_logprob_max_abs_diff": 0.2409982681274414,
|
||||
"top20_logprob_mean_abs_diff": 0.146433162339963,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current",
|
||||
"vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available to get weather for a city. Let me call",
|
||||
"greedy_token_match": "22/32",
|
||||
"exact_match": false,
|
||||
"ref_top1": 1046,
|
||||
"vllm_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "19/20",
|
||||
"top20_logprob_max_abs_diff": 0.4140052795410156,
|
||||
"top20_logprob_mean_abs_diff": 0.21458697107032335,
|
||||
"dtype": "bfloat16"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather",
|
||||
"vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather",
|
||||
"greedy_token_match": "32/32",
|
||||
"exact_match": true,
|
||||
"ref_top1": 1046,
|
||||
"vllm_first_token": 1046,
|
||||
"first_token_agree": true,
|
||||
"top20_logprob_overlap": "20/20",
|
||||
"top20_logprob_max_abs_diff": 2.193450927734375e-05,
|
||||
"top20_logprob_mean_abs_diff": 1.334088810835965e-05,
|
||||
"dtype": "float32"
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{"tag": "vllm-bf16", "model": "XHToken/Spark-X2.5-1.7B", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.7763726822277026, "ppl": 43.657394933805286}
|
||||
{"tag": "vllm-int4-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.871370641182274, "ppl": 48.00814294379576}
|
||||
{"tag": "vllm-int4-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.949247471796542, "ppl": 51.89629870830851}
|
||||
{"tag": "vllm-int4-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.041093142384169, "ppl": 56.88849605431311}
|
||||
{"tag": "vllm-int4-g64-fpembed", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-fpembed", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.884205416004657, "ppl": 48.62828784511256}
|
||||
{"tag": "vllm-int4-g32-rtn", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32-rtn", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.039075346914289, "ppl": 56.7738224375464}
|
||||
{"tag": "vllm-int4-g64-rtn", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-rtn", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.089284296531948, "ppl": 59.69715095314776}
|
||||
{"tag": "expanded-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32-expanded", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.039537154311983, "ppl": 56.80004706364053}
|
||||
{"tag": "expanded-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-expanded", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.872951761505493, "ppl": 48.084109634697604}
|
||||
@@ -0,0 +1,6 @@
|
||||
{"tag": "c2-bf16", "model": "XHToken/Spark-X2.5-1.7B", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.3291262563013975, "ppl": 10.268965168486398}
|
||||
{"tag": "c2-int4-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.446570275075721, "ppl": 11.548669956733075}
|
||||
{"tag": "c2-int4-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.4960842284453117, "ppl": 12.134883374312405}
|
||||
{"tag": "c2-int4-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.4446298061138063, "ppl": 11.526281849868935}
|
||||
{"tag": "c2-gptq-w4a8", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.5695190991422434, "ppl": 13.05954258534955}
|
||||
{"tag": "c2-gptq-w4a16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.5562905847375963, "ppl": 12.887921884302088}
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 6103, "estimated_inference_peak_memory": 154943488, "first_load_time": 173117, "first_load_peak_memory": 113643520, "warm_load_time": 176691, "warm_load_peak_memory": 124940288, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2398992], "first_load_memory_peak_range": [1228800, 10888560], "warm_load_memory_increase_range": [0, 2296256], "warm_load_memory_peak_range": [335872, 10494528], "inference_memory_increase_range": [0, 6340656], "inference_memory_peak_range": [6332416, 15690592], "all_compile_times": [], "all_first_load_times": [173117], "all_warm_load_times": [176691], "all_inference_times": [8046, 6381, 6308, 6330, 6422, 6444, 6621, 6554, 6463, 7236, 6207, 6835, 6499, 6191, 6175, 6205, 6177, 6197, 6160, 6560, 6171, 6167, 6156, 6177, 6149, 6226, 6159, 6146, 6144, 6144, 6362, 6186, 6130, 6141, 6193, 6135, 6392, 6330, 6145, 6146, 6141, 6123, 6151, 6191, 7266, 6796, 6240, 6180, 6170, 6179, 6157, 6303, 6285, 6554, 6221, 6585, 6215, 6153, 6157, 6133, 6271, 6167, 6194, 6285, 6137, 6142, 6153, 6253, 6159, 6185, 6195, 6229, 7027, 6126, 6135, 6129, 6135, 6155, 6499, 6235, 6134, 6145, 6112, 6214, 6160, 6168, 6912, 6158, 6122, 6273, 6404, 6111, 6152, 6135, 6123, 6121, 6107, 6123, 6103, 6181]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 160867}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 135879}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10234}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_399", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 252656}, {"name": "node_Slice_402", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 59233}, {"name": "node_Slice_405", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 62582}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 495}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 249}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 267}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7496}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6813}, {"name": "node_Slice_1001", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5076}, {"name": "node_Slice_1004", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23170}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1040}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1180}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 340}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 963}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17162}, {"name": "node_Slice_2566", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_2569", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3367}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18517}, {"name": "/layers.0/Concat_1"Line truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 803, "estimated_inference_peak_memory": 138977280, "first_load_time": 104796, "first_load_peak_memory": 113455104, "warm_load_time": 105370, "warm_load_peak_memory": 105713664, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [3270000, 12452832], "first_load_memory_peak_range": [11902976, 21085808], "warm_load_memory_increase_range": [26110560, 36053536], "warm_load_memory_peak_range": [15630336, 25573312], "inference_memory_increase_range": [0, 1065008], "inference_memory_peak_range": [0, 9549376], "all_compile_times": [], "all_first_load_times": [104796], "all_warm_load_times": [105370], "all_inference_times": [1533, 870, 840, 852, 838, 813, 812, 808, 811, 814, 821, 973, 861, 843, 852, 837, 835, 843, 833, 824, 807, 808, 815, 812, 816, 822, 837, 835, 815, 807, 815, 808, 815, 806, 805, 805, 806, 857, 826, 805, 805, 804, 804, 810, 803, 804, 806, 804, 816, 805, 806, 811, 812, 818, 810, 805, 806, 806, 818, 807, 805, 806, 806, 819, 807, 806, 807, 848, 815, 1201, 987, 806, 805, 813, 805, 805, 805, 810, 804, 807, 804, 805, 806, 805, 813, 804, 810, 866, 837, 819, 803, 803, 806, 804, 812, 805, 803, 806, 804, 815]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 38815}, {"name": "/layers.0/Pow", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54240}, {"name": "/layers.0/ReduceMean", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10302}, {"name": "/layers.0/Add", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11317}, {"name": "/layers.0/Sqrt", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 72747}, {"name": "/layers.0/Mul", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 87612}, {"name": "/layers.0/Mul_1", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28026}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10746}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_647", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 100071}, {"name": "node_Slice_650", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20997}, {"name": "node_Slice_653", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7438}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 302}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6351}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16354}, {"name": "node_Slice_1165", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8720}, {"name": "node_Slice_1168", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1829}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16759}, {"name": "node_Slice_1388", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2646}, {"name": "node_Slice_1391", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8958}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22033}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2967}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11087}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execLine truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 749, "estimated_inference_peak_memory": 138420224, "first_load_time": 111059, "first_load_peak_memory": 113651712, "warm_load_time": 106186, "warm_load_peak_memory": 126267392, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2499648], "first_load_memory_peak_range": [1368064, 11294720], "warm_load_memory_increase_range": [0, 1585792], "warm_load_memory_peak_range": [1028096, 10936192], "inference_memory_increase_range": [0, 544816], "inference_memory_peak_range": [0, 9582304], "all_compile_times": [], "all_first_load_times": [111059], "all_warm_load_times": [106186], "all_inference_times": [1389, 831, 800, 794, 804, 876, 923, 996, 828, 813, 892, 871, 794, 777, 768, 781, 810, 941, 785, 896, 871, 837, 822, 809, 798, 799, 834, 759, 761, 761, 761, 767, 774, 764, 763, 798, 804, 772, 760, 749, 768, 800, 795, 769, 765, 837, 781, 789, 768, 771, 869, 851, 806, 874, 808, 810, 825, 806, 799, 797, 806, 979, 949, 909, 1089, 929, 816, 1141, 965, 782, 828, 811, 824, 834, 842, 870, 849, 826, 854, 844, 833, 821, 911, 964, 1004, 936, 927, 984, 823, 800, 789, 790, 800, 802, 792, 809, 781, 828, 809, 790]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20618}, {"name": "/layers.0/Pow", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 50076}, {"name": "/layers.0/ReduceMean", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3767}, {"name": "/layers.0/Add", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sqrt", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 264}, {"name": "/layers.0/Mul", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28389}, {"name": "/layers.0/Mul_1", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22514}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6739}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1544", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 104968}, {"name": "node_Slice_1547", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12500}, {"name": "node_Slice_1550", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11968}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 183}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 413}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7419}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3771}, {"name": "node_Slice_1583", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1692}, {"name": "node_Slice_1586", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 586}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 658}, {"name": "node_Slice_760", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2431}, {"name": "node_Slice_763", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4262}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9041}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1201}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2316}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "executionLine truncated
|
||||
@@ -0,0 +1,224 @@
|
||||
{
|
||||
"tag": "bfloat16/None/eager=False",
|
||||
"wall_step_ms": 17.25,
|
||||
"tps": 57.97,
|
||||
"profiled_total_ms_per_step": 16.484,
|
||||
"buckets_ms_per_step": {
|
||||
"gemm": 10.661,
|
||||
"framework": 3.502,
|
||||
"other_op": 1.165,
|
||||
"attn": 1.157
|
||||
},
|
||||
"top": [
|
||||
{
|
||||
"op": "aten::_weight_int4pack_mm_for_cpu",
|
||||
"bucket": "gemm",
|
||||
"ms_per_step": 8.8694,
|
||||
"calls_per_step": 113.0
|
||||
},
|
||||
{
|
||||
"op": "## Call CompiledFxGraph None ##",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 2.7736,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::mm",
|
||||
"bucket": "gemm",
|
||||
"ms_per_step": 1.1547,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::cpu_int4_linear",
|
||||
"bucket": "gemm",
|
||||
"ms_per_step": 0.6367,
|
||||
"calls_per_step": 113.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::unified_kv_cache_update",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.5477,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::unified_attention_with_output",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.5346,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "_C::cpu_attention_with_kv_cache",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.4428,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::argmax",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.2687,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "_C::cpu_attn_reshape_and_cache",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.1792,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::slice",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.1786,
|
||||
"calls_per_step": 230.0
|
||||
},
|
||||
{
|
||||
"op": "Pregraph bytecode",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.1727,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::narrow",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.1042,
|
||||
"calls_per_step": 112.0
|
||||
},
|
||||
{
|
||||
"op": "aten::as_strided",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0839,
|
||||
"calls_per_step": 234.0
|
||||
},
|
||||
{
|
||||
"op": "aten::copy_",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0833,
|
||||
"calls_per_step": 19.0
|
||||
},
|
||||
{
|
||||
"op": "aten::split",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0712,
|
||||
"calls_per_step": 56.0
|
||||
},
|
||||
{
|
||||
"op": "aten::chunk",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.07,
|
||||
"calls_per_step": 56.0
|
||||
},
|
||||
{
|
||||
"op": "aten::view",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0631,
|
||||
"calls_per_step": 62.0
|
||||
},
|
||||
{
|
||||
"op": "aten::empty",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0551,
|
||||
"calls_per_step": 145.0
|
||||
},
|
||||
{
|
||||
"op": "aten::index",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0345,
|
||||
"calls_per_step": 2.0
|
||||
},
|
||||
{
|
||||
"op": "_C::compute_slot_mapping_kernel_impl",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.022,
|
||||
"calls_per_step": 4.0
|
||||
},
|
||||
{
|
||||
"op": "aten::new_empty",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0187,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "_C::get_scheduler_metadata",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0184,
|
||||
"calls_per_step": 4.0
|
||||
},
|
||||
{
|
||||
"op": "aten::_to_copy",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0168,
|
||||
"calls_per_step": 5.0
|
||||
},
|
||||
{
|
||||
"op": "aten::add",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0115,
|
||||
"calls_per_step": 3.0
|
||||
},
|
||||
{
|
||||
"op": "aten::fill_",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0112,
|
||||
"calls_per_step": 12.0
|
||||
},
|
||||
{
|
||||
"op": "aten::resolve_conj",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0096,
|
||||
"calls_per_step": 59.0
|
||||
},
|
||||
{
|
||||
"op": "AOTDispatcher Runtime Wrapper Prologue",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.0077,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::empty_strided",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0073,
|
||||
"calls_per_step": 5.0
|
||||
},
|
||||
{
|
||||
"op": "aten::to",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.007,
|
||||
"calls_per_step": 8.0
|
||||
},
|
||||
{
|
||||
"op": "aten::sub",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0063,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::lt",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0049,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::index_select",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0043,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::detach",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0037,
|
||||
"calls_per_step": 2.0
|
||||
},
|
||||
{
|
||||
"op": "aten::reshape",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0031,
|
||||
"calls_per_step": 4.0
|
||||
},
|
||||
{
|
||||
"op": "aten::alias",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0026,
|
||||
"calls_per_step": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
{
|
||||
"tag": "bfloat16/None/eager=False",
|
||||
"wall_step_ms": 14.345,
|
||||
"tps": 69.71,
|
||||
"profiled_total_ms_per_step": 13.87,
|
||||
"buckets_ms_per_step": {
|
||||
"gemm": 8.263,
|
||||
"framework": 3.017,
|
||||
"other_op": 1.581,
|
||||
"attn": 1.009
|
||||
},
|
||||
"top": [
|
||||
{
|
||||
"op": "aten::_weight_int4pack_mm_for_cpu",
|
||||
"bucket": "gemm",
|
||||
"ms_per_step": 7.7116,
|
||||
"calls_per_step": 113.0
|
||||
},
|
||||
{
|
||||
"op": "## Call CompiledFxGraph None ##",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 2.2035,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::cpu_int4_linear",
|
||||
"bucket": "gemm",
|
||||
"ms_per_step": 0.5513,
|
||||
"calls_per_step": 113.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::unified_attention_with_output",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.4727,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::unified_kv_cache_update",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.4491,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "_C::cpu_attention_with_kv_cache",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.3842,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::copy_",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.2688,
|
||||
"calls_per_step": 75.0
|
||||
},
|
||||
{
|
||||
"op": "aten::argmax",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.2479,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "vllm::spark_head_gate",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.2182,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "_C::cpu_attn_reshape_and_cache",
|
||||
"bucket": "attn",
|
||||
"ms_per_step": 0.1523,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::slice",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.152,
|
||||
"calls_per_step": 230.0
|
||||
},
|
||||
{
|
||||
"op": "Pregraph bytecode",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.1399,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::mul",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.1302,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::sum",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0956,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::as_strided",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0952,
|
||||
"calls_per_step": 290.0
|
||||
},
|
||||
{
|
||||
"op": "aten::narrow",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0844,
|
||||
"calls_per_step": 112.0
|
||||
},
|
||||
{
|
||||
"op": "aten::_to_copy",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0794,
|
||||
"calls_per_step": 61.0
|
||||
},
|
||||
{
|
||||
"op": "aten::chunk",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0631,
|
||||
"calls_per_step": 56.0
|
||||
},
|
||||
{
|
||||
"op": "aten::split",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0547,
|
||||
"calls_per_step": 56.0
|
||||
},
|
||||
{
|
||||
"op": "aten::view",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0531,
|
||||
"calls_per_step": 62.0
|
||||
},
|
||||
{
|
||||
"op": "aten::empty",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0456,
|
||||
"calls_per_step": 145.0
|
||||
},
|
||||
{
|
||||
"op": "aten::to",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0335,
|
||||
"calls_per_step": 64.0
|
||||
},
|
||||
{
|
||||
"op": "aten::empty_strided",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0316,
|
||||
"calls_per_step": 61.0
|
||||
},
|
||||
{
|
||||
"op": "aten::fill_",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0244,
|
||||
"calls_per_step": 40.0
|
||||
},
|
||||
{
|
||||
"op": "aten::index",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0217,
|
||||
"calls_per_step": 2.0
|
||||
},
|
||||
{
|
||||
"op": "aten::unsqueeze",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0184,
|
||||
"calls_per_step": 29.0
|
||||
},
|
||||
{
|
||||
"op": "_C::compute_slot_mapping_kernel_impl",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.018,
|
||||
"calls_per_step": 4.0
|
||||
},
|
||||
{
|
||||
"op": "_C::get_scheduler_metadata",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0161,
|
||||
"calls_per_step": 4.0
|
||||
},
|
||||
{
|
||||
"op": "aten::new_empty",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0149,
|
||||
"calls_per_step": 28.0
|
||||
},
|
||||
{
|
||||
"op": "aten::add",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0095,
|
||||
"calls_per_step": 3.0
|
||||
},
|
||||
{
|
||||
"op": "AOTDispatcher Runtime Wrapper Prologue",
|
||||
"bucket": "framework",
|
||||
"ms_per_step": 0.0063,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::sub",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.005,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::lt",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.004,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::index_select",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0034,
|
||||
"calls_per_step": 1.0
|
||||
},
|
||||
{
|
||||
"op": "aten::reshape",
|
||||
"bucket": "other_op",
|
||||
"ms_per_step": 0.0024,
|
||||
"calls_per_step": 4.0
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 3868, "estimated_inference_peak_memory": 137539584, "first_load_time": 140576, "first_load_peak_memory": 113246208, "warm_load_time": 149076, "warm_load_peak_memory": 125939712, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 1046944], "first_load_memory_peak_range": [753664, 9164752], "warm_load_memory_increase_range": [0, 2845344], "warm_load_memory_peak_range": [2326528, 11678224], "inference_memory_increase_range": [0, 75920], "inference_memory_peak_range": [24576, 8852352], "all_compile_times": [], "all_first_load_times": [140576], "all_warm_load_times": [149076], "all_inference_times": [5046, 4616, 4128, 4077, 4103, 4020, 3940, 3894, 4265, 4737, 4008, 4543, 4147, 4109, 4144, 4048, 3980, 3885, 3903, 3879, 3882, 3901, 3951, 3895, 3926, 3923, 3963, 3895, 3873, 4657, 3889, 3969, 3880, 3892, 4748, 3893, 3895, 3891, 3887, 3876, 3868, 3903, 3879, 3932, 4737, 3915, 3874, 3892, 3889, 3908, 3887, 3898, 3880, 3897, 3877, 3889, 3887, 3890, 3877, 4039, 3941, 3964, 3876, 3892, 3871, 3900, 3887, 3928, 3992, 3956, 3974, 5026, 4694, 4079, 4146, 4183, 4420, 4226, 4090, 4034, 3879, 3956, 3985, 3897, 3879, 3885, 3874, 3906, 3875, 3893, 3881, 4595, 3884, 3885, 3876, 3890, 3877, 3880, 3882, 3875]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11685}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 86165}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1495}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9206236}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35181}]}
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 3862, "estimated_inference_peak_memory": 138051584, "first_load_time": 139684, "first_load_peak_memory": 113541120, "warm_load_time": 146509, "warm_load_peak_memory": 125521920, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [2583936, 12322976], "first_load_memory_peak_range": [11759616, 21498656], "warm_load_memory_increase_range": [0, 1511168], "warm_load_memory_peak_range": [983040, 11197440], "inference_memory_increase_range": [0, 24624], "inference_memory_peak_range": [20480, 9138416], "all_compile_times": [], "all_first_load_times": [139684], "all_warm_load_times": [146509], "all_inference_times": [5092, 4632, 4065, 3881, 3887, 3880, 3894, 3879, 3888, 3888, 3876, 4173, 4120, 3950, 3914, 3967, 3883, 3897, 3912, 3978, 3984, 3885, 3951, 3901, 3879, 3948, 3880, 3891, 3880, 3890, 3965, 3891, 3884, 3891, 3897, 3885, 3881, 3882, 3905, 3880, 3895, 3898, 3909, 3862, 3877, 3877, 3872, 3872, 3923, 3928, 4000, 3953, 3980, 3959, 3962, 4915, 4755, 4061, 4097, 4072, 4171, 4131, 4070, 4054, 3944, 3908, 3892, 3887, 3877, 3918, 3982, 3874, 3873, 3874, 3893, 4553, 3871, 3876, 3888, 4111, 3866, 3893, 3870, 3877, 4307, 4161, 4048, 3968, 3883, 3936, 4038, 3969, 3958, 4774, 4849, 4110, 4140, 4342, 4381, 4277]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14198}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 151909}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1307}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9681411}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35501}]}
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1480, "estimated_inference_peak_memory": 140103680, "first_load_time": 111241, "first_load_peak_memory": 113639424, "warm_load_time": 115382, "warm_load_peak_memory": 125554688, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2062624], "first_load_memory_peak_range": [1265664, 10942624], "warm_load_memory_increase_range": [0, 4168816], "warm_load_memory_peak_range": [3784704, 13497104], "inference_memory_increase_range": [0, 2121776], "inference_memory_peak_range": [0, 8779808], "all_compile_times": [], "all_first_load_times": [111241], "all_warm_load_times": [115382], "all_inference_times": [2204, 1622, 1592, 1598, 1612, 1603, 1548, 1581, 1571, 1685, 2255, 1731, 1547, 1667, 1673, 1719, 1543, 1534, 1513, 1505, 1495, 1494, 1515, 1498, 1492, 1507, 1503, 1529, 1519, 1490, 1489, 1494, 1487, 1491, 1490, 1483, 1495, 1488, 1490, 1489, 1505, 1496, 1497, 1538, 1518, 1532, 1532, 1525, 1530, 1505, 1519, 1521, 1827, 1747, 1985, 1681, 2027, 1567, 1609, 1607, 1751, 1855, 1866, 1829, 1804, 1615, 1550, 1606, 1556, 1560, 1597, 1576, 2375, 1547, 1522, 1544, 1519, 1483, 1494, 1482, 1482, 1497, 1480, 1485, 1503, 1510, 1486, 1511, 1491, 1481, 1498, 1484, 1562, 1670, 1582, 1666, 1716, 1618, 1711, 1642]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6592}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 107275}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23416}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_347", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 251480}, {"name": "node_Slice_350", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 60490}, {"name": "node_Slice_353", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 39159}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 482}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 258}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20778}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8519}, {"name": "node_Slice_521", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_524", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22527}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1026}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1024}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2223}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 979}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8600}, {"name": "node_Slice_476", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_479", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4362}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15065}, {"name": "/layers.0/Concat_1", "type": "/layeLine truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1486, "estimated_inference_peak_memory": 139599872, "first_load_time": 115214, "first_load_peak_memory": 109510656, "warm_load_time": 114073, "warm_load_peak_memory": 125624320, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [14961536, 24144032], "first_load_memory_peak_range": [19734528, 28917024], "warm_load_memory_increase_range": [0, 827344], "warm_load_memory_peak_range": [737280, 10669568], "inference_memory_increase_range": [0, 2129968], "inference_memory_peak_range": [0, 9903648], "all_compile_times": [], "all_first_load_times": [115214], "all_warm_load_times": [114073], "all_inference_times": [2310, 1720, 1648, 1682, 1698, 1625, 1628, 1619, 1561, 1514, 1490, 1986, 1815, 1857, 1898, 1837, 1814, 1797, 1788, 1727, 1612, 1642, 1615, 1601, 1504, 1627, 1506, 1494, 1531, 1499, 1491, 1513, 1499, 1565, 1510, 1493, 1497, 1500, 1494, 1496, 1499, 1488, 1494, 1505, 1488, 1494, 1493, 1488, 1493, 1493, 1491, 1520, 1523, 3012, 1500, 1512, 1507, 1502, 1495, 1501, 1496, 1493, 1508, 1494, 1533, 1518, 1497, 1508, 1520, 1501, 1503, 1500, 1499, 1501, 1497, 1577, 1749, 1645, 1707, 1657, 1586, 1646, 1564, 1676, 1557, 1587, 1524, 1519, 1594, 1578, 1517, 1517, 1586, 1545, 1501, 1543, 1492, 1516, 1486, 1496]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7431}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 121888}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18417}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 262714}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69757}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54734}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 488}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 242}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18138}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9962}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24465}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1032}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1068}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2060}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 958}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13664}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1716}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7234}, {"name": "/layers.0/Concat_1", "type":Line truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 838, "estimated_inference_peak_memory": 138752000, "first_load_time": 104984, "first_load_peak_memory": 113438720, "warm_load_time": 104086, "warm_load_peak_memory": 125595648, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 8178736], "first_load_memory_peak_range": [7778304, 16937392], "warm_load_memory_increase_range": [0, 1753392], "warm_load_memory_peak_range": [872448, 11059632], "inference_memory_increase_range": [0, 1069104], "inference_memory_peak_range": [0, 9035184], "all_compile_times": [], "all_first_load_times": [104984], "all_warm_load_times": [104086], "all_inference_times": [1563, 1071, 986, 987, 979, 919, 891, 900, 880, 880, 874, 1063, 935, 943, 935, 941, 937, 952, 846, 867, 840, 850, 887, 883, 904, 901, 867, 868, 897, 899, 867, 867, 867, 854, 890, 953, 928, 956, 1122, 997, 1083, 1165, 908, 911, 917, 1029, 1172, 913, 953, 929, 933, 943, 960, 947, 876, 894, 953, 979, 917, 932, 917, 916, 942, 932, 962, 922, 924, 907, 862, 917, 928, 930, 933, 940, 942, 907, 930, 936, 916, 838, 845, 866, 885, 871, 873, 873, 887, 877, 966, 890, 869, 874, 862, 871, 856, 842, 872, 873, 867, 877]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 478820}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 53145}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 126}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 83913}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9103}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16251}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5974}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2653}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1451}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 328}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 355}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10257}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 492}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 865}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12237}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1279}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5166}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5196}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11283}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1134}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1710}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11689}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2282}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycleLine truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 751, "estimated_inference_peak_memory": 138072064, "first_load_time": 114182, "first_load_peak_memory": 113254400, "warm_load_time": 112303, "warm_load_peak_memory": 125153280, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2425792], "first_load_memory_peak_range": [1277952, 10691280], "warm_load_memory_increase_range": [0, 3745856], "warm_load_memory_peak_range": [3039232, 12596560], "inference_memory_increase_range": [0, 553008], "inference_memory_peak_range": [0, 9093312], "all_compile_times": [], "all_first_load_times": [114182], "all_warm_load_times": [112303], "all_inference_times": [1352, 814, 790, 796, 789, 800, 795, 858, 806, 771, 780, 919, 781, 782, 779, 833, 846, 867, 796, 777, 793, 790, 814, 793, 771, 778, 755, 756, 752, 801, 760, 761, 751, 753, 752, 752, 758, 768, 755, 762, 763, 778, 807, 822, 786, 801, 827, 763, 752, 768, 762, 784, 765, 769, 765, 789, 817, 798, 802, 769, 804, 779, 844, 806, 790, 798, 824, 776, 767, 763, 766, 764, 774, 778, 762, 781, 766, 792, 822, 1065, 1101, 1252, 967, 1195, 1185, 831, 805, 781, 824, 810, 815, 859, 826, 837, 910, 787, 780, 790, 841, 786]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 196526}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35201}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 50146}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14700}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 41331}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 435}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3404}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 288}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2107}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7760}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7794}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 614}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 322}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2028}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2098}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4610}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4698}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5761}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4413}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1462}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 402}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, Line truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1567, "estimated_inference_peak_memory": 143167488, "first_load_time": 121550, "first_load_peak_memory": 114778112, "warm_load_time": 117498, "warm_load_peak_memory": 126844928, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2441456], "first_load_memory_peak_range": [1359872, 11835536], "warm_load_memory_increase_range": [0, 1691984], "warm_load_memory_peak_range": [0, 10525536], "inference_memory_increase_range": [0, 28720], "inference_memory_peak_range": [0, 9662544], "all_compile_times": [], "all_first_load_times": [121550], "all_warm_load_times": [117498], "all_inference_times": [2677, 2072, 1930, 2042, 1982, 1969, 1903, 1912, 1846, 1784, 1692, 1993, 1785, 1812, 1787, 1756, 1743, 1670, 1613, 1606, 1599, 1621, 1618, 1601, 1596, 2285, 1599, 1607, 1588, 1617, 1597, 1638, 1702, 1693, 1645, 1595, 1616, 1652, 1612, 1591, 1642, 1588, 1581, 1573, 1584, 1595, 1581, 1577, 1570, 1580, 1567, 1638, 1591, 1584, 1598, 1617, 1690, 1738, 1663, 1617, 1629, 1641, 1643, 1917, 2078, 1761, 2192, 1656, 1631, 1637, 1779, 1783, 1766, 1842, 1810, 1917, 1760, 1681, 1768, 1971, 1943, 1774, 1854, 1835, 1790, 1685, 1717, 1633, 1619, 1603, 1642, 1640, 1619, 1674, 1624, 1602, 1630, 1720, 1611, 1605]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 83386}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 171352}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28930}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_141", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 284809}, {"name": "node_Slice_144", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 82995}, {"name": "node_Slice_147", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 48282}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 558}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 288}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 230}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7087}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7297}, {"name": "node_Slice_120", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_123", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 572}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22821}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 342}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 520}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6280}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 312}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3224}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2145}, {"name": "node_Slice_127", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0},Line truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1483, "estimated_inference_peak_memory": 140337152, "first_load_time": 114989, "first_load_peak_memory": 113680384, "warm_load_time": 118746, "warm_load_peak_memory": 121503744, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2375744], "first_load_memory_peak_range": [1220608, 10869504], "warm_load_memory_increase_range": [0, 4568256], "warm_load_memory_peak_range": [0, 9384288], "inference_memory_increase_range": [0, 2134064], "inference_memory_peak_range": [0, 8743104], "all_compile_times": [], "all_first_load_times": [114989], "all_warm_load_times": [118746], "all_inference_times": [2295, 1751, 1827, 1751, 1802, 1710, 1746, 1602, 1577, 1569, 1524, 1734, 1576, 1527, 1514, 1500, 1500, 1557, 1606, 1572, 1678, 1542, 1530, 1579, 1547, 1513, 1522, 1503, 1492, 1499, 1507, 1492, 1502, 1488, 1494, 1510, 1487, 1497, 1487, 1485, 1495, 1483, 1484, 1491, 1515, 1537, 1613, 1611, 1561, 1585, 1540, 1542, 1539, 1516, 1520, 1495, 1488, 1497, 1496, 1495, 1501, 1541, 1500, 1487, 1489, 1495, 1486, 1483, 1490, 1484, 1488, 1512, 1573, 1544, 1609, 1587, 1553, 1563, 1538, 1589, 1529, 1514, 1527, 1489, 1497, 1495, 1488, 1488, 1500, 1524, 1614, 1521, 1518, 1502, 1491, 1489, 1500, 1507, 1505, 1506]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15910}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 88732}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19220}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_414", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 239352}, {"name": "node_Slice_417", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65301}, {"name": "node_Slice_420", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 51953}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 492}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 240}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 238}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18424}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6051}, {"name": "node_Slice_430", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3276}, {"name": "node_Slice_433", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18262}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1016}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1028}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 728}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 996}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14892}, {"name": "node_Slice_437", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_440", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2710}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14963}, {"name": "/layers.0/Concat_1", "type": "/layersLine truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1535, "estimated_inference_peak_memory": 140099584, "first_load_time": 118480, "first_load_peak_memory": 113537024, "warm_load_time": 119765, "warm_load_peak_memory": 124760064, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 3750256], "first_load_memory_peak_range": [2940928, 12303072], "warm_load_memory_increase_range": [0, 1695824], "warm_load_memory_peak_range": [0, 10174496], "inference_memory_increase_range": [0, 2129968], "inference_memory_peak_range": [0, 9041792], "all_compile_times": [], "all_first_load_times": [118480], "all_warm_load_times": [119765], "all_inference_times": [2433, 1845, 1775, 1750, 1746, 1712, 1710, 1617, 1608, 1599, 1592, 2135, 1871, 1768, 1643, 1652, 1625, 1601, 1642, 1628, 1594, 1599, 1586, 1684, 1596, 1673, 1794, 1720, 1643, 1633, 1785, 1822, 1792, 1709, 1677, 1608, 1569, 1561, 1567, 1559, 1559, 1588, 1585, 1571, 1599, 1555, 1553, 1599, 1702, 1771, 1831, 1741, 1813, 1716, 1891, 1808, 1710, 1573, 1564, 1568, 1555, 1546, 1557, 1547, 1544, 1550, 1535, 1550, 1578, 1621, 1620, 1611, 1576, 1583, 1640, 1649, 1675, 1589, 1605, 1734, 1707, 1830, 1744, 1761, 1783, 1618, 1707, 1621, 1652, 1712, 1616, 1601, 1605, 1669, 1593, 1659, 1790, 1640, 1628, 1598]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16153}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 110098}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19199}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_180", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 250822}, {"name": "node_Slice_183", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 73962}, {"name": "node_Slice_186", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 60326}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 468}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 232}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18026}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4499}, {"name": "node_Slice_331", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2506}, {"name": "node_Slice_334", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19291}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1037}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2438}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2102}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 975}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14896}, {"name": "node_Slice_338", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_341", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2937}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11155}, {"name": "/layers.0/Concat_1", "type": "/layerLine truncated
|
||||
@@ -0,0 +1 @@
|
||||
{"execution_summary": {"estimated_inference_time": 1559, "estimated_inference_peak_memory": 142434304, "first_load_time": 111668, "first_load_peak_memory": 105967616, "warm_load_time": 114274, "warm_load_peak_memory": 125005824, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 7911872], "first_load_memory_peak_range": [0, 8989584], "warm_load_memory_increase_range": [0, 2039504], "warm_load_memory_peak_range": [0, 10238496], "inference_memory_increase_range": [0, 36912], "inference_memory_peak_range": [0, 8819840], "all_compile_times": [], "all_first_load_times": [111668], "all_warm_load_times": [114274], "all_inference_times": [2843, 2228, 2052, 1796, 1909, 1731, 1743, 1744, 1772, 1735, 1709, 2235, 2071, 2200, 1893, 1750, 1730, 1763, 1742, 1731, 1727, 1699, 1902, 2682, 1716, 1695, 1750, 1714, 1825, 1681, 1637, 1656, 1635, 1627, 1641, 1620, 1676, 1629, 1640, 1611, 1634, 1657, 1593, 1666, 1585, 1573, 1592, 1578, 1568, 1568, 1572, 1572, 1565, 1575, 1634, 1573, 1568, 1565, 1582, 1562, 1577, 1575, 1593, 1578, 1564, 1576, 1574, 1618, 1630, 2442, 1601, 1653, 1571, 1594, 1579, 1578, 1572, 1582, 1562, 1565, 1561, 1593, 1565, 1563, 1583, 1570, 1617, 1562, 1559, 1572, 1660, 1762, 1855, 1703, 1585, 1563, 1563, 1574, 1569, 1559]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69254}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 97184}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20589}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 243001}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 94930}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 55020}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1211}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 246}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1777}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2550}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 554}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18138}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 318}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3683}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4358}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 299}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3673}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2072}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name"Line truncated
|
||||
Loaded 100 of 162 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user