anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
ab2f2a3d658db5be5ec96a950da55cb74b48e7aa
vllm-omni/baselines/quality_vllm_w4a16.json
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork 
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

817 lines
13 KiB
JSON
Raw Blame History

{
"chat_short": {
"text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"Paris\". But the instruction says \"Answer in one word.\" So I should output \"Paris\" as the final answer.</think>Paris",
"token_ids": [
3066,
599,
5509,
39,
518,
4766,
390,
275,
11862,
314,
17349,
44,
74288,
304,
1135,
5240,
1848,
4358,
275,
6787,
2029,
417,
1839,
518,
110949,
4348,
2964,
275,
23220,
5685,
518,
47648,
304,
1135,
5240,
1848,
4358,
373,
2029,
6340,
518,
110949,
15,
463,
275,
4625,
6787,
27,
4,
110949,
1
]
},
"chat_long": {
"text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check each item's district number and code.\nWait, let's list them out properly. Let's see:\nItem 1: district 1, code",
"token_ids": [
71430,
488,
25,
1553,
1045,
2261,
27,
614,
3754,
390,
17186,
396,
275,
5760,
314,
56774,
20882,
220,
36,
32,
27,
9619,
1045,
1940,
1953,
275,
31946,
2631,
27,
9619,
1045,
4069,
2440,
7006,
1045,
20882,
2823,
317,
5760,
27,
198,
34542,
25,
1553,
1045,
2631,
1421,
971,
17227,
27,
9619,
1045,
2261,
39,
198,
5369,
220,
30,
39,
20882,
220,
30,
25,
5760
]
},
"chat_tools": {
"text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather for Shenzhen, and then I'll need to present it in celsius.\n\nLet me call the get_weather function with the city \"S",
"token_ids": [
1046,
3754,
13025,
312,
1843,
275,
14510,
304,
75040,
103,
872,
2430,
2343,
25,
304,
281,
108364,
27,
373,
1633,
312,
1730,
275,
1259,
76,
64301,
2122,
312,
24534,
275,
3345,
14510,
396,
75040,
103,
872,
25,
317,
1959,
373,
7632,
1633,
312,
3339,
488,
304,
281,
108364,
27,
198,
198,
13104,
713,
2426,
275,
1259,
76,
64301,
2122,
478,
275,
6594,
518,
64
]
},
"plain_0": {
"text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is",
"token_ids": [
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390,
390
]
},
"plain_1": {
"text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission",
"token_ids": [
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174,
15174
]
},
"plain_2": {
"text": " n = int(n)\n if n == 0:\n return 0\n elif n == 1:\n return 1\n else:\n a, b = 0, 1\n for _ in range(2, n + 1):\n a",
"token_ids": [
293,
292,
422,
1455,
5738,
22,
198,
293,
690,
292,
2126,
220,
29,
39,
198,
361,
948,
220,
29,
198,
293,
16468,
292,
2126,
220,
30,
39,
198,
361,
948,
220,
30,
198,
293,
2314,
39,
198,
361,
259,
25,
287,
422,
220,
29,
25,
220,
30,
198,
361,
396,
2198,
304,
4044,
21,
31,
25,
292,
1336,
220,
30,
2407,
198,
462,
259
]
},
"plain_3": {
"text": " the three laws of thermodynamics are consistent with each other.\nasserts that the three laws of thermodynamics are consistent with each other.\nassertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat",
"token_ids": [
275,
3182,
14672,
314,
13294,
121223,
599,
14945,
478,
2440,
1436,
27,
198,
787,
19922,
455,
275,
3182,
14672,
314,
13294,
121223,
599,
14945,
478,
2440,
1436,
27,
198,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837,
21,
13462,
7837
]
},
"plain_4": {
"text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to",
"token_ids": [
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312,
1996,
259,
3650,
128403,
454,
1633,
312
]
},
"plain_5": {
"text": " the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the",
"token_ids": [
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275,
275
]
},
"plain_6": {
"text": " glucose into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into",
"token_ids": [
28714,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464,
5404,
1464
]
},
"plain_7": {
"text": " supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as",
"token_ids": [
20815,
13127,
2129,
463,
54525,
25,
62316,
25,
317,
25124,
27,
401,
20470,
6258,
9905,
390,
259,
1870,
6982,
455,
20815,
13127,
2129,
463,
54525,
25,
62316,
25,
317,
25124,
27,
401,
20470,
6258,
9905,
390,
259,
1870,
6982,
455,
20815,
13127,
2129,
463,
54525,
25,
62316,
25,
317,
25124,
27,
401,
20470,
6258,
9905,
390,
259,
1870,
6982,
455,
20815,
13127,
2129,
463
]
},
"plain_8": {
"text": " the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the",
"token_ids": [
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275,
120455,
314,
10519,
1672,
14288,
572,
19750,
304,
275
]
}
}
Reference in new issue View Git Blame Copy Permalink